Arbitrarily-large resource properties should be allowed in events

As stated in the bug, the issue with the events table is that the
resource_properties is constrained to 2^16 bytes. However, the
resource.properties_data column is a heat.db.sqlalchemy.types.Json
type or 2^32 bytes (4 GB). I think for our purposes, a 4GB column will
satisfy the "arbitrarily large properties" requirement.

So, we can move the properties_data into a separate table, call it
resource_properties_data (rpd). Given that assumption, there are
subtle variations in how we could structure the data.

We could have foreign key column (fk_resource_id) in rpd to refer to
resource.id. But we'll need a way of identifying what the current rpd
entry is (i.e. what is currently stored in
resource.propeties_data). One possibility is to enforce the policy
where only the current rpd entry does have a not-null fk_resource_id.

The events table is more straightforward. We can just replace the
events.resource_properties with a foreign key column that points to
rpd.id. However, We will need to take care to prune an rpd entry
whenever its event is pruned.

I plan on getting started on the above soon. If anyone has strong
opinions on the above, it would be good to hear them now. :-)

No strong opinions, but it may be that we don't need to reference the resource id from the r-p-d table at all. Both the event and the resource hold the resource id already, and they can each just have a reference to the row in the r-p-d table.

One thing you'll have to watch out for is references after the migration of the existing Resource table - you'll either need to ensure that the Resource and the latest Event point to the same r-p-d or have some way of cleaning up r-p-d entries that are not referenced by an event when the resource is updated/deleted.

+1 to your plan though. I agree that 4GB is definitely sufficient for these purposes.

Fix proposed to branch: master
Review: https://review.openstack.org/267953

Change abandoned by Crag Wolfe (<email address hidden>) on branch: master
Review: https://review.openstack.org/267953
Reason: Abandoning in favor of https://review.openstack.org/#/c/363415/ , the last in a series of commits.

Reviewed: https://review.openstack.org/363415
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=57c5aae88b908002d84a933738293c5917926174
Submitter: Jenkins
Branch: master

commit 57c5aae88b908002d84a933738293c5917926174
Author: Crag Wolfe <email address hidden>
Date: Wed Aug 31 01:47:58 2016 -0400

    De-duplicate properties_data between events, resources

    Properties data (encrypted or not) is now stored in the
    resource_properties_data table. We still support reading properties
    data from the legacy locations (separate columns in the resource and
    event tables) but all new properties data are written to
    resource_properties_data.

    Instead of duplicating properties data for a given resource
    and its events, we refer to the same resource_properties_data
    object from both resources and events.

    Notes about encrypting/decrypting properties data:

    1. ResourcePropertiesData takes care of encrypting/decrypting, so
       consumers (events and resources) don't have to worry about it.
    2. Previously (not anymore), heat.engine.resource would take care of
       the encrypting in _store() and _store_or_update().
    3. Previously and currently, heat.object.resource decrypts the legacy
       properties_data field if needed.

    Change-Id: I569e16d4a7d3f5ccc22d57fe881c4fd5994677ac
    Closes-Bug: #1524013

This issue was fixed in the openstack/heat 8.0.0.0b3 development milestone.