instance_system_metadata column type truncates on snapshot

Bug #1117923 reported by Lance Bragstad
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Low
Ameed Ashour

Bug Description

When snapshotting an instance, any additional metadata that is added to the image using 'glance image-update --property key=value' that is over 255 character gets truncated because the nova.instance_system_metadata table has a limit of 255 characters. When the snapshot completes this can be verified by using either 'glance image-show <image-id>' or 'nova image-show <image-id>'.

Steps to reproduce:
1.) Add new image to glance
2.) Use 'glance image-update --property key=value' to add an amount of data larger than 255 character
3.) Launch an instance from the previously edited image so it contains the new metadata
4.) Snapshot the image
5.) Check the metadata property using 'nova image-show <new-snapshot-id>' or 'glance image-show <new-snapshot-id>'

Result:
The expected output would be the entire property initially added to the image before the launching of the original image or the snapshot. The actual result is that the large property will be truncated at 255 characters because of the limit of the column (field = value) type in the nova.instance_system_metadata table:

mysql> describe instance_system_metadata;
+---------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+----------------+
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| deleted | tinyint(1) | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment|
| instance_uuid | varchar(36) | NO | MUL | NULL | |
| key | varchar(255) | NO | | NULL | |
| value | varchar(255) | YES | | NULL | |
+---------------+--------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

The information stored in instance_system_metadata is stored in the corresponding Glance database table glance.image_properties, which doesn't have a limit on the 'value' row. Which is why there isn't an issue adding the large property to Glance. The 'value' field is of type text, which is inconsistent from nova.instance_system_metadata which is of type varchar(255)

mysql> describe image_properties;
+------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| image_id | varchar(36) | NO | MUL | NULL | |
| name | varchar(255) | NO | | NULL | |
| value | text | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| deleted_at | datetime | YES | | NULL | |
| deleted | tinyint(1) | NO | MUL | NULL | |
+------------+--------------+------+-----+---------+----------------+
8 rows in set (0.00 sec)

No error will be thrown since it is not checked for when the value is written to the database.

Tags: api glancev2
description: updated
description: updated
description: updated
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Are we saying we need to remove restriction on the value column in instance_system_metadata just like image_properties?

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I would say that is the first thing to investigate. I am still researching a fix and trying a few things out. Since the glance db stores the same information wouldn't it make sense to have the nova db match?

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I found a check in nova/compute/apy.py, but I am not sure where the truncation actually takes place since I changed the type of the field to 'text' to match what the corresponding glance.image_properties tables has for the same value. The check is here:

https://github.com/openstack/nova/blob/master/nova/compute/api.py#L317

Revision history for this message
Lance Bragstad (lbragstad) wrote :

This might not only be an issue with the database. After adjusting the database, I don't think that is the problem here. I think the real problem lies at the following line:

https://github.com/openstack/nova/blob/master/nova/compute/api.py#L806

Which truncates the property before writing it to the database. According to the comment above, this looks like it might need to be updated.

Revision history for this message
Chuck Short (zulcss) wrote :

I have seen this as well.

Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Lance Bragstad (lbragstad) wrote :

I have a little more to add on my findings. Originally looking at the issue, the truncation seemed to be a side effect of the field type in the database that stores image properties metadata. After writing a specific migration script to test if this was the root cause of the truncation, the problem still persisted. After investigating further down the rabbit hole, I believe this is also related to the fact I am using OpenStack Image API v1, which stores the information for image properties metadata in the header, and that has a limit of 255 character. This leads me to believe that the database constraint was implemented on top of an API limitation.

To fix this we can purpose two separate solutions.
1.) Write a migration script to fix the tables that store the same instance_system_metadata properties in glance.image_properties as nova.instance_system_metadata, that way the tables will be consistent from a size limit perspective since they store the same data
2.) Implement OpenStack's Image API v2 to include create() and update() in glanceclient so image metadata properties are not passed around in the headers but in the body of the request:
https://github.com/openstack/python-glanceclient/blob/master/glanceclient/v2/images.py
This work already contains blueprints and has been partially implemented by Brian Waldon:
https://blueprints.launchpad.net/glance/+spec/api-2

I figured these details should be noted since both the database and API implementation are connected. Thoughts?

tags: added: api v2
Revision history for this message
Lance Bragstad (lbragstad) wrote :

To elaborate a little more on #1 from above, the database fix. We could change the nova.instance_system_metadata table to be consistent with glance.image_properties since Glance holds the larger value of the two. The nova.instance_system_metadata value field is of type VARCHAR(255), because of the API header limit in v1. The glance.image_properties value field is of type TEXT, which allows up to ~64K to be stored. Since that is the larger of the two it would make sense to make Nova database of the same type.

We have some time to think about this fix, since the real issue here is the API implementation. Once that is fixed, more focus can be directed to the database fix.

Sean Dague (sdague)
tags: added: glancev2
removed: database nova snapshot v2
Changed in nova:
assignee: nobody → Anseela M M (anseela-m00)
Sujitha (sujitha-neti)
Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: Anseela M M (anseela-m00) → nobody
Changed in nova:
status: In Progress → Confirmed
STEW TY (stewie925)
Changed in nova:
assignee: nobody → STEW TY (stewie925)
Changed in nova:
status: Confirmed → In Progress
STEW TY (stewie925)
Changed in nova:
assignee: STEW TY (stewie925) → nobody
Changed in nova:
status: In Progress → Confirmed
Ameed Ashour (ameeda)
Changed in nova:
assignee: nobody → Ameed Ashour (ameeda)
Ameed Ashour (ameeda)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/526900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.opendev.org/526900
Reason: This has been in merge conflict for some time and there appear to be some open questions around the impact of this on users. As a result, I'm going to abandon this. Please reopen and rebase if you'd still want to proceed with this

Matt Riedemann (mriedem)
Changed in nova:
status: In Progress → Confirmed
assignee: Ameed Ashour (ameeda) → nobody
importance: Medium → Low
Changed in nova:
assignee: nobody → Ameed Ashour (ameeda)
status: Confirmed → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.