Adding remote image when cache enabled results in image having size of 0

Actually, I think the approach here is to say images shouldn't ever end up in the db with a size of 0. Maybe we don't need to fix this bug so much as reject remote images without a size... thoughts?

Yeah, images should never get to an 'active' status and have no size. So, I don't necessarily think this bug is cache-related...

So, the bug should be: "image size ends up as 0 for active image"? Do we have a reproducible test case for this?

if you do something like 'glance add location=http://localhost/my_image name=My_Image container_format=ovf disk_format=vhd is_public=true', an image entry will be created with a size of 0. If you try to add a size=0 param to the command, glance will ignore it. We do have a few store-specific functional tests that ensure you can retrieve remote images, but it doesn't verify that the size is non-zero.

Bottom line is that no image should ever be in ACTIVE status with a size of 0. The real bug is tracking down any cases where this might happen...

Marking Critical as this is a data corruption bug at its core.

Solution is to have HTTP store add method return a real image size...

This doesn't just affect the HTTP store. It affects all stores. The key piece of info here is to have cacheing on and to pass in a location rather than passing in image data.

Ah, gotcha. OK, changed description again to address cache.

It looks like we have two inter-related problems here:

1. The image size meta-data is incorrectly set to zero for remote images with no explicit size provided.

2. The image cache uses this bad size to set the Content-Length on the HTTP response to a GET of a cached image.

Issue #1 could be addressed most directly by the size meta-data being populated from the get-go by querying the backend store as to the actual size. For example in the case of the HTTP store, a HEAD request on the image URI would cheaply yield the Content-Length that would be returned on a full GET.

Another option would be to set the size to some initial value indicating unkown (e.g. -1), then lazily reset it to the real value when the image is retrieved for the first time (i.e. either pre-fetched or pulled down in response to a user's GET).

Issue #2 should also be fixed, as the cache would know the actual size of the locally cached image before returning it. In fact the HTTP response currently sent is not well-formed, as the Content-Length is zero-ized but the response body still contains the full image content.

So an initial work-around for the underlying problem could be to ensure the image cache sets the Content-Length to the actual cached file size as opposed to relying on the meta-data.

Thoughts?

If there are no objections to the above, I'll go ahead and put together patchs implementing the proposed changes.

Hi Eoghan!

So, not quite... or at least not quite correct for #1. Basically, this bug is about *remote* images and the image cache. Remote images are images added to Glance that actually do not get stored in a Glance backend. More specifically, if you add an image (via the API or through the CLI tool) and set the Location field to something like "http://example.com/images/my-image", then you are adding a remote image, as opposed to uploading an image to Glance.

It is these remote images that this bug refers to, as the image cache is not setting the image_size correctly for these images.

Hope this makes more sense! Of course, you are free to assign yourself to this bug! The fix is to follow the code paths for remote images and ensure that a correct image size is passed back to the image cache on a GET.

-jay

Hi Jay,

Yep that was my understanding too.

I was probably a bit loose in my terminology, causing the confusion - by the 'backend store' I meant the pre-existing remote HTTP or S3 or Swift image store location identified in the X-Image-Meta-Location URI provided on an image POST (i.e. without any further image content being uploaded to the store).

So I think we are actually on the same page re. the underlying problem. I should have a patch ready for submission soon, which will ensure the Content-Length is properly set on a GET of a cached remote issue.

Cheers,
Eoghan

Submitted for review a patch addressing issue #2:

https://review.openstack.org/3065

Reviewed: https://review.openstack.org/3065
Committed: http://github.com/openstack/glance/commit/6fd005465e70f3ac1c1581b10b4e01d041f50f23
Submitter: Jenkins
Branch: master

commit 6fd005465e70f3ac1c1581b10b4e01d041f50f23
Author: Eoghan Glynn <email address hidden>
Date: Mon Jan 16 14:55:42 2012 +0000

    Set correct Content-Length on cached remote images

    Override the image metadata size attribute with the actual
    cached image file size when setting the Content-Length header
    on the response to a GET of a cache remote image.

    This avoids an non-initial GET on a remote image failing when
    caching is enabled, as described in this bug:

      Bug 900959

    Change-Id: I7656bc971d982cce78ce1ed4cf7ea509f8bbdfa8

Fix proposed to branch: master
Review: https://review.openstack.org/3147

Reviewed: https://review.openstack.org/3147
Committed: http://github.com/openstack/glance/commit/3956c546fb725a13784f6e1441a74be1680abd9d
Submitter: Jenkins
Branch: master

commit 3956c546fb725a13784f6e1441a74be1680abd9d
Author: Eoghan Glynn <email address hidden>
Date: Tue Jan 17 23:21:15 2012 +0000

    Set size metadata correctly for remote images.

    Previously the size metadata was always zero'd for remote images,
    which was misleading and led to issues like:

      Bug 900959

    We now query the remote store (HTTP, S3 or Swift) with the
    equivalent of a HTTP HEAD in order to determine the correct
    image size without downloading.

    Where the size is available, the metadata is set appropriately.
    Otherwise it falls back to zero as before.

    Change-Id: I3093eba1b2fa023348558c45febee39e68e1a08f