Cannot store files greater than 5GB in Swift

add a size check on the client side and pass it in as an optional parameter to upload?

Shouldn't there be a content-length header in the post?

Chunked-transfer encoding means that the content-length is not known and chunks of data are read until a \0\r\n is received in the request. Glance (and Swift) do chunked-transfer encoding of the request data so that memory isn't eaten up in the controller (or webob.Request) trying to read the entire image file into memory.

We could do a client-side size check, yes. I'm thinking through how to do this without scrapping chunked-transfer encoding....

> Shouldn't there be a content-length header in the post?

In the case of a chunk-transfer-encoded request (the norm), we won't have a Content-Length up-front... in fact, we won't know until the upload finishes how large it was.

Jay:

I think there is a 3rd option here. It would be to *always* use the Large-Upload mechanism if we don't have the Content-Length present in the request.

We would then stream the data into swift using chunk-transfer-encoding, each time we hit th 5GB boundary, we would write the CTE terminator, end the request, and start a new request for the next chunk. We would keep doing this until we're finished the request body has been completely consumed.

When all of the chunks have been fully uploaded, we would then make one more request to swift to write out the manifest for the chunks we just uploaded. (An optimization here would be to only write the manifest if more than one chunk would uploaded).

(We do this transparent streaming/chunking into 5GB segments in the Iback code already).

Reviewed: https://review.openstack.org/311
Committed: http://github.com/openstack/glance/commit/501b14bf94c7b24361a4bd424e4ada4c89df7753
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 501b14bf94c7b24361a4bd424e4ada4c89df7753
Author: Jay Pipes <email address hidden>
Date: Mon Aug 22 14:43:27 2011 -0400

    Fixes LP Bug #827660 - Swift driver fail 5G upload

    Fixes LP Bug #827660 - Swift driver fails to
    upload image files greater than 5GB. The swift
    store driver was not using the large object
    manifest methods required to upload objects
    greater than 5GB into Swift.

    Adds configuration options to specify the
    size in MB that an image file should be considered
    for large object manifest, and a configurable
    chunk size. We use a disk buffer of this size,
    adding chunks of the image file into Swift and
    then after all chunks are saved, add the manifest
    to Swift.

    Change-Id: I4b4421aa1bba584ab242016a041e864ef19d0214

Reviewed: https://review.openstack.org/346
Committed: http://github.com/openstack/glance/commit/1cee671ce1bfacbd1ac07675dede8f46aae4a8a1
Submitter: Jenkins
Branch: milestone-proposed

 status fixcommitted
 done

commit 1cee671ce1bfacbd1ac07675dede8f46aae4a8a1
Author: Jay Pipes <email address hidden>
Date: Mon Aug 22 14:43:27 2011 -0400

    Fixes LP Bug #827660 - Swift driver fail 5G upload

    Fixes LP Bug #827660 - Swift driver fails to
    upload image files greater than 5GB. The swift
    store driver was not using the large object
    manifest methods required to upload objects
    greater than 5GB into Swift.

    Adds configuration options to specify the
    size in MB that an image file should be considered
    for large object manifest, and a configurable
    chunk size. We use a disk buffer of this size,
    adding chunks of the image file into Swift and
    then after all chunks are saved, add the manifest
    to Swift.

    Change-Id: I4b4421aa1bba584ab242016a041e864ef19d0214