Swift segmented object - segments ordered using ASCII sort
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
python-swiftclient |
Invalid
|
Low
|
Unassigned |
Bug Description
Using the guide here:
http://
I have written a segmented object up loader for Mobile devices (smart-phones, etc) that will upload a single file in chunks over a potentially unreliable network - so that if a chunk fails for whatever reason then only that chunk needs to be re-sent instead of the entire file.
It basically does a PUT for each segment (as specified in the guide) of container/
For testing I am uploading a ~ 64K file, using segment named 1,2,3,4,
This works fine - however, when I come to download the file (again, in chunks, using the Range http header), the file is corrupt. The md5 of the downloaded file does not match the md5 of the one that was uploaded.
Using the swift command line to download the whole object results in the same corrupt file (the same md5 as the corrupted chunked download). Listing in swift produces the order 1,10,11,
Now, it seems that the way that Swift is ordering the objects it is using an ASCII sort on the segment name. This may be by design, and I am not necessarily raising a bug against Swift itself . It's just that it is not explicitly mentioned in the docs that Swift will order the segments using an ASCII sort on the segment names.
The guide I used gives an example of uploading 3 segments of 1 byte each, naming the segments by number (just as I have done but with larger segment sizes). It would be better if this guide mentioned that Swift will sort the segments using an ASCII sort on the segment names.
Incidentally, I am now padding the segment numbers with zeros (00000001, 000000002, etc) so the download order is the same as the upload order.
Thanks,
Mark
Changed in swift: | |
importance: | Undecided → Low |
tags: | added: python-swiftclient |
affects: | swift → python-swiftclient |
Changed in python-swiftclient: | |
status: | New → Invalid |
From the python-swiftclient code:
'obj': '%s/%s/%s/%08d' % (obj, put_headers[ 'x-object- meta-mtime' ], full_size, segment)
I.e. this bug does not affect the swift client.
And in docs:
"All the object segments need to be in the same container, have a common object name prefix, and their names sort in the order they should be concatenated."
I.e. this is not a bug in docs either (although you may argue what kind of "sort" is mentioned there).