Swift segmented object - segments ordered using ASCII sort

Bug #1004436 reported by Mark Collier
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-swiftclient
Invalid
Low
Unassigned

Bug Description

Using the guide here:

http://swift.openstack.org/overview_large_objects.html

I have written a segmented object up loader for Mobile devices (smart-phones, etc) that will upload a single file in chunks over a potentially unreliable network - so that if a chunk fails for whatever reason then only that chunk needs to be re-sent instead of the entire file.

It basically does a PUT for each segment (as specified in the guide) of container/object/segment - where segment is a number between 1 and how ever many segments there are. The default segment size that the Mobile will use is 4k - (4096 bytes) .

For testing I am uploading a ~ 64K file, using segment named 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 and 16

This works fine - however, when I come to download the file (again, in chunks, using the Range http header), the file is corrupt. The md5 of the downloaded file does not match the md5 of the one that was uploaded.

Using the swift command line to download the whole object results in the same corrupt file (the same md5 as the corrupted chunked download). Listing in swift produces the order 1,10,11,12,13,14,15,16,2,3,4,5,6,7,8,9 and inspecting the downloaded file with a hex editor, it seems that segment 10 is placed where segment 2 should be.

Now, it seems that the way that Swift is ordering the objects it is using an ASCII sort on the segment name. This may be by design, and I am not necessarily raising a bug against Swift itself . It's just that it is not explicitly mentioned in the docs that Swift will order the segments using an ASCII sort on the segment names.

The guide I used gives an example of uploading 3 segments of 1 byte each, naming the segments by number (just as I have done but with larger segment sizes). It would be better if this guide mentioned that Swift will sort the segments using an ASCII sort on the segment names.

Incidentally, I am now padding the segment numbers with zeros (00000001, 000000002, etc) so the download order is the same as the upload order.

Thanks,

Mark

Changed in swift:
importance: Undecided → Low
tags: added: python-swiftclient
Thierry Carrez (ttx)
affects: swift → python-swiftclient
Revision history for this message
Constantine Peresypkin (constantine-q) wrote :

From the python-swiftclient code:

'obj': '%s/%s/%s/%08d' % (obj, put_headers['x-object-meta-mtime'], full_size, segment)

I.e. this bug does not affect the swift client.

And in docs:
"All the object segments need to be in the same container, have a common object name prefix, and their names sort in the order they should be concatenated."
I.e. this is not a bug in docs either (although you may argue what kind of "sort" is mentioned there).

Changed in python-swiftclient:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.