Can't do 'Transfer-Encoding: chunked'

Bug #1046384 reported by Kaleb S. KEITHLEY
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Invalid
Undecided
Unassigned

Bug Description

Large objects are never sent using chunked transfers because response.content_length is always set.

FWIW I'm using gluster's version with their plug-ins patched in. I don't believe that affects this issue.

Looking at the rest of the swift code and .../site-packages/eventlet/wsgi.py it's apparent that if response.content_length is not set on large objects the result will be to send using chunked transfer.

This was found in 1.4.8 but looking at 1.6.0 source I can see the bug appears to exist there as well.

With the following change large objects are sent using chunked transfer (verified both with and without using tcpdump)

I chose to keep things simple and do chunk transfers at the disk_chunk_size, but an option in the config file would be okay too.

*** swift/obj/server.py.orig 2012-08-28 14:04:11.896008684 -0400
--- swift/obj/server.py 2012-09-05 10:32:38.788003438 -0400
***************
*** 720,730 ****
                  response.headers[key] = value
          response.etag = file.metadata['ETag']
          response.last_modified = float(file.metadata['X-Timestamp'])
! response.content_length = file_size
! if response.content_length < KEEP_CACHE_SIZE and \
! 'X-Auth-Token' not in request.headers and \
! 'X-Storage-Token' not in request.headers:
! file.keep_cache = True
          if 'Content-Encoding' in file.metadata:
              response.content_encoding = file.metadata['Content-Encoding']
          response.headers['X-Timestamp'] = file.metadata['X-Timestamp']
--- 720,731 ----
                  response.headers[key] = value
          response.etag = file.metadata['ETag']
          response.last_modified = float(file.metadata['X-Timestamp'])
! if file_size <= self.disk_chunk_size:
! response.content_length = file_size
! if response.content_length < KEEP_CACHE_SIZE and \
! 'X-Auth-Token' not in request.headers and \
! 'X-Storage-Token' not in request.headers:
! file.keep_cache = True
          if 'Content-Encoding' in file.metadata:
              response.content_encoding = file.metadata['Content-Encoding']
          response.headers['X-Timestamp'] = file.metadata['X-Timestamp']

Revision history for this message
gholt (gholt) wrote :

I'm a little confused as to what is broken. If the object server always sends with a content-length that is correct, what's the problem? Why does it *need* to use chunked transfer encoding if it knows the content length anyway?

Revision history for this message
Kaleb S. KEITHLEY (kkeithleatredhat) wrote :

It might be better characterized as an enhancement request, but I don't see that as an option anywhere.

Yes, if the size is known it doesn't need to use chunked transfer encoding; we agree on that.

GlusterFS's UFO wants an option to send large objects using chunks "for performance". (Sorry, I'm not in a position to quantify that objectively at this point.)

But it's perfectly acceptable, to me anyway, to reject it. I'm just doing due diligence and pushing upstream. We can carry a patch until the powers-that-be here can be convinced that it isn't accomplishing anything.

Revision history for this message
Kaleb S. KEITHLEY (kkeithleatredhat) wrote :

Withdrawn

Changed in swift:
status: New → Invalid
Revision history for this message
gholt (gholt) wrote :

Ah, okay, thanks for the explanation Kaleb. I can't think of how it would be better to be chunked, but I'm open to learning how; I've certainly been wrong before.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.