[SRU] Multipart base64 file decoding of fails with large files - greater than 47406

This looks similar to https://code.djangoproject.com/ticket/19036, but that bug should be fixed django 1.6.

At 47407 the content length for the message rolls over to > 64k.

ubuntu@trusty-maas6:~$ dd if=/dev/urandom of=blah bs=47406 count=1
1+0 records in
1+0 records out
47406 bytes (47 kB) copied, 0.0118549 s, 4.0 MB/s
ubuntu@trusty-maas6:~$ maas local boot-resources create name=testing/testing architecture=amd64/generic content@=/home/ubuntu/blah
Content-Length 65536

ubuntu@trusty-maas6:~$ dd if=/dev/urandom of=blah bs=47407 count=1
1+0 records in
1+0 records out
47407 bytes (47 kB) copied, 0.0116958 s, 4.1 MB/s
ubuntu@trusty-maas6:~$ maas local boot-resources create name=testing/testing architecture=amd64/generic content@=/home/ubuntu/blah
Content-Length 65540

There's a 64k limit on RPC calls. Could that be related?
On 30 Aug 2014 08:05, "Jason Hobbs" <email address hidden> wrote:

> At 47407 the content length for the message rolls over to > 64k.
>
> ubuntu@trusty-maas6:~$ dd if=/dev/urandom of=blah bs=47406 count=1
> 1+0 records in
> 1+0 records out
> 47406 bytes (47 kB) copied, 0.0118549 s, 4.0 MB/s
> ubuntu@trusty-maas6:~$ maas local boot-resources create
> name=testing/testing architecture=amd64/generic content@=/home/ubuntu/blah
> Content-Length 65536
>
> ubuntu@trusty-maas6:~$ dd if=/dev/urandom of=blah bs=47407 count=1
> 1+0 records in
> 1+0 records out
> 47407 bytes (47 kB) copied, 0.0116958 s, 4.1 MB/s
> ubuntu@trusty-maas6:~$ maas local boot-resources create
> name=testing/testing architecture=amd64/generic content@=/home/ubuntu/blah
> Content-Length 65540
>
> --
> You received this bug notification because you are subscribed to MAAS.
> Matching subscriptions: Triage
> https://bugs.launchpad.net/bugs/1363348
>
> Title:
> Image uploads from the cli fail when file size is greater than 47406
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1363348/+subscriptions
>

This seems to have something to do with the chunking going on in django's MultiPartParser.

End of size 47401 file in encode_multipart_message() from apiclient/multipart.py:
H7hV/Mz+YkfNaWZXLzm3vMxacOkNVUpu4YuYjaqBLkh/QseMiwrvUMucKj56CRVxLGWrvFYHcdud
uf6eyh9ZgpCd1ulrsEWvgSRog2vgVKnoNeRLgHhw1zed10B1HD4KBUcz6LRM3cNLUMiafmSvPivS
6w8uTpoNUSuHrGNbmnqA1/6BtJLhxRXbI37vr1AtUY7+5w==

End of size 47401 file in MultiPartParser.parse() in django/http/multipartparser.py:
H7hV/Mz+YkfNaWZXLzm3vMxacOkNVUpu4YuYjaqBLkh/QseMiwrvUMucKj56CRVxLGWrvFYHcdud
uf6eyh9ZgpCd1ulrsEWvgSRog2vgVKnoNeRLgHhw1zed10B1HD4KBUcz6LRM3cNLUMiafmSvPivS
6w8uTpoNUSuHrGNbmnqA1/6BtJLhxRXbI37vr1AtUY7+5w==

End of size 47407 file in encode_multipart_message() from apiclient/multipart.py:
q2QDeL08zPyp7W3di+Djr+kdoKSbIiuI2JCFEs0lNJsgmo+2KHwrrIdTdTZfVAgNP9Esgw96idNx
a8QGC8pj6iJiRhykTQVPx4/0iVetiE7zX5DrF8iKFW5h8f0vuYC8gnAxDekLkih7hWPrf6n9aBZl
ZD7rmLJ+3n6s1k9tSQFfJqcpzMZPAqcaBA5Xt17h4v1gq2RyVQz0NA==

End of size 47401 file in MultiPartParser.parse() in django/http/multipartparser.py:
q2QDeL08zPyp7W3di+Djr+kdoKSbIiuI2JCFEs0lNJsgmo+2KHwrrIdTdTZfVAgNP9Esgw96idNx
a8QGC8pj6iJiRhykTQVPx4/0iVetiE7zX5DrF8iKFW5h8f0vuYC8gnAxDekLkih7hWPrf6n9aBZl
ZD7rmLJ+3n6s1k9tSQFfJqcpzMZPAqcaBA5Xt17h4v1gq2RyVQ

I can increase the maximum size I can upload by increasing the chunk size in django/http/multipartparser.py. This let's me upload files with content length up to 128kb:

# self._chunk_size = min([2**31-4] + possible_sizes)
        self._chunk_size = 2**17

The error occurs when the content length exceeds the chunk size.

By editing django/http/multipartparser.py to force a chunk size of 2**31-4, I was able to upload a full sized image successfully. Time for bed.

Django's multipartparser splits incoming data into chunks to process it. On my system, these chunk sizes are 64kb at largest, but are always a multiple of 4 when decoding base64 data, since base64 encoded data must be split at 4 byte boundaries. django is taking the len() of the string of data and making sure that's a multiple of 4, and passing the string to b64decode to decode.

I believe the problem is the data apiclient uploads includes CRLF characters. Django counts those characters as a part of the string when considering the length of a chunk of base64 data, but the CRLF characters aren't significant in base64, so the resulting pure base64 data, with whitespace stripped, isn't necessarily aligned to a 4 byte boundary.

Or, the problem may be that django is counting the whitespace characters as part of the length of the chunk when it shouldn't be. I patched django to strip whitespace and things work fine.

The RFC for content transfer encoding says the whitespace should be there. https://www.ietf.org/rfc/rfc2045.txt

Attached the patch for fixing this in django.

I filed a bug and submitted a patch to fix this to the django upstream.

Bug: https://code.djangoproject.com/ticket/23397
Pull request: https://github.com/django/django/pull/3151/files

The pull request linked above also includes a unit test to reproduce the django issue, which is much easier than setting up MAAS to reproduce the django issue.

The attachment "lp-1363348.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Status changed to 'Confirmed' because the bug affects multiple users.

I hit this issue, adding the CentOS image. I hand patched the file, rm'd the .pyc, restarted Apache2, and then reran the command. It looks right now. I can confirm the fix. Thanks.

This bug was fixed in the package python-django - 1.6.6-1ubuntu2

---------------
python-django (1.6.6-1ubuntu2) utopic; urgency=medium

  * debian/patches/fix_test_encoding.patch: Fix test encoding headers,
    otherwise it FTBFS.
 -- Andres Rodriguez <email address hidden> Thu, 18 Sep 2014 19:01:13 -0500

I rejected the upload in trusty because there was an extra '99_fix_test.patch' patch added without explanation. Can you either remove this file and re-upload or explain this patch in the upload. Thanks!
--chris

Hello Jason, or anyone else affected,

Accepted python-django into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-django/1.6.1-2ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

We have tested this and confirmed this works as expect!

This bug was fixed in the package python-django - 1.6.1-2ubuntu0.5

---------------
python-django (1.6.1-2ubuntu0.5) trusty-proposed; urgency=medium

  * debian/patches/99_fix_multipart_base64_decoding_large_files.patch:
    Fix Multipart base64 file decoding with large files ensuring that the
    actual base64 content has a length a multiple of 4. (LP: #1363348)
  * debian/patches/
 -- Andres Rodriguez <email address hidden> Thu, 18 Sep 2014 17:46:45 -0500

The verification of the Stable Release Update for python-django has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.