Retrieving lists of objects using aws-cli fails with error Incorrect Padding

Bug #1820847 reported by Bhaskar Singhal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
New
Undecided
Unassigned

Bug Description

I uploaded around 2000 files in one bucket to the swift using aws-cli, but when I try to list them I only get 1000 and after that the request fails.

<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>tx480c8bbeab164de4a1e42-005c904602</RequestId><Reason>Incorrect padding</Reason></Error>

The swift-proxy-server logs show:
Incorrect padding: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift3/middleware.py", line 80, in __call__#012 resp = self.handle_request(req)#012 File "/usr/lib/python2.7/dist-packages/swift3/middleware.py", line 107, in handle_request#012 res = handler(req)#012 File "/usr/lib/python2.7/dist-packages/swift3/controllers/bucket.py", line 131, in GET#012 decoded = b64decode(req.params['continuation-token'])#012 File "/usr/lib/python2.7/base64.py", line 96, in standard_b64decode#012 return b64decode(s)#012 File "/usr/lib/python2.7/base64.py", line 78, in b64decode#012 raise TypeError(msg)#012TypeError: Incorrect padding (txn: tx480c8bbeab164de4a1e42-005c904602) (client_ip: 172.30.240.205)

The issue is due to the fact that continuation-token calculated by swift3 is
MjAwMGNvdW50L2ZpbGUuMTg5OQ==

but the continuation-token received by the swift3 in the next request is
MjAwMGNvdW50L2ZpbGUuMTg5OQ%3D%3D

This is happening because continuation-token is sent as query param and "==" gets converted into 0x3d hex.

We need to replace %xx in the continuation-token by using unquote.

Revision history for this message
Tim Burke (1-tim-z) wrote :

I think I see how this could happen for swift3: When doing a ?encoding-type=url query, we used to quote() darn near everything [1] -- so the client would get back a response containing something like

 <NextContinuationToken>MjAwMGNvdW50L2ZpbGUuMTg5OQ%3D%3D</NextContinuationToken>

which the client dutifully encodes to

 ?continuation-token=MjAwMGNvdW50L2ZpbGUuMTg5OQ%253D%253D

so even when eventlet unquote()s the QUERY_STRING per spec, we get the escaped version.

Two bits of good news:
- If we stop quoting the NextContinuationToken, everything will Just Work; the equals-signs aren't themselves a problem.
- We've already done that in s3api [2] -- which explains why I haven't been able to repro the problem on tip-of-master swift.

What version of swift are you using? Can you upgrade to 2.20.0 or later and switch from swift3 to s3api?

[1] https://github.com/openstack/swift3/blob/master/swift3/etree.py#L85-L103
[2] https://github.com/openstack/swift/commit/c1c65a7

Revision history for this message
Bhaskar Singhal (bhaskarsinghal) wrote :

Thanks, Tim.
I am able to reproduce the issue with s3api and swift 2.19 (rocky).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.