OpenStack Object Storage (swift)

Percent signs in object names cause trouble for versioned_writes

Bug #1755554 reported by Tim Burke
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
High
Kota Tsuyuzaki

Bug Description

This is likely related to what goes wrong in https://bugs.launchpad.net/swift/+bug/1598093 and https://bugs.launchpad.net/swift/+bug/1678022

In a normal container, you can create objects named '%ff' just fine by encoding the percent sign as %25 in the request:

 $ curl http://saio:8090/v1/AUTH_test/c1 -X PUT

 $ curl -i http://saio:8090/v1/AUTH_test/c1/%25ff -X PUT -d 1234
 HTTP/1.1 201 Created
 Last-Modified: Tue, 13 Mar 2018 17:25:27 GMT
 Content-Length: 0
 Etag: 81dc9bdb52d04dc20036dbd8313ed055
 Content-Type: text/html; charset=UTF-8
 X-Trans-Id: tx6718c67d32364c96b1d59-005aa80986
 X-Openstack-Request-Id: tx6718c67d32364c96b1d59-005aa80986
 Date: Tue, 13 Mar 2018 17:25:26 GMT

But, if you try to do that with a versioned container, it fails with a 412:

 $ curl http://saio:8090/v1/AUTH_test/c2 -X PUT -H x-history-location:v

 $ curl http://saio:8090/v1/AUTH_test/v -X PUT

 $ curl http://saio:8090/v1/AUTH_test/c2/%25ff -X PUT -d 1234 -i
 HTTP/1.1 412 Precondition Failed
 Content-Length: 92
 Content-Type: text/html; charset=UTF-8
 X-Trans-Id: tx0e46490a8a50457a8cdba-005aa80a94
 X-Openstack-Request-Id: tx0e46490a8a50457a8cdba-005aa80a94
 Date: Tue, 13 Mar 2018 17:29:56 GMT

 <html><h1>Precondition Failed</h1><p>A precondition for this request was not met.</p></html>

Logs say

 Mar 13 17:29:56 saio proxy-server: - - 13/Mar/2018/17/29/56 GET /v1/AUTH_test/c2/%25FF HTTP/1.0 499 - Swift - - 29 - tx0e46490a8a50457a8cdba-005aa80a94 - 0.0003 VW - 1520962196.922559977 1520962196.922908068 -
 Mar 13 17:29:56 saio proxy-server: 127.0.0.1 127.0.0.1 13/Mar/2018/17/29/56 PUT /v1/AUTH_test/c2/%2525ff HTTP/1.0 412 - curl/7.35.0 - - 92 - tx0e46490a8a50457a8cdba-005aa80a94 - 0.0018 - - 1520962196.921812057 1520962196.923567057 -
 Mar 13 17:29:56 saio proxy-server: STDERR: 127.0.0.1 - - [13/Mar/2018 17:29:56] "PUT /v1/AUTH_test/c2/%25ff HTTP/1.1" 412 333 0.003989 (txn: tx0e46490a8a50457a8cdba-005aa80a94)

...which is particular strange in that there are *three* distinct object names recorded: %25FF, %2525ff, and %25ff

Adding another layer of encoding lets you PUT:

 $ curl http://saio:8090/v1/AUTH_test/c2/%2525ff -X PUT -d 1234 -i
 HTTP/1.1 201 Created
 Last-Modified: Tue, 13 Mar 2018 17:34:16 GMT
 Content-Length: 0
 Etag: 81dc9bdb52d04dc20036dbd8313ed055
 Content-Type: text/html; charset=UTF-8
 X-Trans-Id: tx06e6e8fe0b7f4fa0ada9b-005aa80b97
 X-Openstack-Request-Id: tx06e6e8fe0b7f4fa0ada9b-005aa80b97
 Date: Tue, 13 Mar 2018 17:34:15 GMT

...but it doesn't have the same name:

 $ curl http://saio:8090/v1/AUTH_test/c2
 %25ff

 $ curl http://saio:8090/v1/AUTH_test/c1
 %ff

Worse, versioning doesn't work for the object!

 $ curl http://saio:8090/v1/AUTH_test/c2/%2525ff -X PUT -d 5678 -i
 HTTP/1.1 201 Created
 Last-Modified: Tue, 13 Mar 2018 17:35:07 GMT
 Content-Length: 0
 Etag: 674f3c2c1a8a6f90461e8a66fb5550ba
 Content-Type: text/html; charset=UTF-8
 X-Trans-Id: txbe9856f0a8954beabc757-005aa80bca
 X-Openstack-Request-Id: txbe9856f0a8954beabc757-005aa80bca
 Date: Tue, 13 Mar 2018 17:35:06 GMT

 $ curl http://saio:8090/v1/AUTH_test/v -i
 HTTP/1.1 204 No Content
 X-Container-Object-Count: 0
 Accept-Ranges: bytes
 X-Storage-Policy: default
 Last-Modified: Tue, 13 Mar 2018 17:29:11 GMT
 X-Container-Bytes-Used: 0
 X-Timestamp: 1520961447.09430
 Content-Type: text/plain; charset=utf-8
 Content-Length: 0
 X-Trans-Id: tx258340276e704cea9442f-005aa80bcd
 X-Openstack-Request-Id: tx258340276e704cea9442f-005aa80bcd
 Date: Tue, 13 Mar 2018 17:35:09 GMT

John Dickinson (notmyname)
Changed in swift:
importance: Undecided → High
Revision history for this message
Kota Tsuyuzaki (tsuyuzaki-kota) wrote :

OK, I confirmed this behavior exists with the current master. The one more odd thing is the successful PUT request with %%25ff was also logged as 409 response in the proxy-server but no any errors exist under the transaction id.

Mar 22 06:46:08 saio proxy-server: User: test uses token AUTH_tk156bef95123d40fe800845b539418eaa (trans_id txf6382b861f04463083921-005ab35130)
Mar 22 06:46:08 saio proxy-server: - - 22/Mar/2018/06/46/08 HEAD /v1/AUTH_test%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf6382b861f04463083921-005ab35130 - 0.0106 GET_ACCOUNT_INFO - 1521701168.470243931 1521701168.480846882 -
Mar 22 06:46:08 saio proxy-server: - - 22/Mar/2018/06/46/08 HEAD /v1/AUTH_test/c2%3Fformat%3Djson HTTP/1.0 204 - Swift - - - - txf6382b861f04463083921-005ab35130 - 0.0198 GET_CONTAINER_INFO - 1521701168.481514931 1521701168.501288891 0
Mar 22 06:46:08 saio proxy-server: User test:tester has admin authorizing. (txn: txf6382b861f04463083921-005ab35130)
Mar 22 06:46:08 saio proxy-server: - - 22/Mar/2018/06/46/08 GET /v1/AUTH_test/c2/%2525ff HTTP/1.0 499 - Swift - - 70 - txf6382b861f04463083921-005ab35130 - 0.0245 VW - 1521701168.505810976 1521701168.530276060 0
Mar 22 06:46:08 saio proxy-server: User test:tester has admin authorizing. (txn: txf6382b861f04463083921-005ab35130) (client_ip: 127.0.0.1)
Mar 22 06:46:08 saio proxy-server: 127.0.0.1 127.0.0.1 22/Mar/2018/06/46/08 PUT /v1/AUTH_test/c2/%252525ff HTTP/1.0 201 - curl/7.47.0 AUTH_tk156bef951... 4 - - txf6382b861f04463083921-005ab35130 - 0.0935 - - 1521701168.469007015 1521701168.562541962 0
Mar 22 06:46:08 saio proxy-server: STDERR: 127.0.0.1 - - [22/Mar/2018 06:46:08] "PUT /v1/AUTH_test/c2/%2525ff HTTP/1.1" 201 314 0.096156 (txn: txf6382b861f04463083921-005ab35130)

I'll dive into deeper what pass was taken in the versions_writes middleware...

Kota Tsuyuzaki (tsuyuzaki-kota)
Changed in swift:
assignee: nobody → Kota Tsuyuzaki (tsuyuzaki-kota)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/555245

Changed in swift:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/570436

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/555245
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0e3e7b9b0953baaf7a0686881fd2348fde7d1e59
Submitter: Zuul
Branch: master

commit 0e3e7b9b0953baaf7a0686881fd2348fde7d1e59
Author: Kota Tsuyuzaki <email address hidden>
Date: Thu Mar 22 19:26:24 2018 +0900

    Fix versioned writes error with url-encoded object name

    With url encoded object name like '%25ff' that can be url-encoded
    value after decoded can cause 412 Precondition Failed. And more,
    that can do nothing (no versioned object creation) even it returns
    a successful response.

    The root causes are in versioned_writes middleware as follows:

    A. unnecessary unquote in object_request method
    B. incorrect use of make_pre_authed_request that takes 'quoted'
       path in the args. That is described at [1] explicitely.

    This patch resolved those 2 bugs at once, and then, now we can create
    %25ff versioned object reported in the launchpad with this patch.

    Perhaps, more tests would be nice to have. This patch added a few
    test cases on that.

    1: https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1174

    Note that make_subrequest and its caller should have *quoted* path but
    make_env should *NOT*. That might make us confused.

    Closes-Bug: #1755554

    Change-Id: Ibcd90cc633c68973929ee5249c6598c22b342e3e

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.openstack.org/570436
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fbb8d7ebb52038fe445e2ca7e5c0459e8ff5ac5e
Submitter: Zuul
Branch: master

commit fbb8d7ebb52038fe445e2ca7e5c0459e8ff5ac5e
Author: Tim Burke <email address hidden>
Date: Thu May 24 13:00:58 2018 -0700

    Clarify that archive location headers should be URL-encoded

    Fix up function tests to actually *do* that quoting, and fix
    _listing_pages_iter to respect that.

    Change-Id: I1554042510819ea878b4c70417721944115e17f4
    Related-Bug: 1229142
    Related-Change: I425440f76b8328f8e119d390bfa4c7022181e89e
    Related-Bug: 1755554
    Related-Change: Ibcd90cc633c68973929ee5249c6598c22b342e3e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.18.0

This issue was fixed in the openstack/swift 2.18.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.