versioned_writes middleware is mis-placed in proxy pipeline

Bug #1537042 reported by Alistair Coles
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Alistair Coles

Bug Description

Updated description:
When versioned_writes is not explicitly configured in the proxy pipeline and *LO middlewares *are* configured, then the versioned_writes middleware gets inserted to the left of DLO, which is incorrect and causes the following incorrect behaviour to manifest...

Original description:
When an SLO manifest A in a versioned container is overwritten by object B, the manifest A is resolved and the assembled large object is copied to the versions container. The original manifest is lost. Then when the object B in the versioned container is deleted, it is replaced with the large object, not the original manifest.

To reproduce:

#create containers, segment and manifest...
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_versions" -X PUT
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_segments" -X PUT
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1" -X PUT -H 'X-Versions-Location: c1_versions'
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_segments/4MB_file" -X PUT -T 4MB_file
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1/slo_manifest?multipart-manifest=put" -X PUT -T slo_manifest

# sanity check, the versions container is empty and the manifest is a manifest...
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_versions?format=json"
[]%
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1/slo_manifest?multipart_manifest=get" -I
HTTP/1.1 200 OK
Content-Length: 4194304
Last-Modified: Fri, 22 Jan 2016 12:16:41 GMT
Connection: close
Etag: "a42db41ea9e4a932e67a57ece7cf361b"
X-Timestamp: 1453465000.92552
Date: Fri, 22 Jan 2016 12:17:56 GMT
Content-Type: application/octet-stream
X-Static-Large-Object: True
X-Trans-Id: tx4fb306cefd4b4ec089ccf-0056a21df4

# now overwrite the manifest, it doesn't matter what the content is...
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1/slo_manifest?multipart-manifest=put" -X PUT -T slo_manifest

# oops, now the versions container has 4MB file
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_versions?format=json"[{"hash": "b5cfa9d6c8febd618f91ac2843d50a1c", "last_modified": "2016-01-22T12:29:45.369080", "bytes": 4194304, "name": "00cslo_manifest/1453465138.03144", "content_type": "application/octet-stream"}]%

# and its not just the listing telling us that when really its a manifest, it really is not a manifest...
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1_versions/00cslo_manifest/1453465138.03144?multipart_manifest=get" -I
HTTP/1.1 200 OK
Content-Length: 4194304
Accept-Ranges: bytes
Last-Modified: Fri, 22 Jan 2016 12:29:46 GMT
Etag: b5cfa9d6c8febd618f91ac2843d50a1c
X-Timestamp: 1453465785.36908
Content-Type: application/octet-stream
X-Trans-Id: tx26f122fcaa2049b49e9d7-0056a220ff
Date: Fri, 22 Jan 2016 12:30:55 GMT

# now delete the object in the versioned container...
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1/slo_manifest" -X DELETE

# and oops, we now have alare object, not the manifest, copied back
% curl -H 'X-Auth-Token: AUTH_tk5a7c8aa2656847769335c3809ab62c6b' "http://localhost:8080/v1/AUTH_test/c1/slo_manifest?multipart_manifest=get" -I
HTTP/1.1 200 OK
Content-Length: 4194304
Accept-Ranges: bytes
Last-Modified: Fri, 22 Jan 2016 12:31:09 GMT
Etag: b5cfa9d6c8febd618f91ac2843d50a1c
X-Timestamp: 1453465868.72903
Content-Type: application/octet-stream
X-Trans-Id: txd37023de79194f558902e-0056a22116
Date: Fri, 22 Jan 2016 12:31:18 GMT

CVE References

Revision history for this message
Alistair Coles (alistair-coles) wrote :

This is different behaviour than reported here https://bugs.launchpad.net/swift/+bug/1365862. I cannot reproduce bug 1365862.

Revision history for this message
Alistair Coles (alistair-coles) wrote :

I have a func test patch that also reveals this bug: https://gist.github.com/alistairncoles/32a3f6674abfa1cbd6e5

Revision history for this message
Alistair Coles (alistair-coles) wrote :
Changed in swift:
assignee: nobody → Janie Richling (jrichli)
Changed in swift:
status: New → In Progress
Revision history for this message
Alistair Coles (alistair-coles) wrote :

This bug only manifests when the versioned_writes middleware is inserted in the pipeline ahead (to left of) SLO. That can happen if the middleware is not explicitly configured but is auto-inserted incorrectly to left of SLO.

This bug does not manifest when versioned_writes is placed according to the proxy-server.conf sample i.e. after SLO. In that case this other bug manifests: https://bugs.launchpad.net/swift/+bug/1365862.

summary: - SLO manifests become the large object when versioned
+ versioned_writes middleware is mis-placed in proxy pipeline
description: updated
Changed in swift:
assignee: Janie Richling (jrichli) → Alistair Coles (alistair-coles)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/273073

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift (master)

Change abandoned by Janie Richling (<email address hidden>) on branch: master
Review: https://review.openstack.org/271883
Reason: In light of what the root problem was, abandoning in favor of https://review.openstack.org/#/c/272724/3

Revision history for this message
Janie Richling (jrichli) wrote :

To be clear, both https://review.openstack.org/271883 and https://review.openstack.org/#/c/272724 fix the original title for this "SLO manifests become the large object when versioned". But neither of those changes address the pipeline ordering issue.

Revision history for this message
Alistair Coles (alistair-coles) wrote :

For the record, the cause of the original description of this bug is:

- versioned_writes gets misplaced in pipeline to left of SLO

- during PUT handling, versioned writes creates a COPY req *with new environ* for the copy of the version to the versions container

- SLO inserts its copy_hook into the new req environ :(

- when obj controller calls back to the copy_hook, SLO performs segment assembly

- so the manifest version becomes the whole large object

When versioned_writes is correctly placed to the right of SLO in the pipeline:

- during handling a PUT request, SLO inserts its copy hook in the PUT req environ

- versioned writes creates a COPY req for the version copy *with a new req environ* and the slo copy_hook is NOT copied to the copy req environ

- during obj controller copy handling, no copy hook is found so manifest object gets copied without any intervention form SLO :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/273073
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=30d74af6534b754e6ad9bfdcbff4ec494277ca83
Submitter: Jenkins
Branch: master

commit 30d74af6534b754e6ad9bfdcbff4ec494277ca83
Author: Alistair Coles <email address hidden>
Date: Wed Jan 27 13:50:57 2016 +0000

    Insert versioned_writes in correct pipeline position

    If not explicitly configured the versioned_writes middleware
    should be auto-inserted in the pipeline after slo and dlo, which
    is where the versioned_writes filter section's comments say it
    should be in proxy-server.conf-sample. At the moment it can end up
    being placed ahead of slo and dlo if they have been explicitly
    configured, which results in the linked bug manifesting.

    Closes-Bug: #1537042
    Change-Id: I6ac95a331f4ef0d4887311940acc6f8bc00fb4eb

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/crypto)

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/278283

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/crypto)
Download full text (17.2 KiB)

Reviewed: https://review.openstack.org/278283
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=adbda24e41dbd8b96d80a54be89513d17da3114e
Submitter: Jenkins
Branch: feature/crypto

commit 87fc21c7cfcce2a3e23a84fddcfd1309cd884716
Author: Alistair Coles <email address hidden>
Date: Mon Feb 8 15:31:42 2016 +0000

    Speed up functional testing

    test/functional/tests.py:TestObjectVersioningUTF8 does not clean
    up the versions files it creates because the class's multiple
    inheritance is such that it does not call the tearDown method in
    TestObjectVersioning.

    As a result, any attempt to clean up account containers wastes
    time retrying container delete requests. This occurs either in
    the setUp for TestSloEnv, if the TestSlo class is included in a
    test run, or in the tests.py package tearDown method.

    On the author's dev machine this patch reduces the execution
    time of functional tests in tests.py by approx 30% or 1 minute.

    Change-Id: I8194672bf2ca82435df5868720b6a55a79b94413

commit b173995666b026def3f6558d9a8c972b75449323
Author: Kota Tsuyuzaki <email address hidden>
Date: Thu Feb 4 05:18:42 2016 -0800

    Fix missing Accept-Ranges

    Since commit 4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10, the response
    header for GET EC object doesn't include the Accept-Ranges header.

    This patch fixes it and also adds a few unittests to prevent regression.

    Closes-Bug: #1542168

    Change-Id: Ibafe56ac87b14bc0028953e620a653cd68dd3f84

commit ae632abbd802228ef98a23c98ee49988e7e5b942
Author: Ondřej Nový <email address hidden>
Date: Tue Feb 2 21:58:52 2016 +0100

    Fixed manpages errors.

    account-server.conf.5
    105: warning: numeric expression expected (got `)')

    container-server.conf.5
    111: warning: numeric expression expected (got `)')

    object-expirer.conf.5
    79: warning: numeric expression expected (got `)')

    object-server.conf.5
    114: warning: numeric expression expected (got `)')

    proxy-server.conf.5
    121: warning: numeric expression expected (got `)')
    331: warning: numeric expression expected (got `[')
    1005: warning: macro `*' not defined

    Change-Id: I203dcfde83035e3b1dcb91109b72b5d08bb7840e

commit e47aaaacf15e8f0c7226400541a97238447a7f56
Author: Tim Burke <email address hidden>
Date: Wed Feb 3 12:52:29 2016 -0800

    Stop nesting functions unnecessarily

    Change-Id: Iff120d0bac8a075c37bbddcd2bb0fe85145f1749

commit 26327e1e8b1d37faa764ec586f5bee0e1560eea2
Author: Darrell Bishop <email address hidden>
Date: Thu Jan 21 11:18:18 2016 -0800

    Allow IPv6 addresses/hostnames in StatsD target

    The log_statsd_host value can now be an IPv6 address or a hostname
    which only resolves to an IPv6 address. In both cases, the new
    behavior is to use an AF_INET6 socket on which .sendto() is called
    with the originally-configured hostname (or IP). This means the
    Swift process is not caching a DNS resolution for the lifetime of
    the process (a good thing).

    If a hostname resolves to both an IPv6 or IPv4 address, an AF_IN...

tags: added: in-feature-crypto
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/hummingbird)

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/290148

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/hummingbird)
Download full text (71.7 KiB)

Reviewed: https://review.openstack.org/290148
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0f7f1de233919a0b046349a3e31ae7fc8675a1c5
Submitter: Jenkins
Branch: feature/hummingbird

commit d6b4587a554b51ba733b151e0d924735b63d07e0
Author: Olga Saprycheva <email address hidden>
Date: Tue Mar 8 10:57:56 2016 -0600

    Removed redundant file for flake8 check

    Change-Id: I4322978aa20ee731391f7709bbd79dee140fc703

commit 643dbce134140530eef2ae62c42fef1107f905ed
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Mar 8 06:35:49 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I96b8ff1287bf219c5f8d56a3a4868c1063a953f9

commit 83713d37f0331c5ce9d377f4b4e8724551ae30ca
Author: Daisuke Morita <email address hidden>
Date: Mon Mar 7 18:30:47 2016 -0800

    Missing comments for storage policy parameter

    There are missing comments about storege_policy_index so appropriate
    comments are added.

    Change-Id: I3de3f0e6864e65918ca1a13cce70f19c23d295f5

commit 2cff2dec3d1c4588f5103e39679c43b3dded6dcb
Author: Olga Saprycheva <email address hidden>
Date: Fri Mar 4 15:19:39 2016 -0600

    Fixed pep8 and flake8 errors in doc/source/conf.py and updated flake8 commands in tox.ini to test it.

    Change-Id: I2add370e4cfb55d1388e3a8b41f688a7f3f2c621

commit 043fbca6d08648baa314ea2236f1ccdca8785f16
Author: Christian Schwede <email address hidden>
Date: Fri Mar 4 09:33:17 2016 +0000

    Remove Erasure Coding beta status from docs

    This removes notes stating support for Erasure coding as beta. Questions
    regarding the stability of EC are coming up regularly, and are often referring
    to the docs that state EC as still in beta.

    Besides this, a note marking statsd support as beta has been removed as well.

    Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15

commit 09c73b86e9255f28fbd4cf571a52c17d549a8f9a
Author: Pete Zaitcev <email address hidden>
Date: Thu Mar 3 10:24:28 2016 -0700

    Fix a crash in exception printout

    Says the number of arguments does not match the number of '%'.

    Change-Id: I8b5e395a07328fb9d4ac7a19f8ed2ae1637bee3b

commit fad5fabe0a22e8a86635a66523dd3d3d3b1fa705
Author: Tim Burke <email address hidden>
Date: Thu Mar 3 15:07:08 2016 +0000

    During functional tests, 404 response to a DELETE is successful

    Previously, we would only consider 204 responses successful, which would
    cause some spurious gate failures, such as

    http://logs.openstack.org/66/287666/3/check/gate-swift-dsvm-functional/c6d2673/console.html#_2016-03-03_13_41_07_846

    Change-Id: Ic8c300647924352a297a2781b50064f7657038b4

commit e91de49d6864b3794f8dc5acd9c1bf0c2f7409d1
Author: Alistair Coles <email address hidden>
Date: Mon Aug 10 10:30:10 2015 -0500

    Update container on fast-POST

    This patch makes a number of changes to enable content-type
    metadata to be updated when using the fast-POST mode of
    operation, as proposed in the associated spec ...

tags: added: in-feature-hummingbird
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/swift 2.7.0

This issue was fixed in the openstack/swift 2.7.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/swift 2.7.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.