[SRU] Failing to create simplestreams bucket on ceph-radosgw

Bug #1346935 reported by Björn Tillenius
42
This bug affects 6 people
Affects Status Importance Assigned to Milestone
simplestreams
Fix Released
Medium
Scott Moser
simplestreams (Ubuntu)
Fix Released
Medium
Unassigned
Trusty
Fix Released
Medium
Felipe Reyes
Vivid
Fix Released
Medium
Felipe Reyes

Bug Description

[Impact]

This bug prevents from using simplestreams bucket creation, so new deployments
that want to benefit from the glance-simplestreams-sync charm to keep
up-to-date images in their private repos is broken.

[Test Case]

1) Deploy an OpenStack cloud
   bzr branch lp:~freyes/+junk/lp1346935
   cd lp1346935
   juju deployer -c default-radosgw.yaml -d -v -s 10 SERIES-kilo # SERIES -> trusty or kilo depending on which package you're verifying
2) Once everything has settled down, run glance-simplestreams-sync.py
   juju ssh glance-simplestreams-sync/0 sudo /var/lib/juju/agents/unit-glance-simplestreams-sync-0/charm/scripts/glance-simplestreams-sync.py
3) Check /var/log/glance-simplestreams-sync.log
   tail -n 16 /var/log/glance-simplestreams-sync.log

4) Log before applying the fix:

ERROR * 09-15 19:15:15 [PID:10596] * root * Exception during do_sync
Traceback (most recent call last):
  File "./glance-simplestreams-sync.py", line 286, in <module>
    do_sync(charm_conf)
  File "./glance-simplestreams-sync.py", line 140, in do_sync
    store = SwiftObjectStore(SWIFT_DATA_DIR)
  File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", line 69, in __init__
    '.r:*,.rlistings'})
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1319, in put_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1243, in _retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 731, in put_container
    http_response_content=body)
ClientException: Container PUT failed: http://10.5.150.54:80/swift/simplestreams 405 Method Not Allowed {"Code":"MethodNotAllowed"}
INFO * 09-15 19:15:15 [PID:10596] * root * sync done.

5) Install the patched package
6) Re-run glance-simplestreams-sync.py (step 2)
7) Check glance-simplestreams-sync.log, log should look like:

DEBUG * 09-15 19:38:22 [PID:10789] * urllib3.connectionpool * "PUT /swift/v1/simplestreams HTTP/1.1" 201 0
DEBUG * 09-15 19:38:22 [PID:10789] * swiftclient * REQ: curl -i http://10.5.150.54:80/swift/v1/simplestreams -X PUT -H "Content-Length: 0" -H "X-Auth-Token: d2bb7d174eb84a4aa09fb27a0d06092f" -H "X-Container-Read: .r:*,.rlistings"
DEBUG * 09-15 19:38:22 [PID:10789] * swiftclient * RESP STATUS: 201 Created
DEBUG * 09-15 19:38:22 [PID:10789] * swiftclient * RESP HEADERS: [('content-length', '0'), ('accept-ranges', 'bytes'), ('keep-alive', 'timeout=5, max=100'), ('server', 'Apache/2.4.10 (Ubuntu)'), ('connection', 'Keep-Alive'), ('date', 'Tue, 15 Sep 2015 19:38:21 GMT'), ('content-type', 'text/plain; charset=utf-8')]

[Regression Potential]

- Minor, this patch is already integrated in Wily[0] and the change only gets rid
  of the logic that removes the version from the endpoint.
- I tested this patch using a real swift endpoint and there is no regression
  detected

[Other Info]

If ceph-radosgw (which provides a Swift API for Ceph) is used instead of
Swift, simplestreams sync fail to create the simplestreams bucket.

I tried using the glance-simplestreams-sync charm, and got the following
error:

INFO * 07-22 14:05:30 [PID:3620] * swiftclient * REQ: curl -i http://10.96.
2.27:80/swift/simplestreams -X PUT -H "Content-Length: 0" -H "X-Auth-Token: a7f0
37d3245a4005834709b99e3e08f3" -H "X-Container-Read: .r:*,.rlistings"
INFO * 07-22 14:05:30 [PID:3620] * swiftclient * RESP STATUS: 405 Method No
t Allowed
INFO * 07-22 14:05:30 [PID:3620] * swiftclient * RESP HEADERS: [('date', 'T
ue, 22 Jul 2014 14:05:30 GMT'), ('accept-ranges', 'bytes'), ('content-type', 'ap
plication/json; charset=utf-8'), ('content-length', '27'), ('server', 'Apache/2.
4.7 (Ubuntu)')]
INFO * 07-22 14:05:30 [PID:3620] * swiftclient * RESP BODY: {"Code":"Method
NotAllowed"}
ERROR * 07-22 14:05:30 [PID:3620] * swiftclient * Container PUT failed: http
://10.96.2.27:80:80/swift/simplestreams 405 Method Not Allowed {"Code":"Method
NotAllowed"}
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1189, in _
retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 674, in pu
t_container
    http_response_content=body)
ClientException: Container PUT failed: http://10.96.2.27:80:80/swift/simplestrea
ms 405 Method Not Allowed {"Code":"MethodNotAllowed"}
ERROR * 07-22 14:05:30 [PID:3620] * root * Exception during do_sync
Traceback (most recent call last):
  File "/etc/cron.daily/glance-simplestreams-sync.py", line 286, in <module>
    do_sync(charm_conf)
  File "/etc/cron.daily/glance-simplestreams-sync.py", line 140, in do_sync
    store = SwiftObjectStore(SWIFT_DATA_DIR)
  File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", l
ine 70, in __init__
    '.r:*,.rlistings'})
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1265, in p
ut_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1189, in _
retry
    rv = func(self.url, self.token, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 674, in pu
t_container
    http_response_content=body)
ClientException: Container PUT failed: http://10.96.2.27:80:80/swift/simplestrea
ms 405 Method Not Allowed {"Code":"MethodNotAllowed"}

Note that it tries to PUT to http://10.96.2.27:80:80/swift/simplestreams, where
it should PUT to http://10.96.2.27:80:80/swift/v1/simplestreams. There seems
to be code in simplestreams that strip the version from the endpoint URL, but
even after I remove that I get a 409 error.

[0] http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/wily/simplestreams/wily/revision/18

tags: added: landscape
tags: added: cloud-installer
Scott Moser (smoser)
Changed in simplestreams:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Scott Moser (smoser) wrote :

Hi,
 I'd like to get this fixed.
 http://paste.ubuntu.com/9465390/

James Page has that patch. I'm apt to just take it.

Revision history for this message
Scott Moser (smoser) wrote :

fixed in revno 362.

Changed in simplestreams:
assignee: nobody → Scott Moser (smoser)
status: Confirmed → Fix Committed
Revision history for this message
Scott Moser (smoser) wrote :

this is fix-released in wily.

Changed in simplestreams (Ubuntu):
status: New → Fix Released
importance: Undecided → Medium
Felipe Reyes (freyes)
tags: added: openstack sts
Felipe Reyes (freyes)
Changed in simplestreams (Ubuntu Trusty):
assignee: nobody → Felipe Reyes (freyes)
Changed in simplestreams (Ubuntu Vivid):
assignee: nobody → Felipe Reyes (freyes)
Changed in simplestreams (Ubuntu Trusty):
importance: Undecided → Medium
Changed in simplestreams (Ubuntu Vivid):
importance: Undecided → Medium
Revision history for this message
Felipe Reyes (freyes) wrote :

Patch for Vivid

description: updated
summary: - Failing to create simplestreams bucket on ceph-radosgw
+ [SRU] Failing to create simplestreams bucket on ceph-radosgw
Revision history for this message
Felipe Reyes (freyes) wrote :

Patch for Trusty

Changed in simplestreams (Ubuntu Trusty):
status: New → In Progress
Changed in simplestreams (Ubuntu Vivid):
status: New → In Progress
Revision history for this message
Martin Pitt (pitti) wrote :

Both uploaded to SRU review queues, unsubscribing ubuntu-sponsors.

Revision history for this message
Scott Moser (smoser) wrote :

Felipe,
  A couple comments on your debdiffs.
a.) Please add dep8 comments, and point to the upstream commits if you can.
b.) in trunk, we dropped use of 'strip_version' everywhere in revno 362 [1] , then a pep8 cleanup in revno 364 [2], but then added the strip_version *back* for glance at revno 365 [3]. revno 365 was 2014-12-12, only 2 days after 362. I have since lost context, but I suspect that doing so broke an older cloud (quite possibly canonistack).

It seems that you have incorporated upstream commits 362 and 364 but not 365.

As I said in commit message at 365, strip_version is certainly nefarious, but seems to be necessary.

--
[1] http://bazaar.launchpad.net/~smoser/simplestreams/trunk/revision/362
[2] http://bazaar.launchpad.net/~smoser/simplestreams/trunk/revision/364
[3] http://bazaar.launchpad.net/~smoser/simplestreams/trunk/revision/365

Revision history for this message
Felipe Reyes (freyes) wrote :

Hi Scott,

Thanks for reviewing this SRU. You're right, I missed r365, I updated the patch including that revision and dep3 information.

Best,

Revision history for this message
Felipe Reyes (freyes) wrote :

Scott,

I rebased the patched on top of the newer versions that were pushed into *-security, so now they should be in shape to be reviewed.

Best,

Revision history for this message
Felipe Reyes (freyes) wrote :
Revision history for this message
Felipe Reyes (freyes) wrote :
Revision history for this message
Felipe Reyes (freyes) wrote :

Rebased patches one more time.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Björn, or anyone else affected,

Accepted simplestreams into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0~bzr354-0ubuntu1.15.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in simplestreams (Ubuntu Vivid):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in simplestreams (Ubuntu Trusty):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Björn, or anyone else affected,

Accepted simplestreams into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0~bzr341-0ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Felipe Reyes (freyes) wrote :

I could successfully sync images (create bucket + upload images) with package from trusty-proposed[0] and vivid proposed. I'm attaching /var/log/glance-simplestreams-sync.log for trusty and vivid as evidence.

[0] *** 0.1.0~bzr341-0ubuntu2.4 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Revision history for this message
Felipe Reyes (freyes) wrote :
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr341-0ubuntu2.4

---------------
simplestreams (0.1.0~bzr341-0ubuntu2.4) trusty-proposed; urgency=medium

  [ Felipe Reyes ]
  * d/p/lp1346935-fix-bucket-creation-on-rgw.patch: Fix bucket creation when
    ceph-radosgw is used to provide a Swift API (LP: #1346935).

 -- Scott Moser <email address hidden> Thu, 08 Oct 2015 15:15:21 -0400

Changed in simplestreams (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for simplestreams has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr354-0ubuntu1.15.04.3

---------------
simplestreams (0.1.0~bzr354-0ubuntu1.15.04.3) vivid-proposed; urgency=medium

  * d/p/lp1346935-fix-bucket-creation-on-rgw.patch: Fix bucket creation when
    ceph-radosgw is used to provide a Swift API (LP: #1346935).

 -- Felipe Reyes <email address hidden> Tue, 29 Sep 2015 18:29:28 -0300

Changed in simplestreams (Ubuntu Vivid):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote : Fixed in simplestreams version 0.1.0.

This bug is believed to be fixed in simplestreams in version 0.1.0. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in simplestreams:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.