devstack

10/7: get-pip failing all over itself in the gate

Bug #1503909 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
Fix Released
Undecided
Sean Dague

Bug Description

Lots of grenade failures today like this:

http://logs.openstack.org/08/230008/1/gate/gate-grenade-dsvm/307726e/logs/old/devstacklog.txt.gz#_2015-10-07_10_26_44_899

2015-10-07 10:26:44.883 | + touch /opt/stack/old/devstack/files/get-pip.py.downloaded
2015-10-07 10:26:44.884 | + sudo -H -E python /opt/stack/old/devstack/files/get-pip.py
2015-10-07 10:26:44.899 | File "/opt/stack/old/devstack/files/get-pip.py", line 1
2015-10-07 10:26:44.899 | <html><body><h1>503 Service Unavailable</h1>
2015-10-07 10:26:44.899 | ^
2015-10-07 10:26:44.899 | SyntaxError: invalid syntax
2015-10-07 10:26:44.902 | ++ err_trap
2015-10-07 10:26:44.902 | ++ local r=1
2015-10-07 10:26:44.902 | stack.sh failed: full log in /opt/stack/old/devstacklog.txt.2015-10-07-102549
2015-10-07 10:26:44.902 | Error on exit
2015-10-07 10:26:44.921 | World dumping... see /opt/stack/old/worlddump-2015-10-07-102644.txt for details

Looks like fixes are up:

https://review.openstack.org/#/c/231976/

https://review.openstack.org/#/c/231986/

Revision history for this message
Matt Riedemann (mriedem) wrote :

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3ludGF4RXJyb3I6IGludmFsaWQgc3ludGF4XCIgQU5EIHRhZ3M6XCJncmVuYWRlLnNoLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDQ0MjU4NjAxODMyLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

90 hits in 24 hours, check and gate, multiple branches, all failures.

Changed in devstack:
status: New → In Progress
Dean Troyer (dtroyer)
Changed in devstack:
assignee: nobody → Sean Dague (sdague)
Revision history for this message
Robert Collins (lifeless) wrote :

12:13 < lifeless> dstufft: did bootstrap.pypa.io have an outage oh 8 hours back?
12:13 < lifeless> https://bugs.launchpad.net/devstack/+bug/1503909
12:14 < dstufft> lifeless: yes
12:15 < dstufft> lifeless: fixed it this morning eastern us time
12:15 * dstufft goes for a bit

We can't really fix this ourselves - we could run our own mirror, but this is the first bootstrap.pypa.io outage I can recall..

Changed in devstack:
status: In Progress → Fix Released
Revision history for this message
Robert Collins (lifeless) wrote :

Bah, told on IRC that the issue is that we got into this state with html on disk. If someone can reopen that would be great.

Revision history for this message
Jesse Pretorius (jesse-pretorius) wrote :

It might be worth implementing a fall back download URL, as we've done here: https://review.openstack.org/231919

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.openstack.org/232452

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/232452
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=fa41b5b47ebbf6f2d973bdde235cb58694a2103f
Submitter: Jenkins
Branch: master

commit fa41b5b47ebbf6f2d973bdde235cb58694a2103f
Author: Sean Dague <email address hidden>
Date: Thu Oct 8 06:05:20 2015 -0400

    make curl fail on pypi errors

    This will make curl fail on pypi errors, and should prevent corrupt
    images from pypi going offline for a few hours randomly, which it does
    from time to time.

    Closes-Bug: #1503909

    Change-Id: Ib4a740b7d1772e1e36aa701e42d3ac0f0ee12883

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.