worker failed on transient swift login exception

Bug #1474729 reported by Martin Pitt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
Won't Fix
Medium
Unassigned

Bug Description

A worker died on a what looks like a transient exception:

2015-07-15 05:01:08,937 INFO:worker Putting results into swift autopkgtest-wily wily/amd64/p/plasma-framework/20150715_050108@
2015-07-15 05:01:08,942 INFO:connectionpool Starting new HTTP connection (1): 10.24.0.132
2015-07-15 05:01:38,946 ERROR:client Authorization Failure. Authorization Failed: Unable to establish connection to http://10.24.0.132:5000/v2.0/tokens
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1181, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1155, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 346, in get_auth
    insecure=insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 283, in get_keystoneclient_2_0
    raise ClientException('Authorization Failure. %s' % err)
ClientException: Authorization Failure. Authorization Failed: Unable to establish connection to http://10.24.0.132:5000/v2.0/tokens
2015-07-15 05:01:38,950 INFO:connectionpool Starting new HTTP connection (1): 10.24.0.132
2015-07-15 05:02:08,954 ERROR:client Authorization Failure. Authorization Failed: Unable to establish connection to http://10.24.0.132:5000/v2.0/tokens
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1181, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1155, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 346, in get_auth
    insecure=insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 283, in get_keystoneclient_2_0
    raise ClientException('Authorization Failure. %s' % err)
ClientException: Authorization Failure. Authorization Failed: Unable to establish connection to http://10.24.0.132:5000/v2.0/tokens
Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 305, in <module>
    main()
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 293, in main
    queue.wait()
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatch_method
    return amqp_method(self, args, content)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver
    func(msg)
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 188, in request
    swift_con.put_container(container, headers={'X-Container-Read': '.rlistings,.r:*'})
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1265, in put_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1181, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1155, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 346, in get_auth
    insecure=insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 283, in get_keystoneclient_2_0
    raise ClientException('Authorization Failure. %s' % err)
swiftclient.exceptions.ClientException: Authorization Failure. Authorization Failed: Unable to establish connection to http://10.24.0.132:5000/v2.0/tokens

Martin Pitt (pitti)
Changed in auto-package-testing:
importance: Undecided → Medium
status: New → Triaged
tags: added: autopkgtest-cloud
Revision history for this message
Martin Pitt (pitti) wrote :

We now auto-restart workers and notify, so we don't want to build complicated code around network hiccups like this.

Changed in auto-package-testing:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.