[SRU] powering multiple nodes errors with max retries

Bug #1311140 reported by Blake Rouse
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
python-seamicroclient (Ubuntu)
Fix Released
Critical
Unassigned
Trusty
Fix Released
Critical
Unassigned

Bug Description

When powering multiple nodes on at once, in different sub-processes. The following error is generated and the node is not powered off or on accordingly.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/dist-packages/provisioningserver/custom_hardware/seamicro.py", line 324, in power_control_seamicro15k_v2
    server.power_on(using_pxe=True)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 28, in power_on
    self.manager.power_on(self, using_pxe)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 114, in power_on
    self._action('power-on', server, action_params)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 219, in _action
    return self.api.client.put(url, body=body)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 165, in put
    return self._cs_request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 155, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 141, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 115, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.220.102', port=80): Max retries exceeded with url: /v2.0/servers/21/0 (Caused by <class 'httplib.BadStatusLine'>: '')

[Impact]
This bug affects the ability of MAAS (and really any other client/user) to power on multiple (at scale) nodes in a Seamicro15000 chassis using their python-seamicroclient APIT, which is what MAAS uses. This prevents MAAS from being able to power manage many nodes at the time.

[Test Case]
1. Install MAAS. (sudo apt-get install maas)
2. Register SM15k. (maas admin node-group probe-and-enlist-hardware <clkuster-uuid) model=seamicro15k mac=<mac> username=<user> password=<pass> power_control=restapi2
3. Power on Many nodes at the time (juju deploy -n 10)
4. Fail.

With the fix:

3. power on many nodes at the time will succeed.

[Regression Potential]
Minimal. This only affects power on/off of nodes. This has been tested against hardware at one of our labs.

Changed in python-seamicroclient (Ubuntu):
status: New → Confirmed
Changed in python-seamicroclient (Ubuntu):
importance: Undecided → Critical
summary: - powering multiple nodes errors with max retries
+ [SRU] powering multiple nodes errors with max retries
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Please, also copy this to ubuntu+1 (ubuntu r)

description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

err ubuntu 'u'

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Blake, or anyone else affected,

Accepted python-seamicroclient into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-seamicroclient/0.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in python-seamicroclient (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → Critical
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

I've tested the fix and it works as expected. Please copy such package into Utopic!

Thanks.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-seamicroclient - 0.2.1-0ubuntu1

---------------
python-seamicroclient (0.2.1-0ubuntu1) trusty-proposed; urgency=medium

  * New upstream bugfix release.
    - This release only fixes connection issues when trying to power on/off
      multiple nodes at the same time. (LP: #1311140)
  * debian/patches/fix-tests.patch: Fix tests.
 -- Andres Rodriguez <email address hidden> Tue, 22 Apr 2014 12:18:19 -0400

Changed in python-seamicroclient (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for python-seamicroclient has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-seamicroclient - 0.2.1-0ubuntu1

---------------
python-seamicroclient (0.2.1-0ubuntu1) trusty-proposed; urgency=medium

  * New upstream bugfix release.
    - This release only fixes connection issues when trying to power on/off
      multiple nodes at the same time. (LP: #1311140)
  * debian/patches/fix-tests.patch: Fix tests.
 -- Andres Rodriguez <email address hidden> Tue, 22 Apr 2014 12:18:19 -0400

Changed in python-seamicroclient (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.