OpenStack Compute (nova)

single xenapi session causes concurrency issues

Bug #879044 reported by Chris Behrens
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Johannes Erdfelt
OpenStack Compute (nova) 2012.1 "essex"

Bug Description

Multiple XenAPI requests can happen at the same time... but there's an issue with that because we only use a single XenAPISession().

I think it's the cause of tracebacks like this periodically:

(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 229, in __call__
(nova.rpc): TRACE: return self.__send(self.__name, args)
(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 133, in xenapi_request
(nova.rpc): TRACE: result = _parse_result(getattr(self, methodname)(*full_params))
(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 203, in _parse_result
(nova.rpc): TRACE: raise Failure(result['ErrorDescription'])
(nova.rpc): TRACE: Failure: ['HANDLE_INVALID', 'VM', 'OpaqueRef:d1054be1-0ad6-532a-fa87-60ea50b863e2']

It has been also reported that httplib will sometimes return CannotSendRequest exceptions... when there appears to be multiple XenAPI requests at nearly the same time...

See original description
Chris Behrens (cbehrens)
Changed in nova:
importance: Undecided → High
milestone: none → essex-1
Johannes Erdfelt (johannes.erdfelt)
description: updated
Johannes Erdfelt (johannes.erdfelt)
Changed in nova:
assignee: nobody → Johannes Erdfelt (johannes.erdfelt)
status: New → In Progress
Revision history for this message
Johannes Erdfelt (johannes.erdfelt) wrote :

CannotSendRequest exceptions are a result of using an httplib connection in an invalid state. That's certainly a bug in the calling code, which appears to be XenAPI.

Revision history for this message
Johannes Erdfelt (johannes.erdfelt) wrote :

Turns out this problem only appears on Python 2.7. This is because Python 2.6 would always create a new HTTP connection in xmlrpclib, but in Python 2.7, it will reuse one HTTP connection.

Since a new connection was always created in Python 2.6, there would be shared state between threads and the problem wouldn't be reproduced.

This would only cause the CannotSendRequest exception, not the XenAPI HANDLE_INVALID failure.

Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/1069
Committed: http://github.com/openstack/nova/commit/97cfccc2ce35b29fd98809b6c7ecdd86c56f76e3
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 97cfccc2ce35b29fd98809b6c7ecdd86c56f76e3
Author: Johannes Erdfelt <email address hidden>
Date: Mon Oct 24 21:55:12 2011 +0000

    Fix concurrency of XenAPI sessions

    Fixes bug 879044

    Nova currently does not serialize access to the XenAPI session which can
    result in multiple (green)threads trying to use the same HTTP connection.
    This will typically only affect Python 2.7 which has updated xmlrpclib to
    try to use one HTTP connection for multiple requests.

    Change-Id: I101d63b822c8bf8c28674a836e4b54aa8259e1a8

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-1 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.