Hello,
When using OpenStack Liberty with XenServer, neutron is not properly closing its XenAPI sessions. Since it creates these so rapidly, the XenServer host eventually exceeds its maximum allowed number of connections:
Mar 17 11:39:05 compute3 xapi: [debug|compute3.openstack.lab.eco.rackspace.com|25 db_gc|DB GC D:bb694b976766|db_gc] Number of disposable sessions in group 'external' in database (401/401) exceeds limit (400): will delete the oldest
This occurs roughly once per minute, with many sessions being invalidated. The effect is that any long-running hypervisor operations (for example a live-migration) will fail with an "unauthorized" error, as their session was invalidated while they were still running:
2016-03-17 11:43:34.483 14310 ERROR nova.virt.xenapi.vmops Failure: ['INTERNAL_ERROR', 'Storage_interface.Internal_error("Http_client.Http_error(\\"401\\", \\"{ frame = false; method = POST; uri = /services/SM;
query = [ session_id=OpaqueRef:8663a5b7-928e-6ef5-e312-9f430b553c7f ]; content_length = [ ]; transfer encoding = ; version = 1.0; cookie = [ ]; task = ; subtask_of = ; content-type = ; host = ; user_agent = xe
n-api-libs/1.0 }\\")")']
The fix is to add a line to neutron-rootwrap-xen-dom0 to have it properly close the sessions.
Before:
def run_command(url, username, password, user_args, cmd_input):
try:
session = XenAPI.Session(url)
session.login_with_password(username, password)
host = session.xenapi.session.get_this_host(session.handle)
result = session.xenapi.host.call_plugin(
host, 'netwrap', 'run_command',
{'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
return json.loads(result)
except Exception as e:
traceback.print_exc()
sys.exit(RC_XENAPI_ERROR)
After:
def run_command(url, username, password, user_args, cmd_input):
try:
session = XenAPI.Session(url)
session.login_with_password(username, password)
host = session.xenapi.session.get_this_host(session.handle)
result = session.xenapi.host.call_plugin(
host, 'netwrap', 'run_command',
{'cmd': json.dumps(user_args), 'cmd_input': json.dumps(cmd_input)})
session.xenapi.session.logout()
return json.loads(result)
except Exception as e:
traceback.print_exc()
sys.exit(RC_XENAPI_ERROR)
After making this change, the logs still show the sessions being rapidly created, but it also shows them being destroyed. The "exceeds limit" error no longer occurs, and live-migrations now succeed.
Regards,
Alex Oughton
Fix proposed to branch: master
Review: https:/