Too many connections error in nova-manage with many compute nodes

Bug #861689 reported by David Kranz
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
New
Undecided
Unassigned

Bug Description

I was bringing up a nova cluster (using scripted automation) with one controller and a dozen compute nodes. After some compute nodes successfully register themselves and can be seen with 'nova-manage host list', the others get exceptions in nova-compute as shown at the bottom of this report. At this point any attempt to run nova-manage gets

Exception TypeError: "'Connection' object is not iterable" in <bound method TpooledConnectionPool.__del__ of <eventlet.db_pool.TpooledConnectionPool object at 0x2794d10>> ignored

But if mysql is restarted then nova-manage works again. Also, if nova-compute is then restarted manually on a failed compute node then it successfully registers its services. It seems to me that nova-compute should expect that there might be contention accessing the remote mysql and have a way to retry, at least for some time period.

2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in init_host
(nova): TRACE: instances = self.db.instance_get_all_by_host(context, self.host)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 536, in instance_get_all_by_host
(nova): TRACE: return IMPL.instance_get_all_by_host(context, host)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 101, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1392, in instance_get_all_by_host
(nova): TRACE: session = get_session()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 53, in get_session
(nova): TRACE: _ENGINE = get_engine()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 87, in get_engine
(nova): TRACE: creator = eventlet.db_pool.ConnectionPool(MySQLdb, **pool_args)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 50, in __init__
(nova): TRACE: order_as_stack=True)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/pools.py", line 108, in __init__
(nova): TRACE: self.free_items.append(self.create())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 246, in create
(nova): TRACE: **self._kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 253, in connect
(nova): TRACE: conn = tpool.execute(db_module.connect, *args, **kw)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
(nova): TRACE: rv = meth(*args,**kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in Connect
(nova): TRACE: return Connection(*args, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in __init__
(nova): TRACE: super(Connection, self).__init__(*args, **kwargs2)
(nova): TRACE: OperationalError: (1040, 'Too many connections')
(nova): TRACE:

Revision history for this message
Vish Ishaya (vishvananda) wrote :

This is fixed by the removal of the db pooling code for now. It is important to be careful that the max connections is set high enough on mysql, but this is much less of an issue when every worker isn't opening up 16 connections to the db.

Revision history for this message
David Kranz (david-kranz) wrote :

Understood. I see that the default is 100 connections, but how does one decide how big that number should be? One per worker, one per worker per service?

Revision history for this message
Vish Ishaya (vishvananda) wrote : Re: [Bug 861689] Too many connections error in nova-manage with many compute nodes
Download full text (5.4 KiB)

The db pooling code was creating up to 16 connections per worker, which maxes out connections very quickly, hence the fix to remove it.

Vish

On Sep 30, 2011, at 7:42 AM, David Kranz wrote:

> *** This bug is a duplicate of bug 838581 ***
> https://bugs.launchpad.net/bugs/838581
>
> Understood. I see that the default is 100 connections, but how does one
> decide how big that number should be? One per worker, one per worker per
> service?
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/861689
>
> Title:
> Too many connections error in nova-manage with many compute nodes
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> I was bringing up a nova cluster (using scripted automation) with one
> controller and a dozen compute nodes. After some compute nodes
> successfully register themselves and can be seen with 'nova-manage
> host list', the others get exceptions in nova-compute as shown at the
> bottom of this report. At this point any attempt to run nova-manage
> gets
>
> Exception TypeError: "'Connection' object is not iterable" in <bound
> method TpooledConnectionPool.__del__ of
> <eventlet.db_pool.TpooledConnectionPool object at 0x2794d10>> ignored
>
> But if mysql is restarted then nova-manage works again. Also, if nova-
> compute is then restarted manually on a failed compute node then it
> successfully registers its services. It seems to me that nova-compute
> should expect that there might be contention accessing the remote
> mysql and have a way to retry, at least for some time period.
>
> 2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
> 2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
> (nova): TRACE: Traceback (most recent call last):
> (nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
> (nova): TRACE: _launcher.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
> (nova): TRACE: return self._exit_event.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
> (nova): TRACE: return hubs.get_hub().switch()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
> (nova): TRACE: return self.greenlet.switch()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
> (nova): TRACE: result = function(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
> (nova): TRACE: server.start()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
> (nova): TRACE: self.manager.init_host()
> (nova): TRACE: F...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.