Too many connections error in nova-manage with many compute nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
New
|
Undecided
|
Unassigned |
Bug Description
I was bringing up a nova cluster (using scripted automation) with one controller and a dozen compute nodes. After some compute nodes successfully register themselves and can be seen with 'nova-manage host list', the others get exceptions in nova-compute as shown at the bottom of this report. At this point any attempt to run nova-manage gets
Exception TypeError: "'Connection' object is not iterable" in <bound method TpooledConnecti
But if mysql is restarted then nova-manage works again. Also, if nova-compute is then restarted manually on a failed compute node then it successfully registers its services. It seems to me that nova-compute should expect that there might be contention accessing the remote mysql and have a way to retry, at least for some time period.
2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return self._exit_
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return hubs.get_
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return self.greenlet.
(nova): TRACE: File "/usr/lib/
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: self.manager.
(nova): TRACE: File "/usr/lib/
(nova): TRACE: instances = self.db.
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return IMPL.instance_
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/
(nova): TRACE: session = get_session()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: _ENGINE = get_engine()
(nova): TRACE: File "/usr/lib/
(nova): TRACE: creator = eventlet.
(nova): TRACE: File "/usr/lib/
(nova): TRACE: order_as_
(nova): TRACE: File "/usr/lib/
(nova): TRACE: self.free_
(nova): TRACE: File "/usr/lib/
(nova): TRACE: **self._kwargs)
(nova): TRACE: File "/usr/lib/
(nova): TRACE: conn = tpool.execute(
(nova): TRACE: File "/usr/lib/
(nova): TRACE: rv = meth(*args,
(nova): TRACE: File "/usr/lib/
(nova): TRACE: return Connection(*args, **kwargs)
(nova): TRACE: File "/usr/lib/
(nova): TRACE: super(Connection, self)._
(nova): TRACE: OperationalError: (1040, 'Too many connections')
(nova): TRACE:
This is fixed by the removal of the db pooling code for now. It is important to be careful that the max connections is set high enough on mysql, but this is much less of an issue when every worker isn't opening up 16 connections to the db.