nova-api crashed with input/output error

 nova services were spawned without redirection of stderr into file or /dev/null.

Nova-api was throwing exceptions like:

Exception TypeError: "'Connection' object is not iterable" in <bound method TpooledConnectionPool.__del__ of <eventlet.db_pool.TpooledConnectionPool object at 0x3ef0410>> ignored

and this was probably caused the crash.

These exceptions seem to be related to db_pool code removed in Essex

IIUC this bug is invalid? Please reopen if I got your last comment wrong.

I have trigger the same issue on a compute node and a network node (Diablo release), but without understanding the root cause.

on network node:
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/hubs/hub.py", line 226, in run
(nova): TRACE: self.wait(sleep_time)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/hubs/poll.py", line 109, in wait
(nova): TRACE: self.squelch_exception(fileno, sys.exc_info())
(nova): TRACE: File "/usr/lib/pymodules/python2.6/eventlet/hubs/hub.py", line 180, in squelch_exception
(nova): TRACE: traceback.print_exception(*exc_info)
(nova): TRACE: File "/usr/lib/python2.6/traceback.py", line 124, in print_exception
(nova): TRACE: _print(file, 'Traceback (most recent call last):')
(nova): TRACE: File "/usr/lib/python2.6/traceback.py", line 13, in _print
(nova): TRACE: file.write(str+terminator)
(nova): TRACE: IOError: [Errno 5] Input/output
(nova): TRACE:

on compute node:
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 216, in run
(nova): TRACE: self.fire_timers(self.clock())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 340, in fire_timers
(nova): TRACE: self.squelch_timer_exception(timer, sys.exc_info())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 264, in squelch_timer_exception
(nova): TRACE: traceback.print_exception(*exc_info)
(nova): TRACE: File "/usr/lib/python2.7/traceback.py", line 124, in print_exception
(nova): TRACE: _print(file, 'Traceback (most recent call last):')
(nova): TRACE: File "/usr/lib/python2.7/traceback.py", line 13, in _print
(nova): TRACE: file.write(str+terminator)
(nova): TRACE: IOError: [Errno 5] Input/output error

In case problem occurs services are down, the only solution is to restart services.

Is there any solution or best practice in order to avoid this issue?

regards,

Bug occured when services were launched in a bad manner (stderr and stdout attached to
2 -> /dev/pts/0 for example).
In case you were disconnected from pts 2 -> /dev/pts/0 (deleted),
stderr is unreachable and you trigger the following error "[Errno 5] Input/output error"

regards,

Blallau,

Do you know how can we avoid this issue when starting the service from the putty client?

We also came this issue with Grizzly Keystone and Quantum. See ->

https://answers.launchpad.net/quantum/+question/229242

The same issue with eventlet was reported against keystone in bug 1356925.