Running revision 1104 of Nova ran into the following when sending 50 simultaneous deletes. Some deletes completed but others did not:
2011-05-25 14:51:02,077 ERROR nova.api.openstack [-] Caught error: Socket closed
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 59, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/wsgi.py", line 364, in __call__
(nova.api.openstack): TRACE: result = method(**arg_dict)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/scheduler/api.py", line 281, in new_f
(nova.api.openstack): TRACE: return f(*args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 120, in delete
(nova.api.openstack): TRACE: self.compute_api.delete(req.environ['nova.context'], id)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/scheduler/api.py", line 216, in wrapped_f
(nova.api.openstack): TRACE: return f(*args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/api.py", line 404, in delete
(nova.api.openstack): TRACE: instance_id, host)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/api.py", line 470, in _cast_compute_message
(nova.api.openstack): TRACE: rpc.cast(context, queue, kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 414, in cast
(nova.api.openstack): TRACE: publisher = TopicPublisher(connection=conn, topic=topic)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 254, in __init__
(nova.api.openstack): TRACE: super(TopicPublisher, self).__init__(connection=connection)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 653, in __init__
(nova.api.openstack): TRACE: self.declare()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 664, in declare
(nova.api.openstack): TRACE: auto_delete=self.auto_delete)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 240, in exchange_declare
(nova.api.openstack): TRACE: return self.channel.exchange_declare(exchange=exchange,
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 179, in channel
(nova.api.openstack): TRACE: self._channel_ref = weakref.ref(self.connection.get_channel())
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 150, in get_channel
(nova.api.openstack): TRACE: return self.connection.channel()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 120, in connection
(nova.api.openstack): TRACE: self._connection = self._establish_connection()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 133, in _establish_connection
(nova.api.openstack): TRACE: return self.create_backend().establish_connection()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 195, in establish_connection
(nova.api.openstack): TRACE: connect_timeout=conninfo.connect_timeout)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 140, in __init__
(nova.api.openstack): TRACE: (10, 30), # tune
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.api.openstack): TRACE: self.channel_id, allowed_methods)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 198, in _wait_method
(nova.api.openstack): TRACE: self.method_reader.read_method()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/method_framing.py", line 215, in read_method
(nova.api.openstack): TRACE: raise m
(nova.api.openstack): TRACE: IOError: Socket closed
(nova.api.openstack): TRACE:
2011-05-25 14:51:02,082 ERROR nova.api.openstack [-] Caught error: Socket closed
