OpenStack Compute (nova)

Socket Closed when doing Mass Delete of instances

Bug #794137 reported by Antony Messerli on 2011-06-07

This bug report is a duplicate of: Bug #797770: 'Socket closed' during API stress test. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Confirmed	High	Unassigned

Bug Description

Running Rev 1156 of Nova. Occurs when loading down API wtih either mass builds or mass deletes.

2011-06-07 16:26:41,586 ERROR nova.api.openstack [-] Caught error: Socket closed
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 60, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 357, in __call__
(nova.api.openstack): TRACE: action_result = self.dispatch(request, action, action_args)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 380, in dispatch
(nova.api.openstack): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/scheduler/api.py", line 281, in new_f
(nova.api.openstack): TRACE: return f(*args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 111, in delete
(nova.api.openstack): TRACE: self.compute_api.delete(req.environ['nova.context'], id)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/scheduler/api.py", line 216, in wrapped_f
(nova.api.openstack): TRACE: return f(*args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/api.py", line 414, in delete
(nova.api.openstack): TRACE: instance_id, host)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/api.py", line 477, in _cast_compute_message
(nova.api.openstack): TRACE: rpc.cast(context, queue, kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 551, in cast
(nova.api.openstack): TRACE: publisher = TopicPublisher(connection=conn, topic=topic)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 346, in __init__
(nova.api.openstack): TRACE: super(TopicPublisher, self).__init__(connection=connection)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 653, in __init__
(nova.api.openstack): TRACE: self.declare()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 664, in declare
(nova.api.openstack): TRACE: auto_delete=self.auto_delete)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 240, in exchange_declare
(nova.api.openstack): TRACE: return self.channel.exchange_declare(exchange=exchange,
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 179, in channel
(nova.api.openstack): TRACE: self._channel_ref = weakref.ref(self.connection.get_channel())
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 150, in get_channel
(nova.api.openstack): TRACE: return self.connection.channel()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 120, in connection
(nova.api.openstack): TRACE: self._connection = self._establish_connection()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 133, in _establish_connection
(nova.api.openstack): TRACE: return self.create_backend().establish_connection()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 195, in establish_connection
(nova.api.openstack): TRACE: connect_timeout=conninfo.connect_timeout)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 140, in __init__
(nova.api.openstack): TRACE: (10, 30), # tune
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.api.openstack): TRACE: self.channel_id, allowed_methods)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 198, in _wait_method
(nova.api.openstack): TRACE: self.method_reader.read_method()
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/method_framing.py", line 215, in read_method
(nova.api.openstack): TRACE: raise m
(nova.api.openstack): TRACE: IOError: Socket closed

Revision history for this message

Antony Messerli (antonym) wrote on 2011-06-07:

Download full text (3.2 KiB)

Above is from nova-api, but I also noticed compute was tossing the error as well:

2011-06-06 21:58:40,394 ERROR nova [-] Exception during message handling
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 87, in _wrap
(nova): TRACE: return f(*args, **kw)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 247, in run_instance
(nova): TRACE: "vpn": is_vpn}})
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 540, in call
(nova): TRACE: rv = list(rv)
(nova): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 529, in wait
(nova): TRACE: raise result
(nova): TRACE: RemoteError: IOError Socket closed
(nova): TRACE: ['Traceback (most recent call last):\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 239, in _process_data\n msg_reply(msg_id, rval, None)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 403, in msg_reply\n publisher = DirectPublisher(connection=conn, msg_id=msg_id)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 386, in __init__\n super(DirectPublisher, self).__init__(connection=connection)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/messaging.py", line 653, in __init__\n self.declare()\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/messaging.py", line 664, in declare\n auto_delete=self.auto_delete)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 240, in exchange_declare\n return self.channel.exchange_declare(exchange=exchange,\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 179, in channel\n self._channel_ref = weakref.ref(self.connection.get_channel())\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 150, in get_channel\n return self.connection.channel()\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 120, in connection\n self._connection = self._establish_connection()\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 133, in _establish_connection\n return self.create_backend().establish_connection()\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 195, in establish_connection\n connect_timeout=conninfo.connect_timeout)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/connection.py", line 140, in __init__\n (10, 30), # tune\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/abstract_channel.py", line 89, in wait\n self.channel_id, allowed_methods)\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/connection.py", line 198, in _wait_method\n self.method_reader.read_method()\n', ' File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/method_framing.py", line 215, in read...

Above is from nova-api, but I also noticed compute was tossing the error as well:

2011-06-06 21:58:40,394 ERROR nova [-] Exception during message handling
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 232, in _process_data
(nova): TRACE:     rval = node_func(context=ctxt, **node_args)
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/exception.py", line 87, in _wrap
(nova): TRACE:     return f(*args, **kw)
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 247, in run_instance
(nova): TRACE:     "vpn": is_vpn}})
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 540, in call
(nova): TRACE:     rv = list(rv)
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 529, in wait
(nova): TRACE:     raise result
(nova): TRACE: RemoteError: IOError Socket closed
(nova): TRACE: ['Traceback (most recent call last):\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 239, in _process_data\n    msg_reply(msg_id, rval, None)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 403, in msg_reply\n    publisher = DirectPublisher(connection=conn, msg_id=msg_id)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/nova\\/rpc.py", line 386, in __init__\n    super(DirectPublisher, self).__init__(connection=connection)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/messaging.py", line 653, in __init__\n    self.declare()\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/messaging.py", line 664, in declare\n    auto_delete=self.auto_delete)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 240, in exchange_declare\n    return self.channel.exchange_declare(exchange=exchange,\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 179, in channel\n    self._channel_ref = weakref.ref(self.connection.get_channel())\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 150, in get_channel\n    return self.connection.channel()\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 120, in connection\n    self._connection = self._establish_connection()\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/connection.py", line 133, in _establish_connection\n    return self.create_backend().establish_connection()\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/carrot\\/backends\\/pyamqplib.py", line 195, in establish_connection\n    connect_timeout=conninfo.connect_timeout)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/connection.py", line 140, in __init__\n    (10, 30), # tune\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/abstract_channel.py", line 89, in wait\n    self.channel_id, allowed_methods)\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/connection.py", line 198, in _wait_method\n    self.method_reader.read_method()\n', '  File "\\/usr\\/lib\\/pymodules\\/python2.6\\/amqplib\\/client_0_8\\/method_framing.py", line 215, in read_method\n    raise m\n', 'IOError: Socket closed\n']

Thierry Carrez (ttx) on 2011-06-10

Changed in nova:
importance:	Undecided → High
status:	New → Confirmed

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #797770 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.