OpenStack Compute (nova)

Cannot delete an instance that was booted from volume or has a volume attached

Bug #1073240 reported by Nikola Đipanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Nikola Đipanov
OpenStack Compute (nova) 2013.1 "grizzly"

Bug Description

When deleting an instance that was booted from volume there is an exception raised, This is the trace (when running on devstack):

2012-10-30 17:14:03 ERROR nova.api.openstack [req-c3443f9c-bede-4488-baf1-8bb925c0ff95 demo demo] Caught error: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 511, in dispatch
    self.engine.dispatch()
  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 817, in dispatch
    self.process(ssn)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1052, in process
    self.send(snd, msg)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1263, in send
    body = enc(msg.content)
  File "/usr/lib/python2.7/site-packages/qpid/messaging/message.py", line 28, in encode
    sc.write_primitive(type, x)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
    getattr(self, "write_%s" % type.NAME)(v)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map
    sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem
    sc.write_primitive(type, v)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
    getattr(self, "write_%s" % type.NAME)(v)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map
    sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem
    sc.write_primitive(type, v)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
    getattr(self, "write_%s" % type.NAME)(v)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 300, in write_list
    type = self.encoding(o)
  File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 59, in encoding
    raise CodecException("no encoding for %r" % obj)
CodecException: no encoding for <nova.db.sqlalchemy.models.BlockDeviceMapping object at 0x578ed10>

2012-10-30 17:14:03 TRACE nova.api.openstack Traceback (most recent call last):
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 81, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return req.get_response(self.application)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1053, in get_response
2012-10-30 17:14:03 TRACE nova.api.openstack application, catch_exc_info=False)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1022, in call_application
2012-10-30 17:14:03 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return resp(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 278, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return self.app(env, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return resp(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return resp(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return resp(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack return resp(environ, start_response)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 208, in call_func
2012-10-30 17:14:03 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 895, in __call__
2012-10-30 17:14:03 TRACE nova.api.openstack content_type, body, accept)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 951, in _process_stack
2012-10-30 17:14:03 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1039, in dispatch
2012-10-30 17:14:03 TRACE nova.api.openstack return method(req=request, **action_args)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 1082, in delete
2012-10-30 17:14:03 TRACE nova.api.openstack self._delete(req.environ['nova.context'], req, id)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 935, in _delete
2012-10-30 17:14:03 TRACE nova.api.openstack self.compute_api.delete(context, instance)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 114, in wrapped
2012-10-30 17:14:03 TRACE nova.api.openstack return func(self, context, target, *args, **kwargs)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 104, in inner
2012-10-30 17:14:03 TRACE nova.api.openstack return function(self, context, instance, *args, **kwargs)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 94, in inner
2012-10-30 17:14:03 TRACE nova.api.openstack return f(self, context, instance, *args, **kw)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 956, in delete
2012-10-30 17:14:03 TRACE nova.api.openstack self._delete(context, instance)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 925, in _delete
2012-10-30 17:14:03 TRACE nova.api.openstack self.compute_rpcapi.terminate_instance(context, instance,
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2012-10-30 17:14:03 TRACE nova.api.openstack self.gen.next()
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 925, in _delete
2012-10-30 17:14:03 TRACE nova.api.openstack self.compute_rpcapi.terminate_instance(context, instance,
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/rpcapi.py", line 519, in terminate_instance
2012-10-30 17:14:03 TRACE nova.api.openstack version='2.4')
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/proxy.py", line 113, in cast
2012-10-30 17:14:03 TRACE nova.api.openstack rpc.cast(context, self._get_topic(topic), msg)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/__init__.py", line 126, in cast
2012-10-30 17:14:03 TRACE nova.api.openstack return _get_impl().cast(cfg.CONF, context, topic, msg)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/impl_qpid.py", line 571, in cast
2012-10-30 17:14:03 TRACE nova.api.openstack rpc_amqp.get_connection_pool(conf, Connection))
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 380, in cast
2012-10-30 17:14:03 TRACE nova.api.openstack conn.topic_send(topic, msg)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 125, in __exit__
2012-10-30 17:14:03 TRACE nova.api.openstack self._done()
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 114, in _done
2012-10-30 17:14:03 TRACE nova.api.openstack self.connection.reset()
2012-10-30 17:14:03 TRACE nova.api.openstack File "/opt/stack/nova/nova/openstack/common/rpc/impl_qpid.py", line 393, in reset
2012-10-30 17:14:03 TRACE nova.api.openstack self.session.close()
2012-10-30 17:14:03 TRACE nova.api.openstack File "<string>", line 6, in close
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 739, in close
2012-10-30 17:14:03 TRACE nova.api.openstack self.sync(timeout=timeout)
2012-10-30 17:14:03 TRACE nova.api.openstack File "<string>", line 6, in sync
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 730, in sync
2012-10-30 17:14:03 TRACE nova.api.openstack snd.sync(timeout=timeout)
2012-10-30 17:14:03 TRACE nova.api.openstack File "<string>", line 6, in sync
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 885, in sync
2012-10-30 17:14:03 TRACE nova.api.openstack if not self._ewait(lambda: self.acked >= mno, timeout=timeout):
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 799, in _ewait
2012-10-30 17:14:03 TRACE nova.api.openstack result = self.session._ewait(lambda: self.error or predicate(), timeout)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
2012-10-30 17:14:03 TRACE nova.api.openstack result = self.connection._ewait(lambda: self.error or predicate(), timeout)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
2012-10-30 17:14:03 TRACE nova.api.openstack self.check_error()
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
2012-10-30 17:14:03 TRACE nova.api.openstack raise self.error
2012-10-30 17:14:03 TRACE nova.api.openstack InternalError: Traceback (most recent call last):
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 511, in dispatch
2012-10-30 17:14:03 TRACE nova.api.openstack self.engine.dispatch()
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 817, in dispatch
2012-10-30 17:14:03 TRACE nova.api.openstack self.process(ssn)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1052, in process
2012-10-30 17:14:03 TRACE nova.api.openstack self.send(snd, msg)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1263, in send
2012-10-30 17:14:03 TRACE nova.api.openstack body = enc(msg.content)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/messaging/message.py", line 28, in encode
2012-10-30 17:14:03 TRACE nova.api.openstack sc.write_primitive(type, x)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
2012-10-30 17:14:03 TRACE nova.api.openstack getattr(self, "write_%s" % type.NAME)(v)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map
2012-10-30 17:14:03 TRACE nova.api.openstack sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem
2012-10-30 17:14:03 TRACE nova.api.openstack sc.write_primitive(type, v)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
2012-10-30 17:14:03 TRACE nova.api.openstack getattr(self, "write_%s" % type.NAME)(v)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map
2012-10-30 17:14:03 TRACE nova.api.openstack sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem
2012-10-30 17:14:03 TRACE nova.api.openstack sc.write_primitive(type, v)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive
2012-10-30 17:14:03 TRACE nova.api.openstack getattr(self, "write_%s" % type.NAME)(v)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 300, in write_list
2012-10-30 17:14:03 TRACE nova.api.openstack type = self.encoding(o)
2012-10-30 17:14:03 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 59, in encoding
2012-10-30 17:14:03 TRACE nova.api.openstack raise CodecException("no encoding for %r" % obj)
2012-10-30 17:14:03 TRACE nova.api.openstack CodecException: no encoding for <nova.db.sqlalchemy.models.BlockDeviceMapping object at 0x578ed10>
2012-10-30 17:14:03 TRACE nova.api.openstack
2012-10-30 17:14:03 TRACE nova.api.openstack
2012-10-30 17:14:04 INFO nova.api.openstack [req-c3443f9c-bede-4488-baf1-8bb925c0ff95 demo demo] http://192.168.123.166:8774/v2/3dab7529f10d4f74b0401650e13dfc49/servers/df0bb585-3b67-4d38-9ed3-a472ba14e4d3 returned with HTTP 500
Exception qpid.messaging.exceptions.InternalError: InternalError('Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 511, in dispatch\n self.engine.dispatch()\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 817, in dispatch\n self.process(ssn)\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1052, in process\n self.send(snd, msg)\n File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 1263, in send\n body = enc(msg.content)\n File "/usr/lib/python2.7/site-packages/qpid/messaging/message.py", line 28, in encode\n sc.write_primitive(type, x)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive\n getattr(self, "write_%s" % type.NAME)(v)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map\n sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem\n sc.write_primitive(type, v)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive\n getattr(self, "write_%s" % type.NAME)(v)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 257, in write_map\n sc.write(string.joinfields(map(self._write_map_elem, m.keys(), m.values()), ""))\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 250, in _write_map_elem\n sc.write_primitive(type, v)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 73, in write_primitive\n getattr(self, "write_%s" % type.NAME)(v)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 300, in write_list\n type = self.encoding(o)\n File "/usr/lib/python2.7/site-packages/qpid/codec010.py", line 59, in encoding\n raise CodecException("no encoding for %r" % obj)\nCodecException: no encoding for <nova.db.sqlalchemy.models.BlockDeviceMapping object at 0x578ed10>\n',) in <bound method ConnectionContext.__del__ of <nova.openstack.common.rpc.amqp.ConnectionContext object at 0x578e850>> ignored
2012-10-30 17:14:04 INFO nova.osapi_compute.wsgi.server [req-c3443f9c-bede-4488-baf1-8bb925c0ff95 demo demo] 192.168.123.166 "DELETE /v2/3dab7529f10d4f74b0401650e13dfc49/servers/df0bb585-3b67-4d38-9ed3-a472ba14e4d3 HTTP/1.1" status: 500 len: 335 time: 36.5771089

The issue seems to be in nova.compute.api._delete method namely line 924 where we just pass a list of nova.db.sqlalchemy.models.BlockDeviceMapping objects returned by block_device_mapping_get_all_by_instance on line 918. This in turn breaks qpid library because it does not know how to deal with SqlAlchemy models as can be seen from the trace.

Nikola Đipanov (ndipanov)
Changed in nova:
assignee: nobody → Nikola Đipanov (ndipanov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/15125

Changed in nova:
status: New → In Progress
Nikola Đipanov (ndipanov)
summary: - Cannot delete an instance that was booted from volume when using qpid as
- the RPC backend
+ Cannot delete an instance that was booted from volume
summary: - Cannot delete an instance that was booted from volume
+ Cannot delete an instance that was booted from volume or has a volume
+ attached
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/15125
Committed: http://github.com/openstack/nova/commit/6390508fc9622fdc1ed87974ce416f17e9983592
Submitter: Jenkins
Branch: master

commit 6390508fc9622fdc1ed87974ce416f17e9983592
Author: Nikola Dipanov <email address hidden>
Date: Wed Oct 31 16:24:30 2012 +0100

    Make bdms primitive in rpcapi.terminate_instance

    This patch converts bdms (a list of block device mappings) passed to
    nova.compute.rpcapi.terminate_instance to primitive types that are
    safe to be sent to the rpc backend.

    Also fixes the fake BDM classes used in:
    * nova.tests.api.ec2.test_cloud
    * nova.tests.api.openstack.compute.test_server_actions
    to pass through josnutils.to_primitive properly and not confuse it due
    to wrongly delegated dotted lookups

    fixes bug #1073240

    Change-Id: I329a48c1a53dde5895a329b39f32854c286de071

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.