OpenStack Compute (nova)

Instance destroy fails in xenapi driver if kernel/ramdisk images were deleted

Bug #1022681 reported by Johannes Erdfelt on 2012-07-09

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Undecided	Johannes Erdfelt	OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

Under the xenapi driver, if an image is booted that uses a separate kernel/ramdisk, then the kernel/ramdisk is deleted from the dom0, and then the instance is destroyed through nova, an exception will be raised and the instance will fail to be deleted.

2012-07-09 18:13:31 DEBUG nova.virt.xenapi.connection [req-b8e49927-5868-48ed-86f1-1a175ce886bf admin openstack] Got exception: ['XENAPI_PLUGIN_EXCEPTION', 'remove_kernel_ramdisk', 'OSError', "[Errno 2] No such file or directory: '/boot/guest/8c32d293-53b5-4eda-b17d-457154678adc'"] from (pid=31888) _unwrap_plugin_exceptions /home/johannes/openstack/nova/trunk/nova/virt/xenapi/connection.py:616
2012-07-09 18:13:31 DEBUG nova.openstack.common.rpc.amqp [-] Sending terminate_instance on notifications.error from (pid=31888) notify /home/johannes/openstack/nova/trunk/nova/openstack/common/rpc/amqp.py:409
2012-07-09 18:13:31 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/openstack/common/rpc/amqp.py", line 274, in _process_data
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/openstack/common/rpc/dispatcher.py", line 106, in dispatch
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/exception.py", line 117, in wrapped
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/exception.py", line 92, in wrapped
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 178, in decorated_function
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 202, in decorated_function
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp sys.exc_info())
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 196, in decorated_function
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 787, in terminate_instance
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp do_terminate_instance()
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/utils.py", line 674, in inner
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 780, in do_terminate_instance
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self._delete_instance(context, instance)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 757, in _delete_instance
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self._shutdown_instance(context, instance)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/compute/manager.py", line 725, in _shutdown_instance
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp block_device_info)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/connection.py", line 217, in destroy
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self._vmops.destroy(instance, network_info)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vmops.py", line 1082, in destroy
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return self._destroy(instance, vm_ref, network_info)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vmops.py", line 1106, in _destroy
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp self._destroy_kernel_ramdisk(instance, vm_ref)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vmops.py", line 1033, in _destroy_kernel_ramdisk
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp vm_utils.destroy_kernel_ramdisk(self._session, kernel, ramdisk)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/vm_utils.py", line 625, in destroy_kernel_ramdisk
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp session.call_plugin('kernel', 'remove_kernel_ramdisk', args)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/connection.py", line 605, in call_plugin
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp host, plugin, fn, args)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/home/johannes/openstack/nova/trunk/nova/virt/xenapi/connection.py", line 614, in _unwrap_plugin_exceptions
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return func(*args, **kwargs)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 229, in __call__
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp return self.__send(self.__name, args)
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 133, in xenapi_request
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp result = _parse_result(getattr(self, methodname)(*full_params))
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 203, in _parse_result
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp raise Failure(result['ErrorDescription'])
2012-07-09 18:13:31 TRACE nova.openstack.common.rpc.amqp Failure: ['XENAPI_PLUGIN_EXCEPTION', 'remove_kernel_ramdisk', 'OSError', "[Errno 2] No such file or directory: '/boot/guest/8c32d293-53b5-4eda-b17d-457154678adc'"]

Johannes Erdfelt (johannes.erdfelt) on 2012-07-09

Changed in nova:
assignee:	nobody → Johannes Erdfelt (johannes.erdfelt)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-09: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/9533

Changed in nova:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-07-09: Fix merged to nova (master)

Reviewed: https://review.openstack.org/9533
Committed: http://github.com/openstack/nova/commit/a44fbea08315a30dec9ad69f92c95a532ea045c7
Submitter: Jenkins
Branch: master

commit a44fbea08315a30dec9ad69f92c95a532ea045c7
Author: Johannes Erdfelt <email address hidden>
Date: Mon Jul 9 18:16:23 2012 +0000

Ignore failure to delete kernel/ramdisk in xenapi driver

Fixes bug 1022681

    If deleting the kernel/ramdisk fails because the files don't exist
    anymore, then ignore the error and continue. This will ensure that the
    instance will get destroyed properly even if the files were deleted
    outside of nova.

Change-Id: I5d1f95ea2a6f552c48efbb9e92bb36767df19e34

Changed in nova:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2012-08-16

Changed in nova:
milestone:	none → folsom-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2012-09-27

Changed in nova:
milestone:	folsom-3 → 2012.2

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #962095

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.