Instance stuck in reboot on libvirt failure

Lets say instance reboot is in progress. At one point, libvirt driver is asked to reboot the actual VM. Now in this case if the VM itself has disappeared, the instance will be stuck in rebooting forever

(This was accidentally discovered when libvirtd was killed using "sudo kill -9" when reboot was in progress. "virsh list" would also not list any instances)

Refer to following code snippet from nova/virt/libvirt/connection.py :

        def _wait_for_reboot():
            """Called at an interval until the VM is running again."""
            try:
                state = self.get_info(instance)['state']
            except exception.NotFound:
                LOG.error(_("During reboot, instance disappeared."),
                          instance=instance)
                raise utils.LoopingCallDone

            if state == power_state.RUNNING:
                LOG.info(_("Instance rebooted successfully."),
                         instance=instance)
                raise utils.LoopingCallDone

Here exception.NotFound block should NOT raise "utils.LoopingCallDone" (which indicates that operation successfully completed - and infinite loop in utils.LoopingCall is broken) Due to this, nova-compute never knows that VM has vanished

Instead it should just "raise" or "raise exception.NotFound". This will (hopefully) cascade the exception and nova-compute will catch and mark the instance as error.

Instances stuck in "rebooting" can't be deleted. Since VM has already disappeared, marking it as Error (Thus allowing delete) seems like correct solution.

There may be similar problems in _wait_for_boot(), _wait_for_running() etc.