nova-compute won't restart after libvirt spawn failure

Bug #983845 reported by Yuriy Taraday
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Confirmed
Low
Unassigned

Bug Description

If instance creation failed because of some trouble with libvirt, nova-compute fails to start with "domain instance-xxxxx not found".
The expected behavior was to put such instances to ERROR state.

Revision history for this message
Russell Bryant (russellb) wrote :

Can you include some logs? The instance creation failure, and the failure to start nova-cmpute

Changed in nova:
status: New → Incomplete
Revision history for this message
Yuriy Taraday (yorik-sar) wrote :

Unfortunately, I don't have any logs around now. This bug is reproduced easily.
You can shut off kvm, e.g. by "chmod a-rwx /dev/kvm", then any instance creation will end with failure like "hvm virtualization is not available" or smth like that.
Then restart nova-compute, it won't start raising exception somewhere in LibvirtDriver.get_info like "dom is not found".

Sorry for such vague explanation, I can provide some logs later.

Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → Low
status: Incomplete → Confirmed
Revision history for this message
David Geng (genggjh) wrote :
Download full text (3.5 KiB)

I encounter the similar error when restart nova-compute, the process can not be strated:

2013-01-17 15:49:40 INFO nova.compute.manager [req-33db31ad-0456-4c21-b0fd-742cf9771e13 None None] [instance: 10085d25-056c-4856-b5ac-0806ce70ead3] Rebooting instance after nova-compute restart.
2013-01-17 15:49:40 CRITICAL nova [-] [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000002/libvirt.xml'
2013-01-17 15:49:40 TRACE nova Traceback (most recent call last):
2013-01-17 15:49:40 TRACE nova File "/usr/bin/nova-compute", line 48, in <module>
2013-01-17 15:49:40 TRACE nova service.wait()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 659, in wait
2013-01-17 15:49:40 TRACE nova _launcher.wait()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 192, in wait
2013-01-17 15:49:40 TRACE nova super(ServiceLauncher, self).wait()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 162, in wait
2013-01-17 15:49:40 TRACE nova service.wait()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 166, in wait
2013-01-17 15:49:40 TRACE nova return self._exit_event.wait()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-01-17 15:49:40 TRACE nova return hubs.get_hub().switch()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2013-01-17 15:49:40 TRACE nova return self.greenlet.switch()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 192, in main
2013-01-17 15:49:40 TRACE nova result = function(*args, **kwargs)
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 132, in run_server
2013-01-17 15:49:40 TRACE nova server.start()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 398, in start
2013-01-17 15:49:40 TRACE nova self.manager.init_host()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 313, in init_host
2013-01-17 15:49:40 TRACE nova block_device_info)
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/exception.py", line 117, in wrapped
2013-01-17 15:49:40 TRACE nova temp_level, payload)
2013-01-17 15:49:40 TRACE nova File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-01-17 15:49:40 TRACE nova self.gen.next()
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/exception.py", line 92, in wrapped
2013-01-17 15:49:40 TRACE nova return f(*args, **kw)
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 981, in resume_state_on_host_boot
2013-01-17 15:49:40 TRACE nova xml = self._get_domain_xml(instance)
2013-01-17 15:49:40 TRACE nova File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 685, in _get_domain_xml
2013-01-17 15:49:40 TRACE...

Read more...

Revision history for this message
David Geng (genggjh) wrote :

catch the Exception when run resume_state_on_host_boot to avoid compute process exist

Revision history for this message
David Geng (genggjh) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.