OpenStack Compute (nova)

LXC instance without injected key doesn't get created

Bug #838154 reported by bastichelaar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Édouard Thuleau
OpenStack Compute (nova) 2011.3 "diablo"

Bug Description

Version: Diablo 4
Hypervisor: LXC
Disk type: both NBD and RAW

When I create an instance using euca2ools without injecting a key (e.g. euca-run-instances -t m1.tiny ami-00000004), the instance does not get created. It never reaches the point that it prepares a block device for usage. See the different outputs below:

Without injected key:

2011-08-31 16:39:53,733 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000022/ from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:39:53,745 INFO nova.virt.libvirt_conn [-] instance instance-00000022: Creating image
2011-08-31 16:39:53,745 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000022//rootfs from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:39:53,770 DEBUG nova.virt.libvirt_conn [-] block_device_list [] from (pid=24084) _volume_in_mapping /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:1020
2011-08-31 16:39:53,771 DEBUG nova.utils [-] Attempting to grab semaphore "1b6453892473a467d07372d45eb05abc2031647a_sm" for method "call_if_not_exists"... from (pid=24084) inner /usr/lib/pymodules/python2.7/nova/utils.py:671
2011-08-31 16:39:53,771 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/1b6453892473a467d07372d45eb05abc2031647a_sm /var/lib/nova/instances/instance-00000022/disk from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:39:56,962 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 595, in spawn
(nova.exception): TRACE: domain = self._create_new_domain(xml)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 1181, in _create_new_domain
(nova.exception): TRACE: domain.createWithFlags(launch_flags)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 531, in createWithFlags
(nova.exception): TRACE: if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
(nova.exception): TRACE: libvirtError: internal error guest failed to start: PATH=/bin:/sbin TERM=linux LIBVIRT_LXC_UUID=8630fcd2-0383-b76f-b43d-0acdfc08584d LIBVIRT_LXC_NAME=instance-00000022 /sbin/init
(nova.exception): TRACE: 16:39:55.158: 1: info : libvirt version: 0.9.4
(nova.exception): TRACE: 16:39:55.158: 1: error : lxcContainerChild:886 : cannot find init path '/sbin/init' relative to container root: No such file or directory
(nova.exception): TRACE: 16:39:55.158: 3454: info : libvirt version: 0.9.4
(nova.exception): TRACE: 16:39:55.158: 3454: error : lxcControllerRun:760 : error receiving signal from container: No such file or directory
(nova.exception): TRACE:
(nova.exception): TRACE:
2011-08-31 16:39:56,963 ERROR nova.compute.manager [-] Instance '34' failed to spawn. Is virtualization enabled in the BIOS? Details: internal error guest failed to start: PATH=/bin:/sbin TERM=linux LIBVIRT_LXC_UUID=8630fcd2-0383-b76f-b43d-0acdfc08584d LIBVIRT_LXC_NAME=instance-00000022 /sbin/init
16:39:55.158: 1: info : libvirt version: 0.9.4
16:39:55.158: 1: error : lxcContainerChild:886 : cannot find init path '/sbin/init' relative to container root: No such file or directory
16:39:55.158: 3454: info : libvirt version: 0.9.4
16:39:55.158: 3454: error : lxcControllerRun:760 : error receiving signal from container: No such file or directory
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 440, in _run_instance
(nova.compute.manager): TRACE: network_info, block_device_info)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 129, in wrapped
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: internal error guest failed to start: PATH=/bin:/sbin TERM=linux LIBVIRT_LXC_UUID=8630fcd2-0383-b76f-b43d-0acdfc08584d LIBVIRT_LXC_NAME=instance-00000022 /sbin/init
(nova.compute.manager): TRACE: 16:39:55.158: 1: info : libvirt version: 0.9.4
(nova.compute.manager): TRACE: 16:39:55.158: 1: error : lxcContainerChild:886 : cannot find init path '/sbin/init' relative to container root: No such file or directory
(nova.compute.manager): TRACE: 16:39:55.158: 3454: info : libvirt version: 0.9.4
(nova.compute.manager): TRACE: 16:39:55.158: 3454: error : lxcControllerRun:760 : error receiving signal from container: No such file or directory

With injected key:

2011-08-31 16:40:42,489 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000023/ from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:42,501 INFO nova.virt.libvirt_conn [-] instance instance-00000023: Creating image
2011-08-31 16:40:42,502 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000023//rootfs from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:42,532 DEBUG nova.virt.libvirt_conn [-] block_device_list [] from (pid=24084) _volume_in_mapping /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:1020
2011-08-31 16:40:42,532 DEBUG nova.utils [-] Attempting to grab semaphore "77de68daecd823babbb58edb1c8e14d7106e83bb_sm" for method "call_if_not_exists"... from (pid=24084) inner /usr/lib/pymodules/python2.7/nova/utils.py:671
2011-08-31 16:40:42,533 DEBUG nova.utils [-] Running cmd (subprocess): cp /var/lib/nova/instances/_base/77de68daecd823babbb58edb1c8e14d7106e83bb_sm /var/lib/nova/instances/instance-00000023/disk from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:42,635 INFO nova.virt.libvirt_conn [-] instance instance-00000023: injecting key into image 3
2011-08-31 16:40:42,636 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --find --show /var/lib/nova/instances/instance-00000023/disk from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:42,657 DEBUG nova.utils [-] Running cmd (subprocess): sudo tune2fs -c 0 -i 0 /dev/loop0 from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,670 DEBUG nova.utils [-] Running cmd (subprocess): sudo mount /dev/loop0 /tmp/tmpmARN6q from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,692 DEBUG nova.utils [-] Running cmd (subprocess): sudo mkdir -p /tmp/tmpmARN6q/root/.ssh from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,713 DEBUG nova.utils [-] Running cmd (subprocess): sudo chown root /tmp/tmpmARN6q/root/.ssh from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,734 DEBUG nova.utils [-] Running cmd (subprocess): sudo chmod 700 /tmp/tmpmARN6q/root/.ssh from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,754 DEBUG nova.utils [-] Running cmd (subprocess): sudo tee -a /tmp/tmpmARN6q/root/.ssh/authorized_keys from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,774 DEBUG nova.utils [-] Running cmd (subprocess): sudo umount /dev/loop0 from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,946 DEBUG nova.utils [-] Running cmd (subprocess): rmdir /tmp/tmpmARN6q from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,957 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --detach /dev/loop0 from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
2011-08-31 16:40:43,978 DEBUG nova.utils [-] Running cmd (subprocess): sudo losetup --find --show /var/lib/nova/instances/instance-00000023/disk from (pid=24084) execute /usr/lib/pymodules/python2.7/nova/utils.py:164
....
....
2011-08-31 16:40:46,159 DEBUG nova.virt.libvirt_conn [-] instance instance-00000023: is running from (pid=24084) spawn /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:596
2011-08-31 16:40:46,209 INFO nova.virt.libvirt_conn [-] Instance instance-00000023 spawned successfully.

Tags: lxc

Related branches

lp:~ethuleau/nova/lp838154
OpenStack release team: Pending requested
Diff: 8638 lines (+3773/-2343) (has conflicts)
66 files modified
MANIFEST.in (+1/-1)
nova/api/ec2/cloud.py (+1/-1)
nova/api/openstack/common.py (+43/-56)
nova/api/openstack/contrib/flavorextradata.py (+46/-0)
nova/api/openstack/flavors.py (+37/-36)
nova/api/openstack/image_metadata.py (+28/-30)
nova/api/openstack/images.py (+57/-78)
nova/api/openstack/ips.py (+28/-33)
nova/api/openstack/limits.py (+38/-40)
nova/api/openstack/schemas/v1.1/addresses.rng (+14/-0)
nova/api/openstack/schemas/v1.1/flavor.rng (+14/-0)
nova/api/openstack/schemas/v1.1/flavors.rng (+6/-0)
nova/api/openstack/schemas/v1.1/flavors_index.rng (+12/-0)
nova/api/openstack/schemas/v1.1/image.rng (+30/-0)
nova/api/openstack/schemas/v1.1/images.rng (+6/-0)
nova/api/openstack/schemas/v1.1/images_index.rng (+12/-0)
nova/api/openstack/schemas/v1.1/limits.rng (+28/-0)
nova/api/openstack/schemas/v1.1/metadata.rng (+9/-0)
nova/api/openstack/schemas/v1.1/server.rng (+3/-3)
nova/api/openstack/servers.py (+108/-36)
nova/api/openstack/versions.py (+91/-167)
nova/api/openstack/views/flavors.py (+3/-0)
nova/api/openstack/views/images.py (+10/-0)
nova/api/openstack/views/versions.py (+1/-1)
nova/api/openstack/wsgi.py (+44/-6)
nova/compute/api.py (+10/-1)
nova/compute/manager.py (+12/-1)
nova/image/fake.py (+1/-2)
nova/image/glance.py (+95/-23)
nova/image/s3.py (+1/-2)
nova/image/service.py (+0/-200)
nova/scheduler/abstract_scheduler.py (+1/-1)
nova/scheduler/base_scheduler.py (+38/-14)
nova/tests/api/openstack/common.py (+22/-0)
nova/tests/api/openstack/contrib/test_createserverext.py (+39/-0)
nova/tests/api/openstack/fakes.py (+40/-87)
nova/tests/api/openstack/test_api.py (+25/-0)
nova/tests/api/openstack/test_common.py (+115/-67)
nova/tests/api/openstack/test_extensions.py (+1/-0)
nova/tests/api/openstack/test_flavors.py (+158/-101)
nova/tests/api/openstack/test_image_metadata.py (+65/-102)
nova/tests/api/openstack/test_images.py (+542/-571)
nova/tests/api/openstack/test_limits.py (+63/-45)
nova/tests/api/openstack/test_servers.py (+207/-32)
nova/tests/api/openstack/test_versions.py (+258/-360)
nova/tests/api/openstack/test_wsgi.py (+34/-23)
nova/tests/fake_network.py (+194/-0)
nova/tests/glance/stubs.py (+73/-1)
nova/tests/image/test_glance.py (+399/-2)
nova/tests/integrated/test_xml.py (+6/-6)
nova/tests/test_compute.py (+14/-5)
nova/tests/test_direct.py (+1/-1)
nova/tests/test_libvirt.py (+137/-147)
nova/tests/test_network.py (+32/-43)
nova/tests/test_virt_drivers.py (+493/-0)
nova/tests/test_vmwareapi.py (+2/-1)
nova/version.py (+1/-1)
nova/virt/driver.py (+2/-1)
nova/virt/fake.py (+1/-1)
nova/virt/hyperv.py (+1/-1)
nova/virt/libvirt/connection.py (+7/-6)
nova/virt/libvirt/firewall.py (+2/-2)
nova/virt/vmwareapi_conn.py (+1/-1)
nova/virt/xenapi/vmops.py (+7/-2)
nova/virt/xenapi_conn.py (+2/-2)
tools/pip-requires (+1/-0)
Todd Willey (community): Approve
Devin Carlen (community): Approve
Diff: 23 lines (+5/-4)
1 file modified
nova/virt/libvirt/connection.py (+5/-4)
lp:~vishvananda/nova/lp838154-ms
OpenStack release team: Pending requested
Diff: 23 lines (+5/-4)
1 file modified
nova/virt/libvirt/connection.py (+5/-4)
Revision history for this message
Thierry Carrez (ttx) wrote :

Nice catch.

Changed in nova:
importance: Undecided → High
status: New → Confirmed
Thierry Carrez (ttx)
tags: added: lxc
Revision history for this message
Chuck Short (zulcss) wrote :

Which version of Ubuntu are you using?

Regards
chuck

Revision history for this message
Édouard Thuleau (ethuleau) wrote :

I think I've found the problem.
In file 'nova/virt/libvirt/connection.py', class 'LibvirtConnection' in method '_create_image' the method 'disk.setup_container' is call only if a key, or network file or metadata are defined for the instance: 'if any((key, net, metadata))'

So if we try to instantiate a lxc VM with OSAPI without key, injecting network or metadata, the instance doesn't start and no error are raised.

So I think, we can put the method 'disk.setup_container' out of the 'if' like is done for UML (cf. branch code)

Édouard Thuleau (ethuleau)
Changed in nova:
status: Confirmed → In Progress
assignee: nobody → Doude (doude)
status: In Progress → Fix Released
Thierry Carrez (ttx)
Changed in nova:
status: Fix Released → In Progress
OpenStack Infra (hudson-openstack)
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.