OpenStack Compute (nova)

Libvirt error launching instance - Device 'virtio-net-pci' could not be initialized

Bug #1304107 reported by Matt Kassawara
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
libvirt (Ubuntu)
Fix Released
Undecided
Chuck Short
qemu (Ubuntu)
Fix Released
High
Serge Hallyn

Bug Description

I'm developing the installation guide for Icehouse. In this particular case, I'm installing and testing RC1 on Ubuntu 12.04 with nova networking. All nodes in this environment run as VMs and the nova-compute service uses QEMU due to hardware limitations with nested VMs. Attempting to launch an instance generates the following error (full traceback attached):

2014-04-07 17:50:52.235 1220 TRACE nova.compute.manager [instance: 4574ce1a-e81f-4bfc-a079-b45c2a1f31ae] libvirtError: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:50:c4:59,bus=pci.0,addr=0x1: PCI: slot 1 function 0 not available for virtio-net-pci, in use by PIIX3
2014-04-07 17:50:52.235 1220 TRACE nova.compute.manager [instance: 4574ce1a-e81f-4bfc-a079-b45c2a1f31ae] qemu-system-x86_64: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:50:c4:59,bus=pci.0,addr=0x1: Device initialization failed.
2014-04-07 17:50:52.235 1220 TRACE nova.compute.manager [instance: 4574ce1a-e81f-4bfc-a079-b45c2a1f31ae] qemu-system-x86_64: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:50:c4:59,bus=pci.0,addr=0x1: Device 'virtio-net-pci' could not be initialized

Package version information:

nova-compute: :2014.1~rc1-0ubuntu1~cloud0
nova-network: :2014.1~rc1-0ubuntu1~cloud0
libvirt-bin: 1.2.2-0ubuntu7~cloud0

Command output:

$ nova service-list
+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| nova-cert | hst-osctl5 | internal | enabled | up | 2014-04-08T00:33:15.000000 | - |
| nova-consoleauth | hst-osctl5 | internal | enabled | up | 2014-04-08T00:33:19.000000 | - |
| nova-scheduler | hst-osctl5 | internal | enabled | up | 2014-04-08T00:33:13.000000 | - |
| nova-conductor | hst-osctl5 | internal | enabled | up | 2014-04-08T00:33:16.000000 | - |
| nova-compute | hst-oscpu5 | nova | enabled | up | 2014-04-08T00:33:15.000000 | - |
| nova-network | hst-oscpu5 | internal | enabled | up | 2014-04-08T00:33:13.000000 | - |
+------------------+------------+----------+---------+-------+----------------------------+-----------------+

$ nova net-list
+--------------------------------------+----------+------------------+
| ID | Label | CIDR |
+--------------------------------------+----------+------------------+
| 7f849be3-4494-495a-95a1-0f99ccb884c4 | demo-net | 172.24.246.24/29 |
+--------------------------------------+----------+------------------+

Revision history for this message
Matt Kassawara (ionosphere80) wrote :
Revision history for this message
Joe Gordon (jogo) wrote :

Unfortunately we don't test libvirt 1.2.x in the gate so its possible we broke something. I think not being able to boot an instance qualifies for a good candidate for RC2

Changed in nova:
milestone: none → icehouse-rc2
Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Also, this issue does not impact a similar environment with neutron networking.

Revision history for this message
Phil Hopkins (phil-hopkins-a) wrote :

I alos am having this using KVM. I believe Matt is using QEMU. Since this seems to be a libvirt problem I would expect both QEMU and KVM to have th eproblem.

Revision history for this message
Russell Bryant (russellb) wrote :

Phil, can you confirm your version of libvirt?

tags: added: icehouse-rc-potential
Changed in nova:
milestone: icehouse-rc2 → none
Revision history for this message
Daniel Berrange (berrange) wrote :

It would be nice to get the QEMU log file from /var/log/libvirt/qemu/$GUESTNAME.log to show this error

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Attaching QEMU instance log.

Revision history for this message
Daniel Berrange (berrange) wrote :

Ok that log is interesting as it shows that Ubuntu have re-named the QEMU machine types

 ... -machine trusty,accel=tcg,usb=off ...

See 'trusty' is the machine type there instead of the more usual "pc-i440fx-1.6"

This unfortunately breaks the ability of libvirt to figure out what type of base board is used by the VM - it can't distinguish i440fx from q35. I'm guessing Ubuntu was probably inspired by the fact that we did a similar thing in RHEL-6, replacing 'pc' with 'rhel-6.1.0'.

The recommendation for distros who want to customize the machine types without breaking libvirt is to *not* replace the entire machine type name, but instead to just replace the upstream version number with a distro specific suffix.

eg change 'pc-i440fx-1.6' to be 'pc-i440fx-trusty'. That way libvirt can still see that this is an i440fx machine type and do the right thing with PCI address validation.

As a reference point, RHEL7 does this correctly now too, using 'pc-i440fx-rhel7.0.0' as machine type name.

Unfortunately there's nothing OpenStack Nova can do to workaround this. Either Ubuntu have to change their QEMU machine type name as illustrated, to maintain the right prefix, or they'll have to hack their libvirt build to cope with the bare name 'trusty' :-(

Russell Bryant (russellb)
Changed in nova:
status: New → Invalid
tags: removed: icehouse-backport-potential icehouse-rc-potential
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks, Daniel. Yes, the intent here was to try and avoid the situation, in the future, that we currently have with distinguishing migrations from precise hosts from migrations from saucy hosts with older machine types.

I'll fix this post haste in qemu

Changed in qemu (Ubuntu):
importance: Undecided → High
status: New → Triaged
assignee: nobody → Serge Hallyn (serge-hallyn)
Chuck Short (zulcss)
Changed in libvirt (Ubuntu):
assignee: nobody → Chuck Short (zulcss)
Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Excellent work... thanks so much!

Revision history for this message
Phil Hopkins (phil-hopkins-a) wrote :

my libvirt is 1.2.2-0ubuntu7~cloud0

running Ubuntu 12.04

Phil

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.2.2-0ubuntu11

---------------
libvirt (1.2.2-0ubuntu11) trusty; urgency=medium

  * debian/patches/recognize-trusty-machine-type.patch: Revert patch
    since it was causing issues with virtio deivces. (LP: #1304107)
 -- Chuck Short <email address hidden> Tue, 08 Apr 2014 12:51:55 -0400

Changed in libvirt (Ubuntu):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 2.0.0~rc1+dfsg-0ubuntu2

---------------
qemu (2.0.0~rc1+dfsg-0ubuntu2) trusty; urgency=medium

  * define-trusty-machine-type.patch: update the trusty machine type name to
    pc-i440fx-trusty (LP: #1304107)
 -- Serge Hallyn <email address hidden> Tue, 08 Apr 2014 11:49:04 -0500

Changed in qemu (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Matt Kassawara (ionosphere80) wrote :

The updated packages resolve this issue for me. Thanks for the quick work!

Chuck Short (zulcss)
Changed in cloud-archive:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.