Ubuntu
libvirt package

Domains created and managed via libvirt/virsh fail to autostart

Bug #1350727 reported by deepblue
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Expired
Medium
Unassigned

Bug Description

1. Release of Ubuntu: 14.04 LTS ( 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux)

2. Version of package

sudo apt-cache policy libvirt-bin
libvirt-bin:
  Installiert: 1.2.2-0ubuntu13.1.1
  Installationskandidat: 1.2.2-0ubuntu13.1.1
  Versionstabelle:
 *** 1.2.2-0ubuntu13.1.1 0
        500 http://de.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.2-0ubuntu13 0
        500 http://de.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

3. What I expected to happen:
Domains (guest machines) that are marked `autostart' should be running after the host was booted.

4. What happened instead:
auto starting domains (guest machines) fails when booting the host OS (Ubuntu Server 14.04 LTS)

Libvirtd is running.
There are symlinks in /etc/libvirt/qemu/autostart.

All domains are using a bridge device (br0) that I specified myself, and using static IP addresses.
I removed the default network created by libvirt (virtual network virbr0 for NAT slows down things
and only recommended for desktop installations).

The bridge device works properly, I can log in my virtual machines via ssh and I use the bridge as
well to communicate to internal network.

sudo brctl show
bridge name bridge id STP enabled interfaces
br0 8000.6cf049051c2f no eth0, vnet0

cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
        address 192.168.2.4
        netmask 255.255.255.0
        network 192.168.2.0
        broadcast 192.168.2.255
        gateway 192.168.2.1
        dns-nameserver 192.168.2.1
        bridge_ports eth0
        bridge_maxwait 0
        bridge_stp off

Robie Basak (racb)
Changed in libvirt (Ubuntu):
importance: Undecided → High
summary: - Domains created and managed via libvirt/virsh do not autostart
+ Domains created and managed via libvirt/virsh fail to autostart
Revision history for this message
Robie Basak (racb) wrote :

I couldn't trivially reproduce this on Trusty, using cloud image 20140724 on amd64.

Steps I followed (from memory):

sudo apt-get update
sudo apt-get install -y libvirt-bin uvtool
newgrp libvirtd
uvt-simplestreams-libvirt sync release=trusty arch=amd64
uvt-kvm create foo
virsh autostart foo
sudo reboot

After reboot, "virsh list" shows "foo" as running.

(uvtool/uvt-* is just a wrapper to set up a libvirt domain quickly; it shouldn't affect this test)

I wonder if there is some interaction with your bridge setup here, but /etc/init/libvirt-bin.conf only starts the libvirt-bin job on entering runlevel 2. Is the bridge guaranteed set up by this point?

Dropping this to Medium, as it only appears to affect a non-default case.

Changed in libvirt (Ubuntu):
importance: High → Medium
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Please show the contents of /var/log/libvirt/qemu/*.log for one of the failing domains. If the domains are attempt to start, then hopefully the logfiles will show how they are failing. Please show the full xml for the same domain.

Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
deepblue (clausgiesenberg) wrote :
Download full text (67.1 KiB)

Log Files
=======

libvirtd.log
=========
2014-07-26 08:29:22.562+0000: 1182: info : libvirt version: 1.2.2
2014-07-26 08:29:22.562+0000: 1182: error : qemuConnectOpen:1068 : internal error: no QEMU URI path given, try qemu:///system
2014-07-26 08:29:22.563+0000: 1179: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-26 08:29:27.640+0000: 1181: error : qemuConnectOpen:1077 : internal error: unexpected QEMU URI path '/', try qemu:///system
2014-07-26 08:29:27.641+0000: 1179: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-27 16:42:00.553+0000: 1703: info : libvirt version: 1.2.2
2014-07-27 16:42:00.553+0000: 1703: error : daemonStreamHandleAbort:615 : stream aborted at client request
2014-07-27 16:42:00.572+0000: 1703: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-27 16:49:51.591+0000: 1703: error : daemonStreamHandleAbort:615 : stream aborted at client request
2014-07-27 16:49:51.594+0000: 1703: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-27 16:51:38.654+0000: 1703: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-27 17:35:37.224+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 17:36:36.984+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 17:47:13.376+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 17:58:08.624+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 19:17:00.273+0000: 1726: error : qemuConnectOpen:1077 : internal error: unexpected QEMU URI path '/sytem', try qemu:///system
2014-07-27 19:17:00.302+0000: 1703: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2014-07-27 19:23:34.328+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 19:24:52.584+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 19:56:36.372+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 20:03:54.452+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-27 20:40:02.324+0000: 1703: error : qemuMonitorIO:656 : internal error: End of file from monitor
2014-07-29 04:23:02.518+0000: 1887: info : libvirt version: 1.2.2
2014-07-29 04:23:02.518+0000: 1887: error : virStorageFileGetMetadataRecurse:1060 : Failed to open file '/share/Win12Server/win12server.iso': No such file or directory
2014-07-29 04:23:02.550+0000: 1887: error : qemuAutostartDomain:292 : Failed to autostart VM 'win12stargate': Failed to open file '/share/Win12Server/win12server.iso': No such file or directory
2014-07-29 04:35:03.064+0000: 1768: error : virStorageFileGetMetadataRecurse:1060 : Failed to open file '/share/Win12Server/win12server.iso': No such file or directory
2014-07-29 05:13:11.710+0000: 1771: error : virStorageFileGetMetadataRecurse:1060 : Failed to open file '/share/Win12Server/win12server.is...

Serge Hallyn (serge-hallyn)
Changed in libvirt (Ubuntu):
status: Incomplete → New
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Could you please show how /virtualmachines and /share are mounted? The last autostart failure I see in the libvirtd logfile is:

2014-07-29 04:23:02.550+0000: 1887: error : qemuAutostartDomain:292 : Failed to autostart VM 'win12stargate': Failed to open file '/share/Win12Server/win12server.iso': No such file or directory

This suggests that adding "and mounted=/share" to the start on line in /etc/init/libvirt-bin.conf may work around the issue for you.

Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
deepblue (clausgiesenberg) wrote :

Hello,

I've created the virtual machine by the command:

sudo virt-install --connect qemu:///system -n win12stargate --arch=x86_64 -r 2028 -f /virtualmachines/win12stargate.qcow2 -s 20 --cdrom /share/Win12Server/win12server.iso --noautoconsole --os-type windows --os-variant win2k8 --vnc -v --vcpus=2 --network bridge=br0

The folder /virtualmachines is mapped on a lvm volume goup:

/dev/mapper/stargate--vg-lv--virtualmachines on /virtualmachines type ext4 (rw)

and has the rights drwxr-xr-x 3 root root 4096 Jul 31 10:36 virtualmachines/

Please refer the value "bridge=br0". It could be a hint to the problem.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1350727] Re: Domains created and managed via libvirt/virsh fail to autostart

Assuming you do not have network manager installed, the
/etc/network/interfaces you showed us defines br0. br0
should therefore be ready at runlevel 2. Libvirt-bin is
not started until runlevel 2. So while a bug somewhere
else is still possible, br0 shouldn't be the problem. Also
I didn't see anything in the libvirt log to suggesting such
an issue.

Is /virtualmachines mounted through /etc/fstab?

Revision history for this message
deepblue (clausgiesenberg) wrote :

Yes, the folder /virtualmachines ist mounted through /etc/fstab?

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/mapper/stargate--vg-lv--root / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
UUID=af6ce3c0-ac8c-4ab6-b26c-21d78e627ca8 /boot ext2 defaults 0 2
/dev/mapper/stargate--vg-lv--data /data ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--home /home ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--opt /opt ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--srv /srv ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--usr /usr ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--var /var ext4 defaults 0 2
/dev/mapper/stargate--vg-lv--virtualmachines /virtualmachines ext4 defaults 0 2
/dev/mapper/stargate--vg-swap_1 none swap sw 0 0

Revision history for this message
deepblue (clausgiesenberg) wrote :

I disable virbr0 NAT Interface as follows:

sudo virsh net-destroy default
sudo virsh net-undefine default
sudo service libvirt-bin restart

Executing the commands:

sudo uvt-simplestreams-libvirt sync release=trusty arch=amd64
sudo uvt-kvm create foo

outputs the error
uvt-kvm: error: libvirt: Network not found: no network with matching name 'default'

Revision history for this message
deepblue (clausgiesenberg) wrote :

Installing

1) a cenos7 virtual machine with command

" sudo virt-install --connect qemu:///system -n centos7 --arch=x86_64 -r 2028 -f /virtualmachines/centos7.qcow2 -s 8 --cdrom /share/CentOS-7.0-1406-x86_64-DVD.iso --noautoconsole --os-type linux --vnc -v --vcpus=2 --network bridge=br0"

2) enabling autostart by command

sudo virsh --connect qemu:///system autostart centos7

3) reboot the host system
sudo reboot

starts up the host system and the virtual machine centos7 very well.

Serge Hallyn (serge-hallyn)
Changed in libvirt (Ubuntu):
status: Incomplete → New
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I've also tested autostart with a domain connected to br0, with no problems.

Looking at your comment #9, the centos7 vm seems to test everything (both /virtualmachines and bridge=br0) and yet come up all right.

Can you do a series of say 5 host boots, and write down for each trial whether centos7 and the ubuntu Vm come up successfully?

Before that test, please copy /var/log/libvirt/libvirtd.log and /var/log/libvirt/qemu/ out of the way, then attach the new /var/log/libvirt/libvirtd.log and /var/log/libvirt/qemu/centos7.log and /var/log/libvirt/qemu/ubuntuvm.log files. Also please attach /var/log/udev and /var/log/syslog.

Finally for completeness also attach the xml files for both VMs ( in case anything has changed during your previous testing)

Serge Hallyn (serge-hallyn)
Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Noel McLoughlin (noelmcloughlin) wrote :

Just to share a resolution. A general fix is to modify the /etc/init/libvirt-bin.conf to wait for general network startup.

   $ head -4 /etc/init/libvirt-bin.conf
   description "libvirt daemon"
   author "Dustin Kirkland <email address hidden>"

   start on runlevel [2345] and net-device-up

To specifically fix this reported bug (br0), you could be more specific. See below. The obvserved behaviour in Ubunty 14.10 Trusty is that "libvirtd" daemon starts and virsh domains autostart once the interface (eth0, wlan0, br0, etc) is up. The value in this file cannot reference a virsh-net device or libvirtd will never startup. It must be a net device configured in /etc/network/interfaces and NetworkManager obviously.

   $ head -4 /etc/init/libvirt-bin.conf
   description "libvirt daemon"
   author "Dustin Kirkland <email address hidden>"

   start on runlevel [2345] and net-device-up IFACE=br0

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.