virsh start domain sometimes fail

in bug 903212 it's suggested that a specific, not-cleanly-cherrypickable patchset might fix it. However, the rationale behind the patchset doesn't match the patchset. asctime_r etc *are* threadsafe. The only thing I can find in the source which isn't is one use of localtime (not localtime_r). It may be worth fixing that and seeing if it helps.

Except that one use is in tools/virsh.c, which isn't threaded.

XML for one of the failing domains:

<domain type='kvm'>
  <name>clean-oneiric-generic-i386</name>
  <uuid>55d032e1-b2e2-b557-3f68-99988111b5da</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.14'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/var/lib/ubuntu-iso-testing/kernel-sru/clean-oneiric-generic-i386/disk0.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:fe:ca:62'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='vmvga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Reproduced after upgrade to precise.

This time, I got

error: failed to get domain 'o4'
error: End of file while reading data: Input/output error

Reproduced with upstream libvirt (git clone from today).

If I hack qemuProcessStop() to not kill a vm, kvm keeps running, but the
monitor sock fd (13) is closed (presumably because libvirt has disconnected).

This seems to confirm that a race triggered by another virsh list/start
action is causing a qemu task which is starting up to get killed.

Finally.

I can reproduce this in fedora, if I create a slow qemu libvirt hook. Presumably then we are hitting this because of the virt-aa-helper hook.

http://people.canonical.com/~serge/breaklibvirt.sh should show all that is needed.

I cannot reproduce this on quantal. Either with my breaklibvirt.sh script, nor by hand (doing

for i in `seq 1 4`; do virsh start cdboot$i >> /tmp/out$i 2>&1 & done; virsh list > /tmp/a 2>&1 & virsh list > /tmp/b 2>&1 & virsh list; virsh list

for example)

Hi, I can reproduce this bug using 'breaklibvirt.sh' on precise, while on quantal it runs through the iterations just fine.
I'm targeting this bug to Precise.

Also I am using:
libvirt0:
  Installed: 0.9.8-2ubuntu17.4
  Candidate: 0.9.8-2ubuntu17.4

So just for completeness, I backported the quantal package into precise:
https://launchpad.net/~christopherarges/+archive/ppa-test/+build/3914168

0.9.13-0ubuntu12 libvirt-bin libvirt0, does not fail after 100 iterations.

@christopherarges,

Could you test with the current libvirt-bin in precise-proposed? I'm wondering whether the fix for bug 1055658 fixes this bug as well.

@serge,
Looks like 0.9.8-2ubuntu17.5 does the trick. 100 iterations with your script and no errors.

Awesome, thanks. I'll go ahead and actually mark this a duplicate then, as I'm pretty sure it was exactly the same thing.