live migration for KVM VM via libvirt from 12.04 to 14.04 fails , unable to change machine type to pc-1.0-precise

Bug #1477488 reported by Roman Ovchinnikov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

When trying to live migrate VM with virsh migrate from 12.04 host to 14.04 host , migration fails:
root@hw06:/root# virsh migrate --live ckeymanager04.pm.lux qemu+tcp://hw20.lux/system --copy-storage-all --persistent --verbose
Please enter your authentication name: virt@hw20
Please enter your password:
error: operation failed: migration job: unexpectedly failed

where hw06 is src node with 12.04, hw20 is dst node with 14.04

expected: machine migrates
happened: migration fails

on src:
2015-07-23 09:58:58.299+0000: 25659: error : daemonStreamHandleAbort:622 : stream aborted at client request
2015-07-23 09:59:15.568+0000: 25660: error : qemuMigrationUpdateJobStatus:903 : operation failed: migration job: unexpectedly failed

on dst:
2015-07-23 09:59:19.923+0000: 60338: info : virNetDevProbeVnetHdr:122 : Enabling IFF_VNET_HDR
2015-07-23 09:59:20.067+0000: 60338: info : virSecurityDACSetOwnership:259 : Setting DAC user and group on '/dev/mapper/shared-ckeymanager04.pm.lux--disk0' to '105:111'
2015-07-23 09:59:20.267+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.268+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.269+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.270+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.278+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.278+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.279+0000: 60338: warning : qemuDomainObjEnterMonitorInternal:1274 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous
2015-07-23 09:59:20.662+0000: 60334: info : virSecurityDACRestoreSecurityFileLabel:303 : Restoring DAC user and group on '/dev/mapper/shared-ckeymanager04.pm.lux--disk0'
2015-07-23 09:59:20.662+0000: 60334: info : virSecurityDACSetOwnership:259 : Setting DAC user and group on '/dev/dm-8' to '0:0'

if i change nic type to e1000 instead of virtio, migration starts, but fails on 99%.

from LP bug #1291321 and https://wiki.ubuntu.com/QemuPTMigration i've did the following:
1) added package with roms on dst
root@hw20:~# dpkg -l|grep kvm-ipxe-precise
ii kvm-ipxe-precise 1.0.0+git-3.55f6c88-0ubuntu1.14.04.1 all PXE ROM's for KVM vms migrated from 12.04
2) set incoming_assume_qemukvm
root@hw20:~# fgrep -ir assume /etc/libvirt/
/etc/libvirt/qemu.conf:incoming_assume_qemukvm = 1
3) tryed to set machine type (MT) on VM as pc-1.0-precise, via virsh edit <machine> but this fails - in return i always get "pc-1.0-qemu-kvm" as MT. Also tryed edit /etc/libvirt/qemu/ckeymanager04.pm.lux.xml while domain is off, but the same result - "pc-1.0-qemu-kvm"

more info:
src:

root@hw06:~# lsb_release -rd
Description: Ubuntu 12.04.4 LTS
Release: 12.04

root@hw06:~# qemu-system-x86_64 -machine help
Supported machines are:
pc-1.0-precise Ubuntu 12.04 Standard PC (alias of pc-1.0-qemu-kvm)
pc-1.0-qemu-kvm Ubuntu 12.04 Standard PC (default)
pc Standard PC (alias of pc-1.0)
pc-1.0 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC

packages
root@hw06:~# dpkg -l|egrep '(qemu|kvm|virt)'
dpkg: warning: ignoring option --foreign-architecture=amd64: this architecture cannot be foreign
ii kvm-ipxe 1.0.0+git-3.55f6c88-0ubuntu1 PXE ROM's for KVM
ii libvirt-bin 0.9.8-2ubuntu17.20 programs for the libvirt library
ii libvirt0 0.9.8-2ubuntu17.20 library for interfacing with different virtualization systems
ii python-libvirt 0.9.8-2ubuntu17.20 libvirt Python bindings
ii qemu-common 1.0+noroms-0ubuntu14.23 qemu common functionality (bios, documentation, etc)
ii qemu-kvm 1.0+noroms-0ubuntu14.23 Full virtualization on i386 and amd64 hardware
ii qemu-utils 1.0+noroms-0ubuntu14.23 qemu utilities
ii virtinst 0.600.1-1ubuntu3.3 Programs to create and clone virtual machines

dst:

root@hw20:~# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04

root@hw20:~# qemu-system-x86_64 -machine help
Supported machines are:
pc-0.13 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0 Standard PC (i440FX + PIIX, 1996)
pc-1.0-qemu-kvm Standard PC (i440FX + PIIX, 1996) (alias of pc-1.0)
pc-1.0 Standard PC (i440FX + PIIX, 1996)
pc-q35-1.7 Standard PC (Q35 + ICH9, 2009)
pc-1.1 Standard PC (i440FX + PIIX, 1996)
q35 Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.0)
pc-q35-2.0 Standard PC (Q35 + ICH9, 2009)
pc-i440fx-1.4 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.5 Standard PC (i440FX + PIIX, 1996)
pc-0.14 Standard PC (i440FX + PIIX, 1996)
pc-0.15 Standard PC (i440FX + PIIX, 1996)
xenfv Xen Fully-virtualized PC
pc-q35-1.4 Standard PC (Q35 + ICH9, 2009)
isapc ISA-only PC
pc-0.10 Standard PC (i440FX + PIIX, 1996)
pc Ubuntu 14.04 PC (i440FX + PIIX, 1996) (alias of pc-i440fx-trusty)
pc-i440fx-trusty Ubuntu 14.04 PC (i440FX + PIIX, 1996) (default)
pc-1.2 Standard PC (i440FX + PIIX, 1996)
pc-0.11 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.7 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.6 Standard PC (i440FX + PIIX, 1996)
none empty machine
xenpv Xen Para-virtualized PC
pc-q35-1.5 Standard PC (Q35 + ICH9, 2009)
pc-1.0-precise Standard PC (i440FX + PIIX, 1996) (alias of pc-1.0-qemu-kvm)
pc-1.0-qemu-kvm Standard PC (i440FX + PIIX, 1996)
pc-q35-1.6 Standard PC (Q35 + ICH9, 2009)
pc-0.12 Standard PC (i440FX + PIIX, 1996)
pc-1.3 Standard PC (i440FX + PIIX, 1996)

packages:
root@hw20:~# dpkg -l|egrep '(qemu|kvm|virt)'
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all PXE boot firmware - ROM images for qemu
ii kvm-ipxe-precise 1.0.0+git-3.55f6c88-0ubuntu1.14.04.1 all PXE ROM's for KVM vms migrated from 12.04
ii libvirt-bin 1.2.2-0ubuntu13.1.12 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.12 amd64 library for interfacing with different virtualization systems
ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.13 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1.13 amd64 QEMU Full virtualization on x86 hardware (transitional package)
ii qemu-system-common 2.0.0+dfsg-2ubuntu1.13 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.13 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 2.0.0+dfsg-2ubuntu1.13 amd64 QEMU utilities
ii virtinst 0.600.4-3ubuntu2 all Programs to create and clone virtual machines

So, from my point of view, problem is in inability to change machine type, but i'm not sure.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug.

Note that while limited VM migration from 12.04 to 14.04 hosts was enabled in bug 1291321, that bug was almost marked wontfix because there are too many ways to make the migration fail.

When I try changing my machine type here, it does work - after a complete vm shutdown and restart. It cannot work while the machine is running because the qemu-kvm binary doesn't support that.

After you change the machine type, stop the vm, and restart it, what does 'ps -ef | grep kvm' show? What does 'virsh dumpxml <vm-name>' show?

Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
Roman Ovchinnikov (coolcold) wrote :

Yes, I was trying to change MT on shutted off VM , like:
virsh destroy ckeymanager04.pm.lux
virsh edit ckeymanager04.pm.lux
( if i run second virsh edit here , i'll see it has been automagically changed to "pc-1.0-qemu-kvm" )
virsh start ckeymanager04.pm.lux

ps aux:
root@hw06:~# ps aux|grep ckeymanager04
109 46659 14.0 0.0 1413952 30440 ? Sl 13:38 0:00 /usr/bin/kvm -S -M pc-1.0-qemu-kvm -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -name ckeymanager04.pm.lux -uuid a2ac3aba-a4e8-6d1e-21be-42d04176ce5f -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/ckeymanager04.pm.lux.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/mapper/shared-ckeymanager04.pm.lux--disk0,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:64:53:98,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

If i change MT to something like "pc-1.0-precisetest", it has been kept in xml definition, like:
virsh destroy ckeymanager04.pm.lux
root@hw06:~# virsh edit keymanager04.pm.lux
Domain ckeymanager04.pm.lux XML configuration edited.

root@hw06:~# grep machine /etc/libvirt/qemu/ckeymanager04.pm.lux.xml
    <type arch='x86_64' machine='pc-1.0-precisetest'>hvm</type>
starting such domains results in fail (obviously):

root@hw06:~# virsh start ckeymanager04.pm.lux
error: Failed to start domain ckeymanager04.pm.lux
error: internal error Process exited while reading console log output: Supported machines are:
pc-1.0-precise Ubuntu 12.04 Standard PC (alias of pc-1.0-qemu-kvm)
pc-1.0-qemu-kvm Ubuntu 12.04 Standard PC (default)
pc Standard PC (alias of pc-1.0)
pc-1.0 Standard PC
pc-0.14 Standard PC
pc-0.13 Standard PC
pc-0.12 Standard PC
pc-0.11 Standard PC, qemu 0.11
pc-0.10 Standard PC, qemu 0.10
isapc ISA-only PC

Revision history for this message
Roman Ovchinnikov (coolcold) wrote :

For now, i'm using something like workaround for this problem, may be it will be useful for someone.
basic idea is - using some 12.04 host as transit one, which will take VMs from other 12.04 online, then will have libvirt & kvm upgraded , restart of VMs (usually 1-2 mins), then can be moved online to 14.04 host

on transit host, 12.04 install, latest updates, without any VMs yet:
into /etc/default/grub set
GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=90"
# update-grub

first, let's make reverting easy with snapshotting / partition (you may need to add /var if it is separated)
lvcreate -n rootfspv -L 50G shared
pvcreate /dev/shared/rootfspv
vgextend system /dev/shared/rootfspv
lvcreate -n rootfssnap -s -L 30G system/root

configure libvirt to accept migration
migrate VMs to such transit host
# for i in $(virsh list |grep runn|awk '{print $2}'|xargs echo);do virsh edit $i;done
here we should set cache to none, if u are using lvm as storage, to look like:
<driver name='qemu' type='raw' cache='none'/>

then, upgrading
set sources.list to trusty ,
#apt-get update
#apt-get install qemu-kvm libvirt-bin librbd1
(i answered no to all requests for configs update)

checking all is going fine:
qemu-system-x86_64 -machine help
output should be regular, like
root@hw06:~# qemu-system-x86_64 -machine help
Supported machines are:
pc-0.13 Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0 Standard PC (i440FX + PIIX, 1996)
pc-1.0-qemu-kvm Standard PC (i440FX + PIIX, 1996) (alias of pc-1.0)

restart libvirt:
#service libvirt-bin restart
checking is ok
#virsh list --all
#apt-get install apparmor-utils
#aa-disable /etc/apparmor.d/usr.sbin.libvirtd
#service libvirt-bin restart
checking is ok
#virsh list --all

the next step is restart VMs and later they can be migrated to 14.04 online

reverting back to 12.04:
removing VMs disks:
#lvremove /dev/shared/*
reverting snapshot:

#lvconvert --merge /dev/system/rootfssnap
#reboot

after boot check there are no snapshots left, if yes, you should reboot one more time (is needed sometimes)
next group can be moved now

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.