Cannot start kvm guest

Bug #589163 reported by James Hogarth
172
This bug affects 37 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Andy Whitcroft

Bug Description

Windows XP 32bit guest (and centos 64 bit guest) with virtio network/disk fails to start.

windows VM:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.11 -enable-kvm -m 512 -smp 1 -name Outlook -uuid 88b86722-be7d-bbcc-f98c-2e90a8482b1f -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/Outlook.monitor,server,nowait -monitor chardev:monitor -localtime -boot c -drive file=/var/lib/libvirt/images/outlook.img,if=virtio,index=0,boot=on -drive file=/home/wizard/Desktop/winxpsp2.ISO,if=ide,media=cdrom,index=2 -net nic,macaddr=54:52:00:54:f7:b4,vlan=0,model=virtio,name=virtio.0 -net tap,fd=34,vlan=0,name=tap.0 -serial none -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:1 -k en-gb -vga cirrus

Centos VM:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 1 -name linuxtest -uuid 7c7aeaba-67e4-9b66-0945-9ef5be2df30c -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/linuxtest.monitor,server,nowait -monitor chardev:monitor -no-reboot -boot d -drive file=/var/lib/libvirt/images/linuxtest.img,if=virtio,index=0,format=raw -drive file=/home/wizard/isos/CentOS-5.5-x86_64-bin-DVD-1of2.iso,if=ide,media=cdrom,index=2 -net nic,macaddr=52:54:00:0f:24:a1,vlan=0,model=virtio,name=virtio.0 -net tap,fd=34,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:1 -vga cirrus
char device redirected to /dev/pts/1

VM just shows black screen on start and CPU spins out at max usage but nothing happens in the VM.

Nothing seems to get logged (the bit above is from the libvirt/qemu log file) but the same VMs work fine on previous kernel of 2.6.32-21-generic.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: linux-image-2.6.32-22-generic 2.6.32-22.35
Regression: Yes
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.32-22.35-generic 2.6.32.11+drm33.2
Uname: Linux 2.6.32-22-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.21.
AplayDevices:
 **** List of PLAYBACK Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC262 Analog [ALC262 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC262 Analog [ALC262 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: wizard 2025 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xe8500000 irq 21'
   Mixer name : 'Realtek ALC262'
   Components : 'HDA:10ec0262,103c280c,00100100'
   Controls : 31
   Simple ctrls : 19
Date: Thu Jun 3 13:58:01 2010
HibernationDevice: RESUME=UUID=63433104-2130-425d-bee7-75f11f152c06
InstallationMedia: Ubuntu 10.04 "Lucid Lynx" - Beta amd64 (20100318)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 virbr0 no wireless extensions.
MachineType: Hewlett-Packard HP xw4400 Workstation
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-22-generic root=UUID=0f363940-aedd-4590-9ffc-632ab2ceab59 ro vga=0x31B quiet splash
ProcEnviron:
 PATH=(custom, no user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
RelatedPackageVersions: linux-firmware 1.34
RfKill:

SourcePackage: linux
dmi.bios.date: 04/04/2006
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: 786D7 v02.03
dmi.board.name: 0A68h
dmi.board.vendor: Hewlett-Packard
dmi.chassis.type: 6
dmi.chassis.vendor: Hewlett-Packard
dmi.modalias: dmi:bvnHewlett-Packard:bvr786D7v02.03:bd04/04/2006:svnHewlett-Packard:pnHPxw4400Workstation:pvr:rvnHewlett-Packard:rn0A68h:rvr:cvnHewlett-Packard:ct6:cvr:
dmi.product.name: HP xw4400 Workstation
dmi.sys.vendor: Hewlett-Packard

CVE References

Revision history for this message
James Hogarth (james-hogarth) wrote :
Revision history for this message
James Hogarth (james-hogarth) wrote :

Testing with IDE devices instead of virtio shos no difference. Manually running kvm with -cpu qem64,-nx also shows no difference so don't believe it is NX related this time (but not ruling it out) however the symptoms are similar to that old bug.

Revision history for this message
Norberto Bensa (nbensa) wrote :

Hello. Same problem after upgrade to kernel 2.6.32-22.35.

Revision history for this message
James Hogarth (james-hogarth) wrote :

zoolook only way I have found to run my VM is to revert to 2.6.32-21 at this time...

note that if you use nvidia (or ati) propriety drivers than you will probably have to use dkms to rebuild/install the nvidia-current (in my case nvidia) for the -21 kernel or else end up with no valid nvidia module found.

Revision history for this message
Norberto Bensa (nbensa) wrote :

Hello James,

thank you. I'll revert to 2.6.32-21 (or < 2.6.32-22.35) when I get back home (7 hours from now)

Best regards,
Norberto

Revision history for this message
James Hogarth (james-hogarth) wrote :

I haven't tried 2.6.32-22.33 yet (it is in the proposed apt repository for lucid still I believe) but looking through the changelog for the kernel release last night .33 didn't include any changes that are kvm related but .35 (which was the one that made it to updates/security) included a fix for CVE-2010-0419 which is directly kvm related.... unfortunately I don't have a pure debian system available to test to see if the issue exists upstream as well.

Revision history for this message
James Hogarth (james-hogarth) wrote :

Okay just pulled down linux-headers-2.6.32-22-generic_2.6.32-22.33_amd64.deb and linux-image-2.6.32-22-generic_2.6.32-22.33_amd64.deb from https://launchpad.net/ubuntu/+source/linux/2.6.32-22.33/+build/1709563 and I can confirm my guests work on that kernel....

So something broke between 2.6.32-22.33 and 2.6.32-22.35 .... That kernel was deleted from the proposed repo already so you'll need to download it from launchpad and install manually - and remember to do dkms if you need to to be on the safe side... (if you use and dkms based module such as nividia current)...

tags: added: kernel-core kernel-needs-review
Changed in linux (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Kees Cook (kees) wrote :

Thank you to everyone who has reported this issue. We have confirmed the regression and identified the patch that caused the problem. It has been reverted and confirmed to solve the problem. A new kernel package is currently being built and will be available soon. We apologize for the inconvenience.

Changed in linux (Ubuntu):
assignee: nobody → Andy Whitcroft (apw)
status: Triaged → Fix Committed
Revision history for this message
James Hogarth (james-hogarth) wrote :

Thanks for the quick turn around...

One thing that does concern me is why the kernel never went through proposed... or do I just misunderstand the point of the proposed apt repository?

I have no problem running my desktop on proposed and filing bug reports before they hit updates/security but if something core like the kernel never hit proposed in the first place it defeats the purpose of changing my apt sources it would seem....

Which part was reverted... or more specifically is the CVE identified involved and is the fix for that reverted in the kernel?

Revision history for this message
StiveG (stive-z) wrote :

Could you please post the new kernel package link here when ready?

Because, right now, the kernel won't install, I can't remove it and I can't downgrade neither...

Pretty stuck, kind of...

Thanks.

Revision history for this message
StiveG (stive-z) wrote :

Well IF my problem is the same as you guys...

When I try to update the kernel, the update process freeze after "found memtest86+ image"... And then, nothing happens.

If I try to remove the kernel, I get a "Sub-process /usr/bin/dpkg returned an error code (1)"

Of course, I can't install an older version neither (the new one isn't remove)...

If this is not the same problem, could you please tell me if that bug has been report? Can't find it and not sure what bug I must be looking for, to be honest...

Revision history for this message
Nathan Lutchansky (nathan-launchpad) wrote :

I'll echo James Hogarth's concerns here that whatever QA happened with this update, if any, obviously didn't include any QEMU/KVM testing. After updating to 2.6.32-22.35, I was unable to get qemu-kvm to run at all, regardless of which options I used. Didn't anybody try running KVM on the new kernel after applying a KVM-related patch?

My VM host was down for an hour this afternoon as a result of applying what was supposed to be a "long-term stable" update. I was bit last year by a KVM update with an obvious data loss bug (see bug #404394), so after today's bug, I'm regretting my choice of Ubuntu for my VM hosts.

Revision history for this message
c.h. (wryfi) wrote :

I too would like to know how in the world this update skipped the -proposed repo. Kernel updates not being qa'd in an LTS release is ... scary.

Revision history for this message
Kees Cook (kees) wrote :

Security updates are QA'd through a separate process (-updates goes through -proposed). Mistakes were made when doing the testing for these updates (note that more than just 10.04's kernel was tested and published, and that not all hardware fails to run KVM with the 10.04 kernel).

The issue was this commit, which has been reverted for the current builds. ETA should be close to 4 hours:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commitdiff;h=0a5699fe34fbc179242d717c61a0592dd50065dc

Installing the prior kernel should be a viable work-around:
  sudo apt-get install linux-image-2.6.32-21-generic

Again, we apologize for the inconvenience.

Revision history for this message
StiveG (stive-z) wrote :

Ok, I don't think I had the same problem then you guys but anyway, for my freezing problem (update fail on the grub part), here what I did :

Restart.
Choose 2.6.32-22 (recovery mode)
Repair broken packages
Update grub bootloader
Resume normal boot

- You will be at the command line at this point.

Reboot

Everything should be ok.

Sorry if i'm not in the right bug but since I talked about my problem here...

Revision history for this message
James Hogarth (james-hogarth) wrote :

Thanks for the quick triage and fix :-)

Mistakes happen... it's inevitable... the important thing is how they are dealt with ;-)

Besides we test updates on non-critical systems that can be rolled back *before* pushing out to important systems for a reason right? ;-)

Incidentally does this mean that cve for kvm applies to the kernel about to hit lucid still? And if so does a new bug need to be raised to cover that security issue?

Revision history for this message
Martin (martin00) wrote :

Still no fix?

I'm using KVM at 3 servers. Very bad, there was no Q/A. So many problems with KVM and ubuntu. Now I'm thinking to change the distri.

Revision history for this message
Kees Cook (kees) wrote :
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Kees Cook (kees) wrote :

As presently understood, CVE-2010-0419 is relatively minor (guest crash from a privileged guest process -- not really a privilege escalation). As it is re-triaged, a better fix will be developed, tested, and published.

Revision history for this message
James Hogarth (james-hogarth) wrote :

Fair enough - thanks for all the quick feedback :)

Revision history for this message
hans maurer (hjm-pmeonline) wrote :

I've downgraded the kernel as Kees Cook mentioned.
>Installing the prior kernel should be a viable work-around:
> sudo apt-get install linux-image-2.6.32-21-generic
Stiill no success with creating a VM.
Is there someting else to do? Thanks.

Revision history for this message
Martin (martin00) wrote :

2.6.32-22.36 (server) still generates 100% load. 2.6.32-22.36 (preempt) is not working. only 2.6.32-21.32 works with apparmor reload. not very satisfying.

Revision history for this message
Gremmlin (rogrem) wrote :

Since upgrading to the newer kernel I have tried different versions but can't get my kvm working again. Even the above mentioned kernel versions didn't work for me. @Martin: how exactly apparmor has influenced your setup. I even tried to move all the libvirt related profiles to disabled or stopped apparmor at all. Nothing brought back my kvm-machines to boot. Every hint will be appreciated.

Revision history for this message
Gremmlin (rogrem) wrote :

Shame on me. Belief it or not, I just haven't loaded the kvm/kvm-amd modules proberly! Perhaps someone else has overseen that detail too ;-)

Revision history for this message
Bo Thorsen (bo.thorsen) wrote :

The problem is back. Just installed Kubuntu 10.04 yesterday, completely updated. Ran this:

kvm -m 1023 -hda xp-64-clean.qcow2 -cdrom isos/xp-64.iso -boot d -cpu qemu64 -net nic,model=ne2k_pci -net user -redir tcp:2222:22

And it gave me this:

"Could not open option rom 'pxe-ne2k_pci.bin': No such file or directory"

I downloaded the rom file and placed that in /usr/share/kvm and the problem went away. But that shouldn't be necessary.

Changed in linux (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Chris Jones (cmsj) wrote :

Bo: Install kvm-pxe. Nothing to do with this bug, but you might consider a new bug (or finding an existing one) about turning that Suggests into a Recommends.

Changed in linux (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.