kvm crash when using virtio for network, hardy guest

Bug #458521 reported by trunet
288
This bug affects 4 people
Affects Status Importance Assigned to Milestone
QEMU
Fix Released
Medium
Anthony Liguori
Release Notes for Ubuntu
Fix Released
Undecided
Dustin Kirkland 
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Hardy
Fix Released
High
John Johansen
Karmic
Fix Released
Undecided
Unassigned
qemu-kvm (Ubuntu)
Fix Released
High
Dustin Kirkland 
Hardy
Invalid
Undecided
Unassigned
Karmic
Fix Released
High
Dustin Kirkland 

Bug Description

Binary package hint: kvm

I was running a kvm host with 5 virtual machines using jaunty. 1 running centos 5.4, 3 running karmic and 1 running hardy.

I upgraded the host to karmic to test it and one of my virtual machines(the vm with hardy) crashs after some seconds of boot! I stack traced the kvm process and found a "virtio-net truncating packet" before process crash.

No error in any log file. The VM just crash.

I changed the model of network interface(the disk is still virtio) in VM.xml to rtl8139, redefined and started. The problem goes away.

I don't know if it's a kvm bug or libvirt(regarding the bridge network) bug but definitely is a bug. All other VMs are running OK with virtio for both network and disk.

===========================
Karmic Release Notes:

KVM Guest Crashes when Guest is Hardy and using Virtio Networking

Ubuntu 8.04 LTS (Hardy) KVM guests using virtio networking may crash, when running on top of Ubuntu 9.10 (Karmic) hosts.

As a workaround, such guests should use either e1000 or rtl839 as the networking model. A fix for the bug is currently in progress and should be addressed in an update to the qemu-kvm package in Karmic.

===========================

===========================
SRU Justification

This bug is a regression from the kvm-84 package in 9.04. Karmic users hosting 8.04 KVM guests and using virtio networking *will* crash their VM when the network connection is saturated. As the virtual machine crashes without sync buffers, there could very well be data lost in the guest filesystem or memory.
I have posted the patch for review, and acked upstream:
 * http://lists.gnu.org/archive/html/qemu-devel/2009-10/msg02495.html
I am bundling this fix with two other minor fixes, for Bug #452323 and Bug #453441.

TEST CASE:
 1) Download and uncompress
  * http://rookery.canonical.com/~kirkland/hardy.img.bz2
 2) Download a bridged networking script
  * http://rookery.canonical.com/~kirkland/bridge.sh
 3) Start the vm with virtio and bridged networking
  * sudo /usr/bin/kvm -m 512 -net nic,model=virtio -net tap,script=bridge.sh -drive file=hardy.img,if=virtio,index=0,boot=on
 4) The VM's username and password are both "ubuntu".
 5) In the guest, have nc listen on a port
  * nc -lp 1234 > /dev/null
 6) On your host, flood the guest with network traffic on that port
  * cat /dev/urandom | nc -w 3 guest_ip 1234
 7) Without this fix, the guest will crash immediately, saying:
  * virtio-net truncating packet
 8) With the fix, you will be able to send the guest with data over the virtio network ad nauseum. I get about 6MB/s throughput.

===========================

Revision history for this message
trunet (wsartori) wrote :
Mathias Gug (mathiaz)
Changed in kvm (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

I also upgraded my VM host to karmic this weekend.

My hardy guest (which generally has months of uptime), cannot stay up for more than a few hours (~11 hours) before crashing.

I'm looking into this.

:-Dustin

affects: kvm (Ubuntu) → qemu-kvm (Ubuntu)
Changed in qemu-kvm (Ubuntu):
status: New → Confirmed
Changed in qemu:
importance: Undecided → Medium
Changed in qemu-kvm (Ubuntu Karmic):
milestone: none → karmic-updates
importance: Medium → High
assignee: nobody → Dustin Kirkland (kirkland)
Changed in qemu:
status: New → Confirmed
assignee: nobody → Anthony Liguori (anthony-codemonkey)
summary: - kvm crash when using virtio for network
+ kvm crash when using virtio for network, hardy guest
Changed in ubuntu-release-notes:
status: New → In Progress
assignee: nobody → Dustin Kirkland (kirkland)
description: updated
Mathias Gug (mathiaz)
Changed in ubuntu-release-notes:
status: In Progress → Fix Committed
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Okay, I can reproduce this very easily by saturating the guest's INCOMING network connection, with:

user@guest$ nc -lp 1234 > /dev/null

and

user@host$ cat /dev/urandom | nc -w 3 guest_ip 1234

In less than a second of sending, the vm crashes with:
  virtio-net truncating packet

Note that I have not reproduced the problem saturating the guest's OUTGOING network.

I have uploaded a full-updated Ubuntu 8.04.3 (Hardy) image (user=ubuntu,pw=ubuntu) that I'm using to reproduce this at:
 * http://rookery.canonical.com/~kirkland/hardy.img.bz2

:-Dustin

Revision history for this message
Steve Langasek (vorlon) wrote :

Documented at <https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#Ubuntu%208.04%20LTS%20crashes%20as%20a%20KVM%20guest%20when%20using%20virtio%20networking>:

Ubuntu 8.04 LTS using virtio networking as a KVM guest may crash when running on an Ubuntu 9.10 host. As a workaround, such guests should use either e1000 or rtl839 as the networking model. A fix for the bug is currently in progress and will be included in an update to the qemu-kvm package in Karmic. (Bug:458521)

Changed in ubuntu-release-notes:
status: Fix Committed → Fix Released
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

I have a fix for this.

Patch attached. Sent to upstream mailing list. I'm going to start rolling an SRU.

:-Dustin

Changed in qemu-kvm (Ubuntu Karmic):
status: Confirmed → In Progress
description: updated
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Patch attached for SRU review.

Note that two other bug fixes will be uploaded along with this one.

:-Dustin

Changed in qemu:
status: Confirmed → In Progress
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Attaching the full debdiff, which fixes 3 bugs, for SRU review.

:-Dustin

Changed in qemu-kvm (Ubuntu Karmic):
status: In Progress → Fix Committed
Martin Pitt (pitti)
Changed in qemu-kvm (Ubuntu):
milestone: karmic-updates → lucid-alpha-1
Revision history for this message
trunet (wsartori) wrote :

I patched the hardy kernel and it's working great.

Thanks for the fix.

Wagner Sartori Junior

Martin Pitt (pitti)
Changed in qemu-kvm (Ubuntu):
status: In Progress → Fix Committed
milestone: lucid-alpha-1 → none
Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 458521] Re: kvm crash when using virtio for network, hardy guest

On Thu, Oct 29, 2009 at 1:38 PM, trunet <email address hidden> wrote:
> I patched the hardy kernel and it's working great.

Huh?

This isn't a patch for the hardy kernel...

:-Dustin

Revision history for this message
trunet (wsartori) wrote :

sorry the qemu-kvm package! ;)

Thanks,

Wagner Sartori Junior

Revision history for this message
Martin Pitt (pitti) wrote :

Accepted qemu-kvm into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

This bug is actually security-critical. I'm copying the Ubuntu-Security team, and updating the changelog for security update publication.

Basically, a guest running virtio networking and a linux kernel <= 2.6.25 (eg hardy) can be remotely DoS'd by any other user on the network flooding an open network port (eg 22) with traffic. The DoS causes crash in the kvm process. Because the guest's operating system was not shut down cleanly, unwritten buffers or dirty filesystem operations could yield data loss in the guest.

Attaching a debdiff for the security team to upload to the security-prosed pocket.

:-Dustin

security vulnerability: no → yes
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

I just added a new task, triaged/high against hardy's linux kernel.

The Hardy linux kernel uses a backport of 2.6.25's virtio-net implementation that causes this bug.

This virtio-net driver needs to be fixed such that hardy vm guests using virtio-net doesn't truncate packets and crash hypervisors.

This appears to be fixed in virtio-net implementations later than 2.6.25, so I think we either need to sync this driver with a new release, or cherry-pick the fixes that solve this.

Assigning to jj. Hopefully he can knock this out easily.

:-Dustin

Changed in linux (Ubuntu Karmic):
status: New → Fix Released
Changed in linux (Ubuntu Hardy):
status: New → Triaged
Changed in linux (Ubuntu):
status: New → Fix Released
Changed in linux (Ubuntu Hardy):
importance: Undecided → High
Changed in qemu-kvm (Ubuntu Hardy):
status: New → Invalid
Changed in linux (Ubuntu Hardy):
assignee: nobody → John Johansen (jjohansen)
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Okay, I ran these tests myself, and verified that the current package in karmic-proposed solves the issue.

Can I get some help from someone else out there? Ideally, the original bug reporter will be able to validate the fix. Trunet?

Revision history for this message
trunet (wsartori) wrote :

I enabled karmic-proposed and installed the qemu-kvm and kvm package from there. The problem goes away and my VMs are ok for 8 hours from now with virtio in network interfaces.

thanks,

Wagner Sartori Junior

Revision history for this message
Michael Jeanson (mjeanson) wrote :

I installed the package from proposed and it solved the problem for me. I ran the test provided in this bug report and some file tranfers with scp that used to crash and it worked flawlessly.

Martin Pitt (pitti)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu6.3

---------------
qemu-kvm (0.11.0-0ubuntu6.3) karmic-security; urgency=low

  * SECURITY UPDATE: linux <= 2.6.25 guests (e.g. hardy) with virtio
    networking are subject to DoS by qemu-kvm application crash;
    the crash can be remotely triggered by a malicious user flooding any
    open network port (LP: #458521)
    - debian/patches/12_whitelist_host_virtio_networking_features.patch:
      fix accounting of virtio networking features available to make
      available to the guests
    - CVE-2009-XXXX
  * debian/kvm-ok: check for other common reasons why KVM might not be
    usable, LP: #452323
  * debian/control: build-depend on libcurl devel, to allow booting from
    ISOs over http, LP: #453441

 -- Dustin Kirkland <email address hidden> Thu, 29 Oct 2009 11:36:18 -0500

Changed in qemu-kvm (Ubuntu Karmic):
status: Fix Committed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote :

Copied karmic-proposed to lucid.

Changed in qemu-kvm (Ubuntu):
status: Fix Committed → Fix Released
Changed in qemu-kvm (Ubuntu Karmic):
status: Fix Released → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu6.3

---------------
qemu-kvm (0.11.0-0ubuntu6.3) karmic-security; urgency=low

  * SECURITY UPDATE: linux <= 2.6.25 guests (e.g. hardy) with virtio
    networking are subject to DoS by qemu-kvm application crash;
    the crash can be remotely triggered by a malicious user flooding any
    open network port (LP: #458521)
    - debian/patches/12_whitelist_host_virtio_networking_features.patch:
      fix accounting of virtio networking features available to make
      available to the guests
    - CVE-2009-XXXX
  * debian/kvm-ok: check for other common reasons why KVM might not be
    usable, LP: #452323
  * debian/control: build-depend on libcurl devel, to allow booting from
    ISOs over http, LP: #453441

 -- Dustin Kirkland <email address hidden> Thu, 29 Oct 2009 11:36:18 -0500

Changed in qemu-kvm (Ubuntu Karmic):
status: Fix Committed → Fix Released
Revision history for this message
mapl (mpierre) wrote :

I installed Hardy as a guest on KVM on RH 5.4 using bridge networking. System will reboot as soon as I tried to pull in a file via either ftp, nfs or scp. Just changed back to virtual networking and the problem goes away. This severly limits my ability to have the guest be accessed externally.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@John, what is the status of this bug on Hardy/linux?

Changed in qemu:
status: In Progress → Fix Released
Changed in linux (Ubuntu Hardy):
status: Triaged → Incomplete
Changed in linux (Ubuntu Hardy):
status: Incomplete → In Progress
Revision history for this message
John Johansen (jjohansen) wrote :

I went back and retested this and it has already been fixed and released (commit fcbc05a1be0a7600153e78207dcb8b62fe753a4a), it was just not properly closed.

mapl,
If you are running an updated hardy guest running the 2.6.24-29 kernel this bug should not be a problem. Can you please provide information on which kernel you are using. If you are still encountering a problem with an updated kernel this is a new bug and we will need to gather more information to reproduce.

Changed in linux (Ubuntu Hardy):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.