Ubuntu
libvirt package

Using LVM as storage pool for libvirt causes kernel panic's

Bug #1388329 reported by mard0
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Expired
Undecided
Unassigned
linux (Ubuntu)
Expired
Medium
Unassigned

Bug Description

When operating in a VM through virtual machine manager I get some random kernel panics in the host related tot libvirt.
After the panic whole libvirt hangs. This usually happens when trying to shutdown the VM, but also during normal use. Attached to this bug report is the dmesg output.

[ 1805.098453] virbr0: port 1(vnet0) entered disabled state
[ 1805.099042] device vnet0 left promiscuous mode
[ 1805.099054] virbr0: port 1(vnet0) entered disabled state
[ 1805.827706] audit: type=1400 audit(1414827748.381:39): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-147c7c68-d1d6-4dd0-8753-f13473e3396d" pid=8452 comm="apparmor_parser"
[ 1805.829964] general protection fault: 0000 [#1] SMP
[ 1805.829967] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc vhost_net vhost macvtap macvlan rfcomm bnep binfmt_misc joydev hid_generic btusb bluetooth usbhid hid 6lowpan_iphc btrfs xor kvm_amd snd_seq_midi snd_seq_midi_event kvm snd_hda_codec_via crct10dif_pclmul snd_hda_codec_generic snd_rawmidi crc32_pclmul snd_hda_codec_hdmi ghash_clmulni_intel snd_hda_intel snd_seq snd_hda_controller snd_hda_codec aesni_intel uas snd_hwdep aes_x86_64 snd_pcm lrw gf128mul glue_helper ablk_helper snd_seq_device psmouse usb_storage cryptd sp5100_tco nvidia(POE) serio_raw edac_core i2c_piix4 edac_mce_amd snd_timer k10temp fam15h_power r8169 tpm_infineon
[ 1805.830000] snd mii drm soundcore mac_hid shpchp vfio_iommu_type1 vfio_pci vfio it87 hwmon_vid raid6_pq nls_iso8859_1 ahci libahci
[ 1805.830009] CPU: 0 PID: 1602 Comm: libvirtd Tainted: P OE 3.16.0-24-generic #32-Ubuntu
[ 1805.830011] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-UD3P, BIOS F1 08/06/2013
[ 1805.830013] task: ffff88044690b2f0 ti: ffff880445dec000 task.ti: ffff880445dec000
[ 1805.830014] RIP: 0010:[<ffffffff814d88cf>] [<ffffffff814d88cf>] __rpm_callback+0x2f/0xe0
[ 1805.830019] RSP: 0018:ffff880445defd80 EFLAGS: 00010212
[ 1805.830020] RAX: 0000000000000008 RBX: ffff880447c2f098 RCX: 0000000000000000
[ 1805.830021] RDX: ffff8803d8a15368 RSI: ffff880447c2f098 RDI: ffff880447c2f098
[ 1805.830023] RBP: ffff880445defd98 R08: 0000000023670101 R09: 0000000000000246
[ 1805.830024] R10: ffffffffc028b148 R11: ffff880423670100 R12: 0016000000600460
[ 1805.830025] R13: ffff880447c2f148 R14: 0000000000000000 R15: fffffffffffffff2
[ 1805.830027] FS: 00007f4c8eeb7700(0000) GS:ffff88045ec00000(0000) knlGS:0000000000000000
[ 1805.830028] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1805.830029] CR2: 00007f4c801cc3d0 CR3: 0000000445df3000 CR4: 00000000000407f0
[ 1805.830030] Stack:
[ 1805.830031] 0000000000000004 ffff880447c2f098 0000000000000004 ffff880445defdc8
[ 1805.830034] ffffffff814d9c6d 0000000000000004 ffff880447c2f098 ffff880447c2f148
[ 1805.830036] 0000000000000246 ffff880445defdf8 ffffffff814d9dbd ffff880447c2f098
[ 1805.830038] Call Trace:
[ 1805.830041] [<ffffffff814d9c6d>] rpm_idle+0x1bd/0x2b0
[ 1805.830044] [<ffffffff814d9dbd>] __pm_runtime_idle+0x5d/0x80
[ 1805.830048] [<ffffffff813e36d4>] pci_device_remove+0x74/0xc0
[ 1805.830052] [<ffffffff814cf76f>] __device_release_driver+0x7f/0xf0
[ 1805.830055] [<ffffffff814cf803>] device_release_driver+0x23/0x30
[ 1805.830058] [<ffffffff814ceb5d>] unbind_store+0xed/0x150
[ 1805.830061] [<ffffffff814cd915>] drv_attr_store+0x25/0x40
[ 1805.830064] [<ffffffff81258eea>] sysfs_kf_write+0x3a/0x50
[ 1805.830066] [<ffffffff8125808e>] kernfs_fop_write+0xee/0x180
[ 1805.830069] [<ffffffff811e0c17>] vfs_write+0xb7/0x1f0
[ 1805.830071] [<ffffffff811dfd89>] ? do_sys_open+0x1b9/0x280
[ 1805.830074] [<ffffffff811e17c6>] SyS_write+0x46/0xb0
[ 1805.830087] [<ffffffff81787ced>] system_call_fastpath+0x1a/0x1f
[ 1805.830088] Code: 90 55 48 89 e5 41 55 41 54 53 f6 86 91 01 00 00 02 49 89 fc 48 89 f3 4c 8d ae b0 00 00 00 74 2c 4c 89 ef e8 34 ef 2a 00 48 89 df <41> ff d4 f6 83 91 01 00 00 02 41 89 c4 4c 89 ef 75 77 e8 5a ea
[ 1805.830111] RIP [<ffffffff814d88cf>] __rpm_callback+0x2f/0xe0
[ 1805.830113] RSP <ffff880445defd80>
[ 1805.830115] ---[ end trace 1759db886910146f ]---

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: libvirt-bin 1.2.8-0ubuntu11
ProcVersionSignature: Ubuntu 3.16.0-24.32-generic 3.16.4
Uname: Linux 3.16.0-24-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.14.7-0ubuntu8
Architecture: amd64
CurrentDesktop: XFCE
Date: Sat Nov 1 08:45:18 2014
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: [modified]
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: '/etc/libvirt/qemu.conf']
mtime.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: 2014-06-05T20:10:02.637036
---
ApportVersion: 2.14.7-0ubuntu8
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: mard0 2212 F.... pulseaudio
 /dev/snd/controlC0: mard0 2212 F.... pulseaudio
 /dev/snd/controlC2: mard0 2212 F.... pulseaudio
CurrentDesktop: XFCE
DistroRelease: Ubuntu 14.10
IwConfig:
 virbr0 no wireless extensions.

 eth3 no wireless extensions.

 lo no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
NonfreeKernelModules: nvidia_uvm nvidia
Package: linux (not installed)
ProcCmdline: root=UUID=dcbc226d-e5a4-4d61-a299-07bc3641b0f4 ro quiet iommu=pt iommu=1 rootfstype=ext4 add_efi_memmap initrd=\EFI\ubuntu\initrd.img
ProcFB: 0 EFI VGA
ProcKernelCmdLine: root=UUID=dcbc226d-e5a4-4d61-a299-07bc3641b0f4 ro quiet iommu=pt iommu=1 rootfstype=ext4 add_efi_memmap initrd=\EFI\ubuntu\initrd.img
ProcVersionSignature: Ubuntu 3.16.0-24.32-generic 3.16.4
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-24-generic N/A
 linux-backports-modules-3.16.0-24-generic N/A
 linux-firmware 1.138
RfKill:
 0: hci0: Bluetooth
  Soft blocked: no
  Hard blocked: no
Tags: utopic utopic
Uname: Linux 3.16.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: daemon kvm libvirtd lp lpadmin plugdev saned video
WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 08/06/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F1
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: 970A-UD3P
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF1:bd08/06/2013:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rn970A-UD3P:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
modified.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: [modified]
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: u'/etc/libvirt/qemu.conf']
mtime.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: 2014-06-05T20:10:02.637036
---
ApportVersion: 2.14.7-0ubuntu8
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: mard0 2212 F.... pulseaudio
 /dev/snd/controlC0: mard0 2212 F.... pulseaudio
 /dev/snd/controlC2: mard0 2212 F.... pulseaudio
CurrentDesktop: XFCE
DistroRelease: Ubuntu 14.10
IwConfig:
 virbr0 no wireless extensions.

 eth3 no wireless extensions.

 lo no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
NonfreeKernelModules: nvidia_uvm nvidia
Package: linux (not installed)
ProcCmdline: root=UUID=dcbc226d-e5a4-4d61-a299-07bc3641b0f4 ro quiet iommu=pt iommu=1 rootfstype=ext4 add_efi_memmap initrd=\EFI\ubuntu\initrd.img
ProcFB: 0 EFI VGA
ProcKernelCmdLine: root=UUID=dcbc226d-e5a4-4d61-a299-07bc3641b0f4 ro quiet iommu=pt iommu=1 rootfstype=ext4 add_efi_memmap initrd=\EFI\ubuntu\initrd.img
ProcVersionSignature: Ubuntu 3.16.0-24.32-generic 3.16.4
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-24-generic N/A
 linux-backports-modules-3.16.0-24-generic N/A
 linux-firmware 1.138
RfKill:
 0: hci0: Bluetooth
  Soft blocked: no
  Hard blocked: no
Tags: utopic utopic
Uname: Linux 3.16.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: daemon kvm libvirtd lp lpadmin plugdev saned video
WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 08/06/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F1
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: 970A-UD3P
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF1:bd08/06/2013:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rn970A-UD3P:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
modified.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: [modified]
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission denied: u'/etc/libvirt/qemu.conf']
mtime.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: 2014-06-05T20:10:02.637036

See original description
Revision history for this message
mard0 (martijn-doornbusch-l) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1388329

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
mard0 (martijn-doornbusch-l) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
mard0 (martijn-doornbusch-l) wrote : BootDmesg.txt

apport information

description: updated
Revision history for this message
mard0 (martijn-doornbusch-l) wrote : AlsaInfo.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : BootDmesg.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : CRDA.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : CurrentDmesg.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : Lspci.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : Lsusb.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : ProcEnviron.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : ProcInterrupts.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : ProcModules.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : PulseList.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : UdevDb.txt

apport information

Revision history for this message
mard0 (martijn-doornbusch-l) wrote : UdevLog.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
penalvch (penalvch) wrote : Re: Libvirt causes random kernel panics

Mard0, thank you for reporting this and helping make Ubuntu better. Could you please test the latest upstream kernel available from the very top line at the top of the page (not the daily folder) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested exactly shown as:
kernel-fixed-upstream-3.18-rc2

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description.

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Incomplete
tags: added: latest-bios-f1
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

[113920.261448] usb usb9-port4: Cannot enable. Maybe the USB cable is bad?
[113922.819406] usb usb9-port4: Could not disable after 2000 ms
[113922.931620] usb 9-4: USB disconnect, device number 3
[113922.935163] scsi 7:0:0:0: rejecting I/O to offline device
[113922.935170] scsi 7:0:0:0: [sdd] killing request
[113922.935197] scsi 7:0:0:0: [sdd] Unhandled error code
[113922.935199] scsi 7:0:0:0: [sdd]
[113922.935201] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[113922.935202] scsi 7:0:0:0: [sdd] CDB:
[113922.935204] Read(10): 28 00 00 00 00 00 00 00 08 00
[113922.935210] end_request: I/O error, dev sdd, sector 0
[113922.935212] Buffer I/O error on device sdd, logical block 0
[113922.935243] ldm_validate_partition_table(): Disk read failed.
[113922.935252] Dev sdd: unable to read RDB block 0
[113922.935264] sdd: unable to read partition table
[113922.935269] sdd: partition table beyond EOD, enabling native capacity

Given this in CurrentDmesg it looks like here is a bad disk or bad usb cable (or both) involved.

Revision history for this message
mard0 (martijn-doornbusch-l) wrote :

The error about the malfunctioning USB device was actually a SD-card reader that was connected but was missing a card to read. I removed it and got a whole new error. This time not the "general protection fault" but "BUG: unable to handle kernel paging request".

Also, I think the bug might have something to do with the use of LLVM as storage device for the VM. I have one VM with a raw disk image and I have yet to crash that VM.

I will try to report the results of the mainline kernel as soon as possible.

Revision history for this message
mard0 (martijn-doornbusch-l) wrote :

I tried kernel 3.18-rc3 and this completely hung the host.

tags: added: kernel-bug-exists-upstream-3.18-rc3
Revision history for this message
mard0 (martijn-doornbusch-l) wrote :

I can confirm that the bug is caused by the use of LVM as storagepool. I copied a VM and changed the hard disk into an image on the physical drive that I dd'ed from the original LVM disk. With this configuration I was unable to reproduce the bug.

mard0 (martijn-doornbusch-l)
summary: - Libvirt causes random kernel panics
+ Using LVM as storage pool for libvirt causes kernel panic's
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
penalvch (penalvch) wrote :

mard0, did this problem not occur in a release prior to Utopic?

As well, could you please test the latest mainline kernel 3.18-rc4 and advise to the results?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Serge Hallyn (serge-hallyn)
Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
Dominique Ramaekers (dominique-ramaekers) wrote :

I've got a similar situation.

While saving (pauzing) a VM the system crashed. I'm also using LVM as storage pool. Actualy my .raw files are on a LVM logical volume.

I couldn't apport-collect, so I used apport-bug. => Bug#1398883

mard0 (martijn-doornbusch-l)
tags: added: kernel-bug-exists-upstream-3.18.0
Revision history for this message
mard0 (martijn-doornbusch-l) wrote :

Sorry for the long wait, I haven't gotten allot of time to test. I did however tried some things:

- The bug didn't exist in Trusty, but after I was able to reproduce the bug in kernel 3.15.10
- I also tried inducing the system to crash through pausing the VM like described by Dadio, but to no success

Revision history for this message
mard0 (martijn-doornbusch-l) wrote :

I'm wondering if the kernel panic's aren't from the use of LVM but might be from GPU pass through and the vdio module. I wasn't able to get that to work in Trusty.

Revision history for this message
penalvch (penalvch) wrote :

mard0, just to clarify, was there a 3.13.x kernel from the list you could boot into that doesn't reproduce the problem?

tags: added: kernel-bug-exists-upstream-3.18
removed: kernel-bug-exists-upstream-3.18-rc3 kernel-bug-exists-upstream-3.18.0
tags: added: regression-release
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

Changed in linux (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.