apparmor profile prevents libvirtd from creating a socket

Bug #1386465 reported by Christian Kirbach
78
This bug affects 15 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Expired
High
Unassigned

Bug Description

I'd like to emphasize that I upgraded from Ubuntu Gnome 14.4 to 14.10
I installed systemd.

libvirtd fails to start on the stock upgrade system. Examining the log files it looks like apparmor prevents libvirtd from creating a net socket.

Okt 28 00:31:49 rivendell kernel: audit: type=1400 audit(1414452709.808:42): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=15162 comm="libvirtd" family="netlink" sock_type="raw" protocol=9
Okt 28 00:31:49 rivendell kernel: audit: type=1400 audit(1414452709.808:43): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=15162 comm="libvirtd" family="netlink" sock_type="raw" protocol=0

I tried to run

 aa-logprof

in order to have apparmor fix the permissions but that did not work. that is why modified apparmor profiles are attached.

Putting apparmor in audit mode

 aa-audit /usr/sbin/libvirtd

enables me to start libvirt

 systemctl restart libvirtd

It looks like the apparmor profile permissions have to be adjusted.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: libvirt-bin 1.2.8-0ubuntu11
ProcVersionSignature: Ubuntu 3.16.0-23.31-generic 3.16.4
Uname: Linux 3.16.0-23-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.14.7-0ubuntu8
Architecture: amd64
CurrentDesktop: GNOME
Date: Tue Oct 28 01:20:45 2014
InstallationDate: Installed on 2013-01-08 (657 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
KernLog:

SourcePackage: libvirt
UpgradeStatus: Upgraded to utopic on 2014-10-23 (4 days ago)
modified.conffile..etc.apparmor.d.usr.sbin.libvirtd: [modified]
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Keine Berechtigung: '/etc/libvirt/qemu.conf']
mtime.conffile..etc.apparmor.d.usr.sbin.libvirtd: 2014-10-28T00:33:09.824586

Revision history for this message
Christian Kirbach (christian-kirbach-e) wrote :
Changed in libvirt (Ubuntu):
status: New → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug. The default libvirt profile does provide 'network netlink', so I'm not sure why you are having this problem.

Could you please attach the /etc/apparmor.d/usr.sbin.libvirtd from a fresly updated host that is having this issue?

Changed in libvirt (Ubuntu):
importance: Undecided → High
status: Confirmed → Incomplete
Revision history for this message
Christian Kirbach (christian-kirbach-e) wrote :

I did not have libvirt-bin installed before the upgrade, I installed it afterwards.

I renamed /etc/apparmor.d/usr.sbin.libvirtd , purged libvirt-bin and reinstalled it.
to my surprise the diff between /etc/apparmor.d/usr.sbin.libvirtd and the renamed file is zero.

For some reason I am no longer able to reproduce the issue. apparmor is set to enforcing

root@rivendell:/etc/apparmor.d# aa-status --verbose
apparmor module is loaded.
24 profiles are loaded.
24 profiles are in enforce mode.
...
   /usr/sbin/libvirtd

thanks for your efforts

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1386465] Re: apparmor profile prevents libvirtd from creating a socket

Thanks you for the information - that's quite frustrating. It seems quite
clear (between this bug and some others) that there is a hard-to-trigger
bug in the libvirt apparmor policy, but I've not yet spotted any obvious
trigger.

I'm marking this bug invalid meaning "cannot be reproduced", but if
anyone sees anything like it again please to mark it confirmed and
add what informatino you can.

 status: invalid

Changed in libvirt (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Ben Schweikert (b-schweikert) wrote :

Hi,
I have similar problems after upgrading my KVM host from 14.04 to 14.10. I cannot start libvirt. In the libvirt log I see these errors:

2014-11-07 08:46:15.137+0000: 4550: error : virAuditOpen:62 : Unable to initialize audit layer: Permission denied
2014-11-07 08:46:15.138+0000: 4550: error : virNetlinkEventServiceStart:544 : cannot connect to netlink socket with protocol 0: Permission denied

Only workaround is the mentioned
aa-audit /usr/sbin/libvirtd

I try to attach a apport

Revision history for this message
Fredrik Normann (smartypants) wrote :

I can reproduce this bug

Changed in libvirt (Ubuntu):
status: Invalid → Confirmed
Revision history for this message
Fredrik Normann (smartypants) wrote :
Download full text (3.5 KiB)

[49578.653469] audit: type=1400 audit(1420284820.446:128): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=24625 comm="apparmor_parser"
[49578.677733] audit: type=1400 audit(1420284820.470:129): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/libvirt/virt-aa-helper" pid=24627 comm="apparmor_parser"
[49578.833315] audit: type=1400 audit(1420284820.622:130): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24713 comm="libvirtd" family="netlink" sock_type="raw" protocol=9
[49578.835134] audit: type=1400 audit(1420284820.626:131): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24713 comm="libvirtd" family="netlink" sock_type="raw" protocol=0
[49578.838352] init: libvirt-bin main process (24713) terminated with status 6
[49578.838367] init: libvirt-bin main process ended, respawning
[49578.881341] audit: type=1400 audit(1420284820.670:132): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24743 comm="libvirtd" family="netlink" sock_type="raw" protocol=9
[49578.883046] audit: type=1400 audit(1420284820.674:133): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24743 comm="libvirtd" family="netlink" sock_type="raw" protocol=0
[49578.885421] init: libvirt-bin main process (24743) terminated with status 6
[49578.885431] init: libvirt-bin main process ended, respawning
[49578.983903] audit: type=1400 audit(1420284820.774:134): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24773 comm="libvirtd" family="netlink" sock_type="raw" protocol=9
[49578.985971] audit: type=1400 audit(1420284820.778:135): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24773 comm="libvirtd" family="netlink" sock_type="raw" protocol=0
[49578.988992] init: libvirt-bin main process (24773) terminated with status 6
[49578.989004] init: libvirt-bin main process ended, respawning
[49579.032821] audit: type=1400 audit(1420284820.822:136): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24792 comm="libvirtd" family="netlink" sock_type="raw" protocol=9
[49579.034446] audit: type=1400 audit(1420284820.826:137): apparmor="DENIED" operation="create" profile="/usr/sbin/libvirtd" pid=24792 comm="libvirtd" family="netlink" sock_type="raw" protocol=0
[49579.037827] init: libvirt-bin main process (24792) terminated with status 6
[49579.037843] init: libvirt-bin main process ended, respawning
[49579.084741] init: libvirt-bin main process (24812) terminated with status 6
[49579.084757] init: libvirt-bin main process ended, respawning
[49579.134834] init: libvirt-bin main process (24831) terminated with status 6
[49579.134844] init: libvirt-bin main process ended, respawning
[49579.182895] init: libvirt-bin main process (24850) terminated with status 6
[49579.182930] init: libvirt-bin main process ended, respawning
[49579.229375] init: libvirt-bin main process (24869) terminated with status 6
[49579.229388] init: libvirt-bin main process ended, respawning
[49579.275437] init: libvirt-bin main process (24888) terminated with status 6
[49579.275450] init: libvirt-bin m...

Read more...

Revision history for this message
Fredrik Normann (smartypants) wrote :

After aa-audit

[49992.022321] audit: type=1400 audit(1420285233.962:152): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=25593 comm="apparmor_parser"
[50004.285006] audit: type=1400 audit(1420285246.230:153): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/etc/ld.so.cache" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285015] audit: type=1400 audit(1420285246.230:154): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/etc/ld.so.cache" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285035] audit: type=1400 audit(1420285246.230:155): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-lxc.so.0.1002.8" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285045] audit: type=1400 audit(1420285246.230:156): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-lxc.so.0.1002.8" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285074] audit: type=1400 audit(1420285246.230:157): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-qemu.so.0.1002.8" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285080] audit: type=1400 audit(1420285246.230:158): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-qemu.so.0.1002.8" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285107] audit: type=1400 audit(1420285246.230:159): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-common.so.3.5.3" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285113] audit: type=1400 audit(1420285246.230:160): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-common.so.3.5.3" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50004.285142] audit: type=1400 audit(1420285246.230:161): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-client.so.3.2.9" pid=25618 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
[50005.905499] ip_tables: (C) 2000-2006 Netfilter Core Team
[50005.935465] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[50005.973242] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready

Revision history for this message
Fredrik Normann (smartypants) wrote :

Now I have my virbr0 interface and libvirt-bin starts

Revision history for this message
Jeff Burns (admiraljkb) wrote :

I had the same issue upgrading from 14.04 to 14.10. Purged/reinstallled libvirt-bin as in Message3, but with no effect. Installed apparmor-utils, and put libvirt into audit mode per first message, and libvirt works again. Thanks Christian for the workaround.

error from /var/log/libvirt/libvirtd.log:
2015-01-13 03:43:18.605+0000: 16428: info : libvirt version: 1.2.8, package: 1.2.8-0ubuntu11.2
2015-01-13 03:43:18.605+0000: 16428: error : virAuditOpen:62 : Unable to initialize audit layer: Permission denied
2015-01-13 03:43:18.607+0000: 16428: error : virNetlinkEventServiceStart:544 : cannot connect to netlink socket with protocol 0: Permission denied

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Jeff and @Smartypants,

are you both running systemd in 14.10, as the original bug reported was? Or are you running upstart?

Revision history for this message
datenteiler (datenteiler) wrote :

Same problem here: I have upgraded vom 14.04 to 14.10.

Putting apparmor in audit mode

 aa-audit /usr/sbin/libvirtd

enables me to start libvirt. I use Upstart:

$ ps -eaf|grep [u]pstart
root 541 1 0 18:57 ? 00:00:00 upstart-udev-bridge --daemon
root 1155 1 0 18:57 ? 00:00:00 upstart-socket-bridge --daemon
root 1171 1 0 18:57 ? 00:00:00 upstart-file-bridge --daemon
christi+ 2219 2209 0 18:57 ? 00:00:00 upstart --user
christi+ 2435 2219 0 18:57 ? 00:00:00 upstart-event-bridge
christi+ 2469 2219 0 18:57 ? 00:00:00 upstart-file-bridge --daemon --user
christi+ 2513 2219 0 18:57 ? 00:00:00 upstart-dbus-bridge --daemon --session --user --bus-name session
christi+ 2514 2219 0 18:57 ? 00:00:00 upstart-dbus-bridge --daemon --system --user --bus-name system

$ ps -eaf|grep [s]ystemd
root 426 1 0 18:57 ? 00:00:00 /sbin/cgmanager --sigstop -m name=systemd
root 549 1 0 18:57 ? 00:00:00 /lib/systemd/systemd-udevd --daemon
root 1214 1 0 18:57 ? 00:00:00 /lib/systemd/systemd-logind

$ sudo /sbin/init --version
init (upstart 1.13.2)

$ type init
init ist /sbin/init

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

The bug description shows you have a modified /etc/apparmor.d/usr.sbin.libvirtd. It seems possible that a package update was therefore not installed, preventing you from getting, for instance, the needed 'network netlink,' line.

Could you please attach your /etc/apparmor.d/usr.sbin.libvirtd file?

Changed in libvirt (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Christian Kirbach (christian-kirbach-e) wrote :

This is my libvirtd apparmor profile, however I have no issues any more at this time.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks - as you have no more issues i'll mark the bug 'invalid' meaning cannot currently be reproduced. If it happens again please re-open the bug.

Changed in libvirt (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Norberto Bensa (nbensa) wrote :

Hello. I'm having the same problem. I just upgraded to 14.10 (from 14.04).

'service libvirt-bin start' fails to start (althrough it gives a pid).

/var/log/libvirt/libvirtd.log

2015-03-10 03:22:13.546+0000: 10223: info : libvirt version: 1.2.8, package: 1.2.8-0ubuntu11.4
2015-03-10 03:22:13.546+0000: 10223: error : virAuditOpen:62 : Unable to initialize audit layer: Permission denied
2015-03-10 03:22:13.548+0000: 10223: error : virNetlinkEventServiceStart:544 : cannot connect to netlink socket with protocol 0: Permission denied

/etc/apparmor.d/usr.sbin.libvirtd

# Last Modified: Mon Jul 6 17:23:58 2009
#include <tunables/global>
@{LIBVIRT}="libvirt"

/usr/sbin/libvirtd {
  #include <abstractions/base>
  #include <abstractions/dbus>
  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.sbin.libvirtd>

  capability kill,
  capability net_admin,
  capability net_raw,
  capability setgid,
  capability sys_admin,
  capability sys_module,
  capability sys_ptrace,
  capability sys_nice,
  capability sys_chroot,
  capability setuid,
  capability dac_override,
  capability dac_read_search,
  capability fowner,
  capability chown,
  capability setpcap,
  capability mknod,
  capability fsetid,
  capability ipc_lock,
  capability audit_write,

  # Needed for vfio
  capability sys_resource,

  network inet stream,
  network inet dgram,
  network inet6 stream,
  network inet6 dgram,
  network packet dgram,
  network netlink,

  dbus bus=system,
  signal,
  ptrace,
  unix,

  # for now, use a very lenient profile since we want to first focus on
  # confining the guests
  / r,
  /** rwmkl,

  /bin/* PUx,
  /sbin/* PUx,
  /usr/bin/* PUx,
  /usr/sbin/* PUx,
  /lib/udev/scsi_id PUx,
  /usr/lib/xen-common/bin/xen-toolstack PUx,
  /usr/lib/xen-*/bin/pygrub PUx,
  /usr/lib/xen-*/bin/libxl-save-helper PUx,

  # Required by nwfilter_ebiptables_driver.c:ebiptablesWriteToTempFile() to
  # write and run an ebtables script.
  /var/lib/libvirt/virtd* ixr,

  # force the use of virt-aa-helper
  audit deny /sbin/apparmor_parser rwxl,
  audit deny /etc/apparmor.d/libvirt/** wxl,
  audit deny /sys/kernel/security/apparmor/features rwxl,
  audit deny /sys/kernel/security/apparmor/matching rwxl,
  audit deny /sys/kernel/security/apparmor/.* rwxl,
  /sys/kernel/security/apparmor/profiles r,
  /usr/lib/libvirt/* PUxr,
  /etc/libvirt/hooks/** rmix,
  /etc/xen/scripts/** rmix,

  # allow changing to our UUID-based named profiles
  change_profile -> @{LIBVIRT}-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*,

}

After 'aa-audit /usr/sbin/libvirtd' everything seems to work, but with a lot of chat in dmesg.

I can spend some time debugging this but I'll need someone to guide me.

Regards,
Norberto

Revision history for this message
Norberto Bensa (nbensa) wrote :

After 'aa-audir -r /usr/sbin/libvirtd' and a reboot, libvirtd works as expected (and with no chat in dmesg).

Anyway, I still offer my help in debugging this.

Revision history for this message
Antony Chen (tchen) wrote :

Having the same issue. Can't create the libvirt.sock when trying to start up libvirt.

By using "aa-audit /usr/sbin/libvirtd", I get a lot of chatter in dmesg:

[67418.135152] audit: type=1400 audit(1426857324.439:5864): apparmor="AUDIT" operation="file_perm" profile="/usr/sbin/libvirtd" name="/run/libvirt/libvirt-sock" pid=3057 comm="libvirtd" requested_mask="w" fsuid=0 ouid=0

I have "network netlink" in my usr.sbin.libvirtd configuration as well.

Could it be that for netlink, you have to specify TYPE (e.g. raw)? I know for the rest, having no subsequent parameter assumes "all", but perhaps for netlink, it's changed behavior. I'll test and report back.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@nbensa,

I'm sorry, Idon't know what aa-audir is.

@tchen,

have you been able to verify whether behavior changed at all (per comment #18)? If you are still having this issue in uptodate 15.04, please

set log_level=1 in /etc/libvirt/libvirtd.log
stop libvirt-bin
rm /var/log/libvirt/libvirtd.log
run "sudo /usr/sbin/libvirtd 2>&1 | tee libvirt.debug" until it (I assume) breaks.
Attach libvirt.debug and any apparmor DENIED messages relating to libvirt from syslog.

Revision history for this message
David Peall (dkpeall) wrote :
Download full text (4.0 KiB)

I upgraded from 14.04 to 14.10 installed libvirt and got the same error:

From syslog I have pre aa-audit and then with aa-audit it seems to aa-audit clears the bug with audit removed it continues to work.

Post audit log:
Apr 14 20:19:50 dnshost11 kernel: [ 1284.666816] audit_printk_skb: 36 callbacks suppressed
Apr 14 20:19:50 dnshost11 kernel: [ 1284.666820] audit: type=1400 audit(1429035590.212:108): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=11745 comm="apparmor_parser"
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734099] audit: type=1400 audit(1429035595.284:109): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/etc/ld.so.cache" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734119] audit: type=1400 audit(1429035595.284:110): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/etc/ld.so.cache" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734156] audit: type=1400 audit(1429035595.284:111): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-lxc.so.0.1002.8" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734177] audit: type=1400 audit(1429035595.284:112): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-lxc.so.0.1002.8" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734241] audit: type=1400 audit(1429035595.284:113): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-qemu.so.0.1002.8" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734255] audit: type=1400 audit(1429035595.284:114): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/libvirt-qemu.so.0.1002.8" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734308] audit: type=1400 audit(1429035595.284:115): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-common.so.3.5.3" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734322] audit: type=1400 audit(1429035595.284:116): apparmor="AUDIT" operation="getattr" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-common.so.3.5.3" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:55 dnshost11 kernel: [ 1289.734380] audit: type=1400 audit(1429035595.284:117): apparmor="AUDIT" operation="open" profile="/usr/sbin/libvirtd" name="/usr/lib/x86_64-linux-gnu/libavahi-client.so.3.2.9" pid=11755 comm="libvirtd" requested_mask="r" fsuid=0 ouid=0
Apr 14 20:19:56 dnshost11 kernel: [ 1290.908063] Bridge firewalling registered
Apr 14 20:19:56 dnshost11 kernel: [ 1290.988004] ip_tables: (C) 2000-2006 Netfilter Core Team
Apr 14 20:19:56 dnshost11 kernel: [ 1291.129991] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Apr 14 20:19:56 dnshost11 kernel: [ 1291.233695] I...

Read more...

Changed in libvirt (Ubuntu):
status: Invalid → Confirmed
Revision history for this message
David Peall (dkpeall) wrote :

This is very reproducible
- install 14.04
- do-release-upgrade
- restart
- apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
- /etc/init.d/libvirt-bin start

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I cannot reproduce this following the above recipe.

Please add the information requested in comment #19,

set log_level=1 in /etc/libvirt/libvirtd.log
stop libvirt-bin
rm /var/log/libvirt/libvirtd.log
run "sudo /usr/sbin/libvirtd 2>&1 | tee libvirt.debug" until it (I assume) breaks.
Attach libvirt.debug and any apparmor DENIED messages relating to libvirt from syslog.

Changed in libvirt (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Lee Revell (rlrevell-k) wrote :

I can reproduce the bug. Attaching the requested information.

Revision history for this message
Lee Revell (rlrevell-k) wrote :
Revision history for this message
Lee Revell (rlrevell-k) wrote :
Revision history for this message
Lei Wang (raywang) wrote :

this is bug is reproducible.

Install 14.04
dist-upgrade
install libvirtd-bin

I also hit this bug..

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Quoting Ray Wang (<email address hidden>):
> this is bug is reproducible.
>
> Install 14.04
> dist-upgrade

to 14.10?

> install libvirtd-bin
>
> I also hit this bug..

I'm still unable to reproduce. To be sure, are you running upstart
and a mostly stock Ubuntu system?

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
Revision history for this message
mahmoh (mahmoh) wrote :

Hi Serge,

I hit this problem on a stock install of 14.04, only installed MAAS and libvirt-bin + dist-upgrade. When I ran your debug commands (@22) libvirt-bin failed to crash but when I start the process it still fails, here's the only log output I see below:

/var/log/libvirt/libvirtd.log:
Sep 16 11:14:55 maas kernel: [ 1310.741951] init: libvirt-bin main process (30028) terminated with status 6
Sep 16 11:14:55 maas kernel: [ 1310.741980] init: libvirt-bin main process ended, respawning
Sep 16 11:15:05 maas kernel: [ 1320.786451] init: libvirt-bin post-start process (30033) terminated with status 1

/var/log/upstart/libvirt-bin.log:
/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.
Giving up waiting for /var/run/libvirt/libvirt-sock.
libvirt-bin stop/post-start, (post-start) process 30244

One thing I did notice is that I had set /etc/default/libvirt-bin: 'libvirtd_opts="-d -l" ' without changing any /etc/libvirt/libvirtd.conf, and removing this option allows it to start again fine.

Recipe:
1) Trusty stock install (and install MAAS from stable PPA?)
2) apt-get install libvirt-bin
3) /etc/default/libvirt-bin: ' libvirtd_opts="-d -l" '
4) apt-get dist-upgrade
5) sudo service libvirt-bin restart

Linux maas 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ii libvirt-bin 1.2.2-0ubuntu13.1.14 amd64 programs for the libvirt library
ii upstart 1.12.1-0ubuntu4.2 amd64 event-based init daemon
ii apparmor 2.8.95~2430-0ubuntu5.3 amd64 User-space parser utility for AppArmor

Additional:

$ sudo /usr/sbin/libvirtd -l 2>&1 | tee libvirt.debug
2015-09-16 15:30:51.587+0000: 30946: info : libvirt version: 1.2.2
2015-09-16 15:30:51.587+0000: 30946: error : virNetTLSContextCheckCertFile:117 : Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory

$ dmesg | grep -i armo | grep libv
[ 835.996698] audit: type=1400 audit(1442416021.069:40): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/libvirt/virt-aa-helper" pid=30866 comm="apparmor_parser"
[ 836.134080] audit: type=1400 audit(1442416021.205:45): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/libvirtd" pid=30868 comm="apparmor_parser"

The problem may just be the /etc/libvirt/libvirtd.conf default settings and maybe poor messaging or user error? Hope this helps someone. Changing the conf file to this fixed my problem:

listen_tls = 0
listen_tcp = 1

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks @mahmoh,

that's interesting. Perhaps we should add a comment in the shipped /etc/default/libvirt-bin?

Why had you added the -l? Is there a published recipe you were following, and should that be updated?

Changed in libvirt (Ubuntu):
status: Expired → Confirmed
Revision history for this message
aahernan (aahernan) wrote :

Problemas con mi kvm

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@mahmoh - ping (question in comment #30).

Stefan Bader (smb)
Changed in libvirt (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
Revision history for this message
Trent Lloyd (lathiat) wrote :

I had this issue today after upgrading from trusty->wily (yes I know not technically supported). Notably I was running the lts-wily kernel on trusty, and I had a cached profile.

So I am wondering if this combination results in the cache needing regeneration but not being triggered for regeneration?

I fixed the issue with:
 apparmor_parser --purge-cache
 apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
 apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd
 systemctl restart libvirt-bin

Revision history for this message
Thomas B. Rücker (thomas-ruecker) wrote :

The bug is STILL present and breaks libvirt-bin upon upgrade from 14.04 to 16.04

Performing the steps from comment 34 worked around the breakage.
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465/comments/34

Changed in libvirt (Ubuntu):
status: Expired → Confirmed
tags: added: xenial
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thomas,

would you mind filing a new bug using apport? In particular I'd like to see any local changes to your /etc/libvirt/libvirtd.conf and /etc/default/libvirt-bin and see the upgrade log if possible. If you can reproduce at will in clean vms by taking particular steps, that would be great. But this may well be a new bug and so it would be good to keep the new information separate so we can better track down the cause.

Revision history for this message
Andrea Bernabei (faenil) wrote :

@Serge why a new bug?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Because this bug report has quite a bit of information and history has taught me that not keeping information from different reporters separate can greatly complicate matters.

Since I'm asking for apport-uploaded information, it would be best that it not be mixed with information from another system.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Also note that there are already several different directions into which this bug has been taken. A crucial question is whether step 3 in @mahmoh's recipe is really needed:

 3) /etc/default/libvirt-bin: ' libvirtd_opts="-d -l" '

If not then there are different bugs at work.

Revision history for this message
guessi (guessi) wrote :

also hit this issue with a clean setup, but not sure how to reproduce,
since I've setup multiple server with the same "script",
but only one server hit the problem, and problem still after applying the workaround mention in #34

here's how I setup the services,

1. apt-get install qemu-kvm
2. apt-get install libvirt-bin
3. boot up VMs, and make sure it is running ( virsh list --all )
4. reboot host
5. wait for server start-up
6. login, check service libvirt-bin running state => not running, and VMs not start, of course
7. try to apply the workaround mention in #34, trick of apparmor_parser -r / -R
8. reboot again
9. login, check service libvirt-bin running state => running (at the first time)
10. reboot again
11. login, check service libvirt-bin running state => not running (seems like workaround not always works?)

look into the `syslog`, each time libvirt-bin unsuccessful start-up at boot, it will have the following log in syslog,

==> Apr 16 21:02:09 host2 kernel: [ 313.059830] init: libvirt-bin post-start process (2430) terminated with status 1

and here's my system information,

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty

$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

hope these information could help,

reference:
- https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465/comments/34

Revision history for this message
guessi (guessi) wrote :

Hi,

I've opened another bug/issue report, including patch, for the issue of "libvirt-bin not start",

please see #1571209 for detail,
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209

it seems to the problem I've ran into, I'm wondering if it could fix your problem,
please give help to test it, thanks !!!

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@lathiat,

when I take a 14.04 system,

grep netlink /etc/apparmor.d/usr.sbin.libvirtd

returns nothing; then do-release-upgrade -d, agree to the reboot, and

grep netlink /etc/apparmor.d/usr.sbin.libvirtd

returns

  network netlink,

Is it possible that you did not reboot after the release upgrade?

Changed in libvirt (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for libvirt (Ubuntu) because there has been no activity for 60 days.]

Changed in libvirt (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.