[Lucid] Error starting domain: could not remove profile

Dustin,
can you attach the dmesg from immediately after this happens

That last message was intended for Chad. Chad, can you attach your dmesg after seeing the failure?

Sure... attempts from two different VMs...

[ 1287.883490] type=1503 audit(1265396838.963:19): operation="profile_replace" pid=5756
[ 1287.896774] device vnet0 entered promiscuous mode
[ 1287.897141] br0: port 2(vnet0) entering forwarding state
[ 1287.930561] br0: port 2(vnet0) entering disabled state
[ 1287.960322] device vnet0 left promiscuous mode
[ 1287.960326] br0: port 2(vnet0) entering disabled state

[ 1331.035140] type=1503 audit(1265396882.114:20): operation="profile_replace" pid=5810
[ 1331.049876] device vnet0 entered promiscuous mode
[ 1331.050380] virbr0: topology change detected, propagating
[ 1331.050384] virbr0: port 1(vnet0) entering forwarding state
[ 1331.090549] virbr0: port 1(vnet0) entering disabled state
[ 1331.120416] device vnet0 left promiscuous mode
[ 1331.120421] virbr0: port 1(vnet0) entering disabled state

I haven't touched apparmor in any way.

Actually, this is not a profiling issue. The error from /var/log/libvirt/qemu/....log is:
libvir: Security Labeling error : error calling aa_change_profile()

I see that in my logs too. Let me know if there's anything else you need from me. Thanks for being so responsive.

I just encountered the same error. After reading the blog post at http://penguindroppings.wordpress.com/2009/11/03/apparmor-svirt-security-driver-for-libvirt/ I tried to set
security_driver = "none"
in /etc/libvirt/quemu.conf but the problem still occurs. Can I somehow help solving this problem?

As a workaround to this problem, disabling the apparmor profiles helped. I disabled all profiles, but it might be a good idea to only disable the profiles of libvirt. To disable all profiles you may use:
sudo aa-complain /etc/apparmor.d/*
More Information can be found at https://help.ubuntu.com/community/AppArmor

flolle, please do not disable all of the profiles. This is not needed and there are many profiles in place that are not causing the problem.

To workaround this, you can edit qemu.conf as suggested (and restart libvirtd), or simply temporarily disable the apparmor profile until a reboot with:
$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd

This bug was fixed in the package libvirt - 0.7.5-5ubuntu6

---------------
libvirt (0.7.5-5ubuntu6) lucid; urgency=low

  * debian/patches/9013-apparmor-dont-clear-caps.patch: Don't clear
    capabilities when calling virt-aa-helper. When built with libcap-ng,
    clearing caps makes virt-aa-helper lose MAC_ADMIN, which is (obviously)
    needed by apparmor_parser. This restores libcap-ng behavior to what it was
    when not built with libcap-ng. (LP: #517714)
 -- Jamie Strandboge <email address hidden> Fri, 05 Feb 2010 16:48:42 -0600

This bug still exists on karmic

Just to chime in: I had this bug this morning (Apr 15) after I ran upgraded my Lucid:

"Error starting domain: could not remove profile for 'libvirt-55359786-ed36-5577-0d71-f252432b9388'"

raceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 300, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: could not remove profile for 'libvirt-55359786-ed36-5577-0d71-f252432b9388'

virt-manager:
  Installed: 0.8.2-2ubuntu8
  Candidate: 0.8.2-2ubuntu8
  Version table:
 *** 0.8.2-2ubuntu8 0
        500 http://gb.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

LSB Version: core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:cxx-3.0-amd64:cxx-3.0-noarch:cxx-3.1-amd64:cxx-3.1-noarch:cxx-3.2-amd64:cxx-3.2-noarch:cxx-4.0-amd64:cxx-4.0-noarch:desktop-3.1-amd64:desktop-3.1-noarch:desktop-3.2-amd64:desktop-3.2-noarch:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.0-amd64:graphics-3.0-noarch:graphics-3.1-amd64:graphics-3.1-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-3.2-amd64:printing-3.2-noarch:printing-4.0-amd64:printing-4.0-noarch:qt4-3.1-amd64:qt4-3.1-noarch

I also encountered this issue just now. It is caused by an update of the apparmor profile:

--- /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper 2010-04-14 14:19:00.000000000 +0200
+++ /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper.dpkg-dist 2010-04-15 08:17:39.000000000 +0200
@@ -3,6 +3,7 @@

 /usr/lib/libvirt/virt-aa-helper {
   #include <abstractions/base>
+ #include <abstractions/user-tmp>

   # needed for searching directories
   capability dac_override,
@@ -14,9 +15,30 @@
   deny @{PROC}/[0-9]*/mounts r,
   @{PROC}/filesystems r,

+ # for hostdev
+ /sys/devices/ r,
+ /sys/devices/** r,
+
   /usr/lib/libvirt/virt-aa-helper mr,
   /sbin/apparmor_parser Ux,

   /etc/apparmor.d/libvirt/* r,
   /etc/apparmor.d/libvirt/libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]* rw,
+
+ # For backingstore, virt-aa-helper needs to peek inside the disk image, so
+ # allow access to non-hidden files in @{HOME} as well as storage pools, and
+ # removable media and filesystems. A virt-aa-helper failure when checking a
+ # disk for backinsgstore is non-fatal (but obviously the backingstore won't
+ # be added).
+ audit deny @{HOME}/.* mrwkl,
+ audit deny @{HOME}/.*/ rw,
+ audit deny @{HOME}/.*/** mrwkl,
+ audit deny @{HOME}/bin/ rw,
+ audit deny @{HOME}/bin/** mrwkl,
+ @{HOME}/ r,
+ @{HOME}/** r,
+ /var/lib/libvirt/images/ r,
+ /var/lib/libvirt/images/** r,
+ /{media,mnt,opt,srv}/** r,
+ deny /dev/** mrwkl,
 }

I reverted to the old one and virt-manager was able to start virtual machines again.

Since Ubuntu counts "me too!!" as people affected: ME TOO!!

Nothing more to add.

Can people post the output of of 'dmesg' after they see this error. Also, can you give the XML for the machine that is failing to start (virsh dumpxml <machine name>')?

dmesg output:

[ 160.754633] device vnet0 entered promiscuous mode
[ 160.755215] br0: port 2(vnet0) entering learning state
[ 160.756086] device vnet1 entered promiscuous mode
[ 160.756580] br1: port 2(vnet1) entering learning state
[ 160.824741] br0: port 2(vnet0) entering disabled state
[ 160.861780] device vnet0 left promiscuous mode
[ 160.861783] br0: port 2(vnet0) entering disabled state
[ 160.932202] br1: port 2(vnet1) entering disabled state
[ 160.971772] device vnet1 left promiscuous mode
[ 160.971775] br1: port 2(vnet1) entering disabled state

libvirt-bin:
  Installed: 0.7.5-5ubuntu23
  Candidate: 0.7.5-5ubuntu23
  Version table:
 *** 0.7.5-5ubuntu23 0
        500 http://de.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

XML file for machine (from OpenNebula, but that does not matter here since I start the machine locally by hand):

<domain type='kvm'>
 <name>one-103</name>
 <memory>3145728</memory>
 <os>
  <type>hvm</type>
  <boot dev='hd'/>
 </os>
 <devices>
  <emulator>/usr/bin/kvm</emulator>
  <disk type='file' device='disk'>
   <source file='/var/lib/one//103/images/disk.0'/>
   <target dev='hda'/>
  </disk>
  <interface type='bridge'>
   <source bridge='br0'/>
   <mac address='52:54:00:10:b4:e9'/>
  </interface>
  <interface type='bridge'>
   <source bridge='br1'/>
   <mac address='52:54:00:10:b4:ea'/>
  </interface>
  <graphics type='vnc' listen='127.0.0.1' port='-1'/>
 </devices>
 <features>
  <acpi/>
 </features>
</domain>

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 300, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: could not remove profile for 'libvirt-f56646fb-00e2-d2db-b1c1-fb78c49d66ca'

dmesg:
[ 378.512081] device vnet0 entered promiscuous mode
[ 378.514705] br0: port 2(vnet0) entering forwarding state
[ 378.531817] br0: port 2(vnet0) entering disabled state
[ 378.560888] device vnet0 left promiscuous mode
[ 378.560892] br0: port 2(vnet0) entering disabled state

/var/log/libvirt/qemu:
libvir: Security Labeling error : error calling aa_change_profile()

Reverting /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper as described above worked for me.

dmesg output:
[ 1341.500672] type=1503 audit(1271335124.806:17): operation="open" pid=15949 parent=1779 profile="/usr/lib/libvirt/virt-aa-helper" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/data/sas/vmimages/zimbra.img"
[ 1341.500853] type=1503 audit(1271335124.806:18): operation="open" pid=15949 parent=1779 profile="/usr/lib/libvirt/virt-aa-helper" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/data/sas/isos/jeos-8.04.3-jeos-i386.iso"
[ 1341.509836] device vnet0 entered promiscuous mode
[ 1341.512314] br1: port 2(vnet0) entering learning state
[ 1341.640057] br1: port 2(vnet0) entering disabled state
[ 1341.718121] device vnet0 left promiscuous mode
[ 1341.718125] br1: port 2(vnet0) entering disabled state

no libvirt xml because the error happens during "virt-install"

Can people seeing this issue comment out the following line in /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper:
  #deny /dev/** mrwkl,

Then perform:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

Please report back if the machine starts (don't worry about denied messages at this point if the machine starts).

That worked for me, Jamie! :)

-RR

Yes, I can confirm this once I unload all my VM profiles. I had a hard time confirming because they were already loaded and didn't unoload on VM shutdown/destroy. The 'deny /dev/** mrwkl,' is much too zealous.

The attached profile for virt-aa-helper should fix the issue. Please report back if you have any problems with it.

I've the same problem since the latest updates to the kernel (2.6.32-20-server -> 2.6.32-21-server), some updates to apparmor/profiles and libvirt.

Since some days I've 100% cpu load with 10.04 guest and host (4 cpus, lvm2, virtio net and hd). So I can't use any guest.

Martin, the 100% CPU load is a different issue (see bug #549428).

The fixed "usr.lib.libvirt.virt-aa-helper" from Jamie and a "apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper" helps and the guest is starting again.

The newest kernel, apparmor and libs fixed my problem with 100% load and very slow guests.

This bug was fixed in the package libvirt - 0.7.5-5ubuntu24

---------------
libvirt (0.7.5-5ubuntu24) lucid; urgency=low

  * debian/apparmor/usr.lib.libvirt.virt-aa-helper: eek, the /dev change from
    the last upload was a wee bit too aggressive. Revert that and allow access
    to .img, .qcow{,2}, and .vmdk (file extensions that actually support
    backingstore) and .[iI][sS][oO] since it is so common (LP: #517714)
 -- Jamie Strandboge <email address hidden> Thu, 15 Apr 2010 08:52:27 -0500

I don't suppose there is any explanation why I just upgraded to 10.04 and this popped back up ehh?
Error reported in window:

 "Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 300, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: monitor socket did not show up.: Connection refused"

Attempted: "sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd" to no avail.
Nothing generated in dmesg output during error.

commented out the following line and fixed my issues.

audit deny /etc/apparmor.d/libvirt/** wxl,

I am also seeing the same connection refused error as skogs. Should we open a separate ticket about it?

AppArmor should not be not causing this connection refused message because skogs said there is nothing in the dmesg indicating a denial. Indeed, skogs removal of 'audit deny /etc/apparmor.d/libvirt/** wxl,' should not have done anything because the libvirtd profile disallows this access regardless (and that rule just makes sure it is logged. What probably happened is the libvirtd restart or system reboot 'cleaned up' libvirt so it would function correctly again.

Please file a different bug-- this one is fixed.

I've been should on before, but I did remove my comment and everything still works even after a system reboot. I withdraw my complaint. Shame on me. Of course now I have no idea what originally caused it to not work, but I've been trusting in FM for a long time.

Hey guys, I know this has been closed a while, but was still getting the error when trying to start a VM in Lucid RC:
libvirtError: monitor socket did not show up.: Connection refused

When I updated my file and rebooted I'm now getting:
Unable to open connection to hypervisor URI 'qemu:///system':
unable to connect to '/var/run/libvirt/libvirt-sock': No such file or directory
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 896, in _try_open
    None], flags)
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 111, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: unable to connect to '/var/run/libvirt/libvirt-sock': No such file or directory

I uninstalled libvirt0 (and everything that depended on it) and then re-installed ubuntu-virt-server and ubuntu-virt-mgmt and now I'm back to the first error (Connection refused).

I'm using a fakeraid array with /dev/mapper. Could this be causing this?

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 588, in run_domain
    vm.startup()
  File "/usr/share/virt-manager/virtManager/domain.py", line 150, in startup
    self._backend.create()
  File "/usr/lib/python2.6/dist-packages/libvirt.py", line 300, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: monitor socket did not show up.: Connection refused

This bug is fixed in Lucid. People who are still having problems should file a new bug with 'ubuntu-bug libvirt'.

Hey all,

I had this exact same error pop up for me recently, but the fix was completely different.

Upon further investigation of /var/log/libvirt/qemu/<Windows XP image name>, I saw that the KVM was attempting to mount a physical CDrom when there was no CDROM in the CD tray. Because of that, it was giving me the same error as listed in the original error. However, I was not having any problems getting other KVMs working properly.

The fix, for this problem was to open the KVM's details (I am using virtual machine manager in 10.04), and disconnect the CDROM device.

I hope this helps other people.