Apparmor denies qemu access to a number of important directories.

We should not allow access to /tmp and /var/tmp as that breaks application isolation. As for /var/lib/charm/ceph/ceph.conf, this sounds like something virt-aa-helper should be adding. Can you attach the domain xml for the affected VM?

I just realized there's another bug opened that's similar to this
bug 1365261.

However, the resolution for the /tmp/ errors and the resolution for /var/lib/charm/ceph errors may be different, so I'll leave this one open.

Is there any way to get rid of the messages? Perhaps we should patch qemu to not attempt to access /tmp or /var/tmp?

Basically these messages are now filling up logs on openstack compute nodes.

xml for the created vm.

This applies to all vm's created via openstack through libvirt.

qemu doesn't normally need /tmp and /var/tmp. Something is making it use it (ie, VMs launched under local libvirt (ie, not OpenStack) don't have this problem). One could add an explicit deny rule to /etc/apparmor.d/abstractions/libvirt-qemu to deny /tmp and /var/tmp, but I think it would be better to understand the problem (and that might break testing environment that legitimately put the disk in /tmp).

The attached xml isn't what I was looking for. When an affected VM is running, can you do:
$ virsh dumpxml <domain>

where '<domain>' can be found from 'virsh list'.

Somehow the earlier xml upload was corrupted trying again.

Launchpad fail... One more time.

Launchpad fail... One more time.

This is in the domain xml:
      <auth username='nova-compute'>
        <secret type='ceph' uuid='514c9fca-8cbe-11e2-9c52-3bc8c7819472'/>
      </auth>

there is nothing else in it regarding ceph. I looked at https://libvirt.org/storage.html and don't see where you would tell qemu to look at /var/lib/charm/ceph/ceph.conf. The way forward:
1. adjust the charm to not have qemu need to access ceph.conf
2. add the following to /etc/apparmor.d/abstractions/libvirt-qemu (assuming there is nothing sensitive in there):
  /var/lib/charm/ceph/ceph.conf r,
3. if /var/lib/charm/ceph/ceph.conf is set via some libvirt directive, adjust virt-aa-helper and/or libvirt to add this setting to the VM-specific .files file in /etc/apparmor.d/libvirt

I'm not well-versed with ceph and how OpenStack is using it to make a recommendation on which is best (but am happy to discuss the correct path forward if someone can discuss how /var/lib/charm/ceph/ceph.conf is being set).

Dave and I spoke on irc. Here is a summary:
 * qemu uses librbd which under the hood looks at ceph.conf, etc
 * /etc/ceph/ceph.conf -> /etc/alternatives/ceph.conf -> /var/lib/charm/ceph/ceph.conf
 * we already allow access to /etc/ceph/ceph.conf in /etc/apparmor.d/abstraction/libvirt-qemu
 * /var/lib/charm/ceph/ceph.conf does not contain anything particularly sensitive
 * /var/lib/charm/ceph/ceph.conf does tell qemu (via librbd) where to find the keyring and other options

Therefore, adding this to /etc/apparmor.d/abstraction/libvirt-qemu is ok:
  /var/lib/charm/ceph/ceph.conf r,

but note that /var/lib/charm/ceph/ceph.conf seems rather charm-specific. If we add more of these, we'll want to consider a way to clean this up (eg, via a glob).

If qemu then goes to look at the keyring files, then libvirt/aa-virt-helper needs to be updated to add the VM-specific keyring files to the VM-specific .files file in /etc/apparmor.d/libvirt such that only VMs specifiying ceph have access to the keyring files, and the VMs should be limited to accessing only the keyring file it should have access to. Based on Dave's feedback, the VM doesn't seem to need access to the keyring files and the readonly access to /var/lib/charm/ceph/ceph.conf is sufficient.

As for the /tmp and /var/tmp denials, Dave mentioned that adding the following rules silenced the denials:
  /tmp/ r,
  /var/tmp/ r,

I'm not a fan of those rules in general, because it gives the VMs read access to the directory and they can see what is in there. However I also don't want to break existing setups by adding an explicit deny rule that would block all access to /tmp and /var/tmp if the user updated policy for that or is putting disks in /tmp for testing environments.

As such I suggest the following:
1. for stable releases, add the following to /etc/apparmor.d/abstractions/libvirt-qemu:
  /tmp/ r,
  /var/tmp/ r,
2. for vivid, add the following to /etc/apparmor.d/abstractions/libvirt-qemu:
  deny /tmp/{,**} r,
  deny /var/tmp/{,**} r,

'1' is suitable for SRU since it only allows access where it wasn't allowed before. If we get bug reports in 15.04+ for '2', the proper solution is to have libvirt setup a vm-specific tmp dir, and have aa-virt-helper add this directory to the .files file for that VM.

We should implement (2) right after the holiday break (in case there are actually breakages).

This bug was fixed in the package libvirt - 1.2.8-0ubuntu19

---------------
libvirt (1.2.8-0ubuntu19) vivid; urgency=medium

  * apparmor libvirt-qemu template: allow reading charm-specific ceph config
    and silence denials for /tmp/**. (LP: #1403648)
 -- Serge Hallyn <email address hidden> Tue, 06 Jan 2015 10:27:33 -0600

@Dave,

could you please update the description with a concise test case for the SRU process?

Hello Dave, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Dave, or anyone else affected,

Accepted libvirt into utopic-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/libvirt/1.2.8-0ubuntu11.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package libvirt - 1.2.2-0ubuntu13.1.9

---------------
libvirt (1.2.2-0ubuntu13.1.9) trusty-proposed; urgency=medium

  * apparmor libvirt-qemu template: allow reading charm-specific ceph config
    and allow reading under /tmp and /var/tmp (for SRU only) (LP: #1403648)
  * numa-cgroups-fix-cpuset-mems-init.patch - cherrypicked, refreshed patch
    (by Richard Laager) to fix failure to start on numa node 1 (LP: #1404388)
  * libvirt-qemu: add r to sgabios.bin (LP: #1393548)
 -- Serge Hallyn <email address hidden> Tue, 06 Jan 2015 10:39:15 -0600

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Ideally, this would've been:

/var/lib/charm/*/ceph.conf r,

service name can be different. For example:

[534614.823259] type=1400 audit(1422553522.478:143): apparmor="DENIED" operation="open" profile="libvirt-a7605ca2-0b10-4afa-b25e-3b6a83ae5552" name="/var/lib/charm/az1-compute/ceph.conf" pid=43088 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=110 ouid=0

This bug was fixed in the package libvirt - 1.2.8-0ubuntu21

---------------
libvirt (1.2.8-0ubuntu21) vivid; urgency=medium

  * d/apparmor/libvirt-qemu: Update the ceph.conf allow rule (LP: #1403648)
 -- Serge Hallyn <email address hidden> Fri, 30 Jan 2015 10:02:20 +0100