libvirt no longer probes chained backing stores

A temporary workaround in my case is to:
1/ add the denied file in the Template /etc/apparmor.d/libvirt/TEMPLATE :
"/U/libvirt-image/base/lenny_compressed.qcow2" r,
2/ Remove the generated file /etc/apparmor.d/libvirt/libvirt-c05e04f2-fe2e-b8de-c129-bd074c908476*
3/ Reload apparmor and start the VM

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal:

    apport-collect 656173

When reporting bugs in the future please use apport, using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Please could you also provide details of how you created the .qcow2 files and how you created you virtual machine to use these files. This will help us reproduce your issue more accurately.

I've been unable to re-produce this issue on either Lucid or Maverick although they do exhibit different behaviour.

test.qcow2 -> test_base.qcow2 -> base/lenny_vase.qcow2 (sym link to lenny.qcow2)

Lucid apparmor profile:

  "/var/log/libvirt/**/test.log" w,
  "/var/lib/libvirt/**/test.monitor" rw,
  "/var/run/libvirt/**/test.pid" rwk,
  "/home/jamespage/vms/test_base.qcow2" rw,
  "/home/jamespage/vms/base/lenny.qcow2" rw,
  "/home/jamespage/vms/test.qcow2" rw,
  "/home/jamespage/reference/isos/ubuntu-server/maverick-server-i386.iso" r,
  # don't audit writes to readonly files
  deny "/home/jamespage/reference/isos/ubuntu-server/maverick-server-i386.iso" w,

Maverick apparmor profile:

  "/var/log/libvirt/**/test.log" w,
  "/var/lib/libvirt/**/test.monitor" rw,
  "/var/run/libvirt/**/test.pid" rwk,
  "/home/jamespage/vms/test.qcow2" rw,
  "/dev/sr0" r,
  # don't audit writes to readonly files
  deny "/dev/sr0" w,

No apparmor messages in kern.log, and no impact on functionality.

OK; I've now managed to re-produce the issue; It appears that virt-aa-helper only parses backing_files one level; in this case the full chain is two levels/three files, so the base qcow2 image is not included in the apparmor profile:

  "/var/log/libvirt/**/test.log" w,
  "/var/lib/libvirt/**/test.monitor" rw,
  "/var/run/libvirt/**/test.pid" rwk,
  "/home/jamespage/vms/test.qcow2" rw,
  "/home/jamespage/vms/test_base.qcow2" r,
  # don't audit writes to readonly files
  deny "/home/jamespage/vms/test_base.qcow2" w,

I incidentally found a potential bug in virt-install; it does not appear to recognise .qcow2 files and generates an xml definition with the disk type as raw.

Enabling the 'allow_disk_format_probing = 1' in /etc/libvirt/qemu.conf and restarting libvirtd-bin re-instates the automated probing of backing_files in Maverick.

This behavior changed in libvirt 0.8.3 and the pending lucid-security libvirt update and is part of the fix for CVE-2010-2237, CVE-2010-2238 and CVE-2010-2239. From /etc/libvirt/qemu.conf:
# If allow_disk_format_probing is enabled, libvirt will probe disk
# images to attempt to identify their format, when not otherwise
# specified in the XML. This is disabled by default.
#
# WARNING: Enabling probing is a security hole in almost all
# deployments. It is strongly recommended that users update their
# guest XML <disk> elements to include <driver type='XXXX'/>
# elements instead of enabling this option.
# allow_disk_format_probing = 1

So people can either:
1. adjust /etc/apparmor.d/libvirt/libvirt-<uuid> to have the extra files
2. adjust /etc/libvirt/qemu.conf for the above

The former is preferred for security reasons, but has to be done for each virtual machine.

Suggested release note text:

Previous libvirt versions would probe a qemu disk to determine its format and did not require that the format be declared in the XML. This is considered a security problem in most deployments and newer versions of libvirt will default to the 'raw' format when the format is not specified in the XML. As a result, non-raw disks without unspecified disk format will no longer be available in
virtual machines.

The libvirt-migrate-qemu-disks tool is provided to aid in transitioning virtual machine definitions to the new required format. In essence, it will check all domains for affected virtual machines, probe the affected disks and update the domain definition accordingly. This command will be run automatically on upgrade. See 'man 1 libvirt-migrate-qemu-disks' for details.

Users who require the old behavior can adjust the 'allow_disk_format_probing' option in /etc/libvirt/qemu.conf.

The automated run of libvirt-migrate-qemu-disks is great if your virtual machines are permanently defined within libvirt. However, if you simply inject and run a VM as needed, with something like the following:

  virsh create vm.xml

Which is what I've been doing. The VM definitions will not hang around after the VM finishes executing. As such, this bug is very likely to silently hit you. I believe the upgrade needs a more prominent notice of this change.

Jamin, your vm.xml does not contain enough information-- you must specify the disk format when defining a new VM. Simply put, if you are using a qcow2 disk you must specify "<driver name='qemu' type='qcow2'/>" in the vm.xml you are using.

Updated suggested release note text (my last one had a typo and I've tried to address Jamin's concerns):

Previous libvirt versions would probe a qemu disk to determine its format and did not require that the format be declared in the XML. This is considered a security problem in most deployments and newer versions of libvirt will default to the 'raw' format when the format is not specified in the XML. As a result, non-raw disks without a specified disk format will no longer be available in
existing virtual machines.

The libvirt-migrate-qemu-disks tool is provided to aid in transitioning virtual machine definitions to the new required format. In essence, it will check all domains for affected virtual machines, probe the affected disks and update the domain definition accordingly. This command will be run automatically on upgrade. For new virtual machines using non-raw images, the disk format must be specified in the domain XML provided to libvirt, otherwise the disk will not be available to the virtual machine. See 'man 1 libvirt-migrate-qemu-disks' for details.

Users who require the old behavior can adjust the 'allow_disk_format_probing' option in /etc/libvirt/qemu.conf.

Jamie, I understand that the vm.xml does not contain enough information for the version of libvirt in maverick. I understand that with maverick's version one now has to supply a disk format. However, this was not the case with libvirt under lucid. Furthermore, not providing the disk format does not result in any meaningful error message that would guide the user to the actual cause of the problem. My point is that the very same vm.xml worked perfectly under the version of libvirt in lucid. Hence this is a regression, what did work no longer works.

Jamin, that is why there will be a release note. There isn't much else that can be done. Applications change with new versions and people need to adapt the applications that use the changed applications appropriately. The tools that most people use are either already adjusted or are going to be as I mentioned in the other bug. If someone is going to be creating machines with raw XML, he/she is going to have to create the correct XML.

The implication of your comments suggests that you believe the behavior should somehow change. The default (which is the same for upstream and also an upcoming lucid security update) should not change, but people who require the old behavior can adjust /etc/libvirt/qemu.conf accordingly (but why not just fix the script/template that creates the XML?). The changed behavior fixes an important security flaw in libvirt. Imagine a VM with 2 raw disks and a libvirt that probes the disk for its format. A privileged user *in the guest* could write to the beginning of the second disk the necessary (and widely known) qemu disk structure, declare some arbitrary file on the system as its backing store, then wait for a reboot. On reboot, the guest now can access that file because libvirt probes the 2nd disk, discovers it is a qcow2 with backingstore, and calls the AppArmor (or SELinux, or DAC, it doesn't matter) security driver to add it to its list of allowed files, and the attacker has access to that data (the AppArmor driver will protect against some parts of the host, such as /etc, but most of the system is available to an attacker still-- including other virtual machine's disks). Granted it is a one shot deal since the attacker won't be able to modify the disk structure once qemu recognizes it as a qcow2, but the impact is potentially very high if she gets it right.

Sure applications change with versions. However, they normally also provide useful error messages when something that was previously valid no longer is. This is *not* the case here. There is *no* error message.

I'm not saying that users shouldn't provide valid XML when defining a VM. However, I am saying that if the XML being provided was previously valid and no longer is, that some clear indication about the change in validity should be given. There is no indication being given by the application.

Yes, I believe the current behavior should change. No, I'm not disagreeing that this is a necessary security fix, nor am I saying that in the end this is somehow an undesired fix. What I am disputing is how the change is currently being handled. I do see that an effort has been made to migrate older configurations, and it does appear likely that this effort will catch the majority of cases. However, it won't catch all cases and the current handling leaves all remaining users wondering what the hell is going on with no guidance.

I still see two points of contention. The first, and largest point of contention, is that due to the behavior change, previously valid XML configurations will now fail to operate as expected and will do so without any indication of why they are failing to operate. A change such as this should generate a log entry or something that would at least guide a user to documentation about the change and possible resolution. Perhaps an entry could be kicked out to the /var/log/libvirt/qemu/ per machine log file. Something along the lines of:

WARNING: no drive image file format specified, defaulting to raw mode. Please specify drive image format. For more information please see: ....

Second, I don't believe a blurb in the release notes alone is sufficient for this change. Why? Because, as indicated in the first point, there is no output from the application to indicate that the expected behavior has changed. Additionally, you've indicated that the existing behavior in lucid will be changing. As I understand it, this will be a simple security update. A change in behavior like this is definitely not something one would expect from a security update, not without a rather "in your face" notice about it.

I can see already a Problem with the opennebula package.
It will no longer work, since it does not generate this information on the domain xmls - it would not have this information anyway - else if it would probe for the data type of the image, this - we are being told - being a security issue.
@Jamie: I think the OS *in the guest* should never ever have access to the file of the image!
It should only have access to the virtualized disks, not to the file itself, where the backingstore informations are.
If it is now so, then the issue is rooted in qemu/kvm and not in libvirt.

@nerling:

Whatever is creating the domain XML (ie with 'create' or 'define') needs to add the "<driver name='qemu' type='foo'/>" bit, after which it is stored in the XML. If OpenNebula creates virtual machines and adds them to libvirt, it must be adjusted for this. If OpenNebula is creating virtual machines, then it should know what type of disk format it is creating. If it always creates one format, then it is easy-- OpenNebula should just add "<driver name='qemu' type='opennubula's disk format'/>".

While I am the one commenting on these changes, I would like to reiterate that these are upstream changes and not Ubuntu specific. Yes, the behavior was changed in Lucid because it can be a significant security issue and Ubuntu users need to be protected. I suggest working with OpenNebula upstream to see how they are handling libvirt's new behavior, then patches can be applied to OpenNebula on Lucid (and possibly Maverick, if it is affected also) in an SRU (https://wiki.ubuntu.com/StableReleaseUpdates) or if the patch is simple enough it can be published in the security queue like was done for virtinst and vm-builder.