VM permanently tries to read /dev/shm/lttng-ust-wait-5

Bug #1432644 reported by Franck
94
This bug affects 18 people
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Fix Released
High
Unassigned
libvirt (Ubuntu)
Fix Released
High
Unassigned

Bug Description

VM permanently tries to read /dev/shm/lttng-ust-wait-5

When used with enforced apparmor profile, this results in logs filled with denied messages.
As I understand it, liblttng is here to allow tracing / debugging in python (not sure). Maybe virt-manager shouldn't be shipped with debugging mode enabled by default, or maybe it is a perfectly legitimate feature and the apparmor profile should be updated.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: virt-manager 1:1.0.1-4ubuntu3
ProcVersionSignature: Ubuntu 3.19.0-9.9-generic 3.19.1
Uname: Linux 3.19.0-9-generic x86_64
ApportVersion: 2.16.2-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Mon Mar 16 14:25:39 2015
InstallationDate: Installed on 2014-12-13 (92 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Release amd64 (20141022.1)
PackageArchitecture: all
SourcePackage: virt-manager
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Franck (alci) wrote :
affects: virt-manager (Ubuntu) → libvirt (Ubuntu)
tags: added: apparmor
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu):
status: New → Confirmed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

We should not allow access to /dev/shm/lttng-ust-wait-5 to VMs unless libvirt exposes the files in the domain definition and virt-aa-helper can update the policy on a per VM basis. We could add a rule to the libvirt-qemu abstraction, but it would be too generic 'owner /dev/shm/lttng-ust-wait-* rw,' and therefore break guest isolation (though that is of course fine for users to manually add if they need this functionality and understand the compromise).

Revision history for this message
Franck (alci) wrote :

So maybe it should be explicitly denied to avoid the log flooding ? But the question remains: why should VMs access this file?

Changed in libvirt (Ubuntu):
importance: Undecided → Low
Revision history for this message
Chris J Arges (arges) wrote :

Increasing the priority of this since this blocks normal users of qemu-system-x86 from even creating VMs.

The following illustrates the dependencies:
qemu-system-x86 -> librados2 -> liblttng-ust0

So either we should remove the dependencies here such that LTTng isn't being installed by default, or amend the apparmor profile so that it can accommodate this.

Changed in libvirt (Ubuntu):
importance: Low → High
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Please remove the dependency on liblttng-ust0 since it breaks guest isolation.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Another option is making the failure to open /dev/shm/lttng-ust-wait-* non-fatal. This is what Ubuntu Touch does on app launch.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I should have been more clear with this comment: "Please remove the dependency on liblttng-ust0 since it breaks guest isolation."

Please remove the dependency on liblttng-ust0, make the failure to open /dev/shm/lttng-ust-wait-* non-fatal or put the lttng file in a guest-specific directory. Modifying the apparmor policy to allow /dev/shm/lttng-ust-wait-* is not an option because it breaks guest isolation.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.2.12-0ubuntu8

---------------
libvirt (1.2.12-0ubuntu8) vivid; urgency=medium

  * silence denial of attempted reads of lttng files (LP: #1432644)
 -- Serge Hallyn <email address hidden> Fri, 27 Mar 2015 21:36:27 -0500

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Harald Hetzner (haraldhetzner) wrote :

With libvirt 1.2.12-0ubuntu8 being installed, virsh still fails to start an LXC operating system container, resulting in the following dmesg:

[ 126.832553] audit: type=1400 audit(1427571328.167:150): apparmor="DENIED" operation="open" profile="/usr/lib/libvirt/virt-aa-helper" name="/dev/shm/lttng-ust-wait-5" pid=4285 comm="virt-aa-helper" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 126.832646] audit: type=1400 audit(1427571328.167:151): apparmor="DENIED" operation="open" profile="/usr/lib/libvirt/virt-aa-helper" name="/dev/shm/lttng-ust-wait-5" pid=4285 comm="virt-aa-helper" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 127.195264] audit: type=1400 audit(1427571328.527:152): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-9d578815-a1e9-4596-aef9-a70717574f0e" pid=4310 comm="apparmor_parser"

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@haraldhetzner

please show the xml for the contianer which is failing to start.

Revision history for this message
Harald Hetzner (haraldhetzner) wrote :

This is the XML (virsh -c lxc:/// dumpxml test):

<domain type='lxc'>
  <name>test</name>
  <uuid>9d578815-a1e9-4596-aef9-a70717574f0e</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/sbin/init</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/lxc/test/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='00:16:00:69:5b:88'/>
      <source bridge='virbr0'/>
      <guest dev='eth0' actual='vnet1'/>
    </interface>
    <console type='pty'>
      <target type='lxc' port='0'/>
    </console>
  </devices>
</domain>

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I looked into this with libvirt-lxc and can confirm that the domains to not start, but the apparmor denial is a red herring. Ie, if I add this to /etc/apparmor.d/abstractions/libvirt-lxc:
   /dev/shm/lttng-ust-wait-* rw,

and this to /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper:
  /dev/shm/lttng-ust-wait-* rw,

Then do:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

I can try to start a container with:
$ virsh -c lxc:// start o1
error: Failed to start domain o1
error: internal error: guest failed to start: Message did not receive a reply (timeout by message bus)

but there are no denials.

Serge, feel free to add an explicit deny in /etc/apparmor.d/abstractions/libvirt-* and an allow rule for /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper, but know that won't fix this bug.

Revision history for this message
Petter Adsen (ducasse) wrote :

I have applied the update on vivid, and creating a new VM still fails:

Unable to complete install: 'internal error: process exited while connecting to monitor: libust[443/443]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
libust[443/445]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:958)
libust[443/445]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:958)
2015-03-31T13:21:23.715646Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20-cloud.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied

In my log:

[14770.402367] audit: type=1400 audit(1427808083.050:26): apparmor="DENIED" operation="open" profile="/usr/lib/libvirt/virt-aa-helper" name="/dev/shm/lttng-ust-wait-5" pid=400 comm="virt-aa-helper" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[14770.402404] audit: type=1400 audit(1427808083.050:27): apparmor="DENIED" operation="open" profile="/usr/lib/libvirt/virt-aa-helper" name="/dev/shm/lttng-ust-wait-5" pid=400 comm="virt-aa-helper" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[14770.772184] audit: type=1400 audit(1427808083.422:28): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-5b2de2b1-0776-4d51-92cc-c4356daa075a" pid=441 comm="apparmor_parser"
[14770.796265] audit: type=1400 audit(1427808083.446:29): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="qemu_bridge_helper" pid=441 comm="apparmor_parser"
[14771.000518] audit: type=1400 audit(1427808083.650:30): apparmor="DENIED" operation="open" profile="libvirt-5b2de2b1-0776-4d51-92cc-c4356daa075a" name="/sys/devices/system/node/" pid=443 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
[14771.000541] audit: type=1400 audit(1427808083.650:31): apparmor="DENIED" operation="open" profile="libvirt-5b2de2b1-0776-4d51-92cc-c4356daa075a" name="/sys/devices/system/cpu/" pid=443 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0

Am I missing something?

Revision history for this message
Petter Adsen (ducasse) wrote :

I missed the last message from the log, sorry:

audit: type=1400 audit(1427810727.733:51): apparmor="DENIED" operation="mknod" profile="libvirt-5b2de2b1-0776-4d51-92cc-c4356daa075a" name="/var/lib/libvirt/qemu/channel/target/fedora20-cloud.org.qemu.guest_agent.0" pid=3147 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=119 ouid=119

Revision history for this message
James Page (james-page) wrote :

Discussed with Serge and we're going to disable the lttng support in ceph for vivid - needs to be a little less hard than it is right now.

raising a ceph task.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.93-0ubuntu6

---------------
ceph (0.93-0ubuntu6) vivid; urgency=medium

  * d/control,rules,*.symbols: Disable lttng support until we can make
    it play a bit nicer with libvirt and apparmor, drop associated
    symbols (LP: #1432644).
 -- James Page <email address hidden> Wed, 01 Apr 2015 10:37:03 +0100

Changed in ceph (Ubuntu):
status: New → Fix Released
Revision history for this message
Petter Adsen (ducasse) wrote :

I don't understand how this is going to help, as I don't have/use ceph. Anyway, the problem still exists. Is there a way to completely disable apparmor for libvirt temporarily, until a fix can be found, as I *really* need it to work? Of course I could revert to 14.10, but I am trying to help find bugs in 15.04, so that's really not an option.

Revision history for this message
Petter Adsen (ducasse) wrote :
Download full text (3.5 KiB)

Sorry for not being clearer, I got rid of liblttng-ust0, but it still fails:

Unable to complete install: 'internal error: process exited while connecting to monitor: 2015-04-02T11:15:40.061933Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
'
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1819, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3422, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2015-04-02T11:15:40.061933Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied

journalctl -f:
april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/node/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/cpu/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.093:50): apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/node/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.093:51): apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/cpu/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="mknod" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0" pid=23690 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=119 ouid=119
april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.137:52): apparmor="DENIED" operation="mknod" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0" pid=23690 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=119 ouid=119
april 02 13:17:54 monster systemd-machined[23691]: Machine qemu-fedora20-2 terminated.
april 02 13:17:54 monster libvirt...

Read more...

Revision history for this message
Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1432644] Re: VM permanently tries to read /dev/shm/lttng-ust-wait-5
Download full text (3.8 KiB)

Quoting Petter Adsen (<email address hidden>):
> Sorry for not being clearer, I got rid of liblttng-ust0, but it still
> fails:
>
> Unable to complete install: 'internal error: process exited while connecting to monitor: 2015-04-02T11:15:40.061933Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
> '
> Traceback (most recent call last):
> File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
> callback(asyncjob, *args, **kwargs)
> File "/usr/share/virt-manager/virtManager/create.py", line 1819, in do_install
> guest.start_install(meter=meter)
> File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
> noboot)
> File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
> dom = self.conn.createLinux(start_xml or final_xml, 0)
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3422, in createLinux
> if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
> libvirtError: internal error: process exited while connecting to monitor: 2015-04-02T11:15:40.061933Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
>
> journalctl -f:
> april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/node/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
> april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/cpu/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
> april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.093:50): apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/node/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
> april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.093:51): apparmor="DENIED" operation="open" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/sys/devices/system/cpu/" pid=23690 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=119 ouid=0
> april 02 13:17:54 monster audit[23690]: <audit-1400> apparmor="DENIED" operation="mknod" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0" pid=23690 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=119 ouid=119
> april 02 13:17:54 monster kernel: audit: type=1400 audit(1427973474.137:52): apparmor="DENIED" operation="mknod" profile="libvirt-3d049f0b-f90d-4ef6-af58-6a2e2b238adb" name="/var/lib/libvirt/qemu/channel/target/fedora20-2.org.qemu.guest_agent.0" pid=23690 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=119 ouid=119
> april 02 13:17:54 monster sy...

Read more...

Revision history for this message
Sage Weil (sage-newdream) wrote :

FWIW, we are disabling the lttng support in the final hammer release to avoid this issue (until we come up with a better solution).

Revision history for this message
George (lmihaiescu) wrote :
Download full text (3.7 KiB)

None of the solutions provided work for me.

root@compute1:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

root@compute1:~# dpkg -l | grep libvirt
ii libvirt-bin 1.2.12-0ubuntu13~cloud0 amd64 programs for the libvirt library
ii libvirt0 1.2.12-0ubuntu13~cloud0 amd64 library for interfacing with different virtualization systems
ii nova-compute-libvirt 1:2014.2.3-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.2.2-0ubuntu2 amd64 libvirt Python bindings

root@compute1:~# aa-complain /etc/apparmor.d/usr.sbin.libvirtd
Setting /etc/apparmor.d/usr.sbin.libvirtd to complain mode.

root@compute1:~# service libvirt-bin restart
libvirt-bin stop/waiting
libvirt-bin start/running, process 19311

root@compute1:~# tail /var/log/syslog
Jun 30 20:10:15 localhost kernel: [5209244.436201] type=1400 audit(1435709415.215:1986158): apparmor="DENIED" operation="open" profile="libvirt-4a554600-47e6-4654-87bf-66d8c68bfdcb" name="/run/shm/lttng-ust-wait-5" pid=6082 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:17 localhost kernel: [5209247.156976] type=1400 audit(1435709417.935:1986159): apparmor="DENIED" operation="open" profile="libvirt-9393961e-4385-46f4-bd81-a17368bedcac" name="/run/shm/lttng-ust-wait-5" pid=23521 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:20 localhost kernel: [5209249.440373] type=1400 audit(1435709420.215:1986160): apparmor="DENIED" operation="open" profile="libvirt-4a554600-47e6-4654-87bf-66d8c68bfdcb" name="/run/shm/lttng-ust-wait-5" pid=6082 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:20 localhost kernel: [5209250.008276] ip_set: protocol 6
Jun 30 20:10:22 localhost kernel: [5209252.161148] type=1400 audit(1435709422.935:1986161): apparmor="DENIED" operation="open" profile="libvirt-9393961e-4385-46f4-bd81-a17368bedcac" name="/run/shm/lttng-ust-wait-5" pid=23521 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:25 localhost kernel: [5209254.444532] type=1400 audit(1435709425.215:1986162): apparmor="DENIED" operation="open" profile="libvirt-4a554600-47e6-4654-87bf-66d8c68bfdcb" name="/run/shm/lttng-ust-wait-5" pid=6082 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:27 localhost kernel: [5209257.165333] type=1400 audit(1435709427.935:1986163): apparmor="DENIED" operation="open" profile="libvirt-9393961e-4385-46f4-bd81-a17368bedcac" name="/run/shm/lttng-ust-wait-5" pid=23521 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=0
Jun 30 20:10:30 localhost kernel: [5209259.448704] type=1400 audit(1435709430.215:1986164): apparmor="DENIED" operation="open" profile="libvirt-4a554600-47e6-4654-87bf-66d8c68bfdcb" name="/run/shm/lttng-ust-wait-5" pid=6082 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid...

Read more...

Revision history for this message
Seth Arnold (seth-arnold) wrote :

George, if you want to allow the lttng accesses, edit /etc/apparmor.d/libvirt/TEMPLATE and the other similar profiles in /etc/apparmor.d/libvirt/ and add:

  /run/shm/lttng-ust-wait-5 rw,

Then run apparmor_parser --replace $(ls -1 /etc/apparmor.d/libvirt/libvirt* | grep -v files)

This does allow for cross-domain contamination. If you want to deny these accesses instead you can prepend "deny" to that rule above; I don't know if libvirt handles that gracefully or not, but it would prevent cross-domain contamination.

Thanks

Revision history for this message
Alex (xpk) wrote :

@jdstrand

I'm running the latest ceph-docker/daemon image with the ceph package version 0.94.2-1trusty and see the same issue:

libust[5136/5462]: Error: Error opening shm /lttng-ust-wait-5-0 (in get_wait_shm() at lttng-ust-comm.c:886)
libust[5136/5461]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:886)

Is it possible to somehow get the fix that was made in 0.93-0ubuntu6 over here as well?

Thanks

Revision history for this message
Antonio Messina (arcimboldo) wrote :

I'm affected too by the "log flooding" issue, although I can start VMs.

[4220163.899438] type=1400 audit(1447230173.207:7763726): apparmor="DENIED" operation="open" profile="libvirt-a9c8bb9b-f2cc-4e4f-b7b1-c6152d8029c4" name="/run/shm/lttng-ust-wait-5" pid=69948 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=111 ouid=0

I'm running Ubuntu 14.04.3 with cloud and ceph repositories.

ceph 0.94.3-1trusty
libvirt0 1.2.12-0ubuntu14.1~cloud0

Revision history for this message
Mohammed Naser (mnaser) wrote :

It looks like this bug has regressed due to the path of lttng-ust-wait-5 path changing to the following:

/run/shm/lttng-ust-wait-5

Would someoone be kind enough to release a fix for this?

Revision history for this message
Matt Fischer (mfisch) wrote :

Issue still exists in 0.94.5-1trusty (hammer). Will this be ever fixed in hammer?

Changed in libvirt (Ubuntu):
status: Fix Released → Confirmed
Changed in ceph (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Matt Fischer (mfisch) wrote :

Reopening since this is not fixed in either ceph or libvirt to my knowledge. Package versions referenced here still have the issue.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

The bug was fixed in vivid (and later). Which libvirt version are you using?

It sounds like we need to SRU this to trusty.

Revision history for this message
Paz (pazt) wrote :

Hi All

We had the same issue, but in our end it was misconfiguration of the key.
Make sure that the key you see in trace log is indeed the key you have in /etc/ceph/ceph.client.cinder.keying, if not try to fix in /etc/libvirt/secrets/<your uuid>, and reload with virsh secret define.
If it configured OK, make sure that it also configured on ceph cluster (ceph auth list).

Revision history for this message
Matt Fischer (mfisch) wrote :

We're using libvirt 1.2.12-0ubuntu13~cloud0.

Patz: There is noting in syslog about a key. I don't believe this bug has anything to do with a key.

Revision history for this message
David Medberry (med) wrote :

Serge we (Matt Fischer and myself) are using UCA Kilo with Trusty.

Revision history for this message
David Medberry (med) wrote :

@serge this is a different path in Hammer that needs also to have a fix similar to 1432644 so ANOTHER lttng issue.

Revision history for this message
David Medberry (med) wrote :

And since that was self-referential, maybe we just need a brand new bug.

Revision history for this message
David Medberry (med) wrote :

So Mfischer said "reopening" I think the bug recurred in a slightly different way. So maybe we need a new bug.... either way we'd like the fix.

Revision history for this message
Matt Fischer (mfisch) wrote :

adding /run/shm/lttng-ust-wait-5 rw, to /etc/apparmor.d/abstractions/libvirt-qemu is the fix.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Note that adding that entry may allow virtual machines an unexpected and unwelcome amount of influence over the host system. If you just want the errors silenced, use 'deny /run/shm/lttng-ust-wait-5 rw,' instead. If you actually want lttng to function, then feel free to continue using the allow rule but be sure you know why you want it and what it allows guests to do.

Thanks

Revision history for this message
Ken Dreyer (Red Hat) (kdreyer-redhat) wrote :

FYI ceph v0.94.6 will load LTTng-UST only when specifically configured. This should avoid SELinux / AppArmor denials in most cases. See http://tracker.ceph.com/issues/13274

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Ken, that's great: denying lttng in the profile just to silence the logs is certainly unfortunate for the people who want to use lttng to measure and inspect their VMs as the reason why lttng doesn't work is impossible to discover.

Thanks

Revision history for this message
James Page (james-page) wrote :

Marking ceph task as invalid, packages in Ubuntu don't enable this feature unlike upstream.

Changed in ceph (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Serge Hallyn (serge-hallyn) wrote : test case

Wily is going to be EOL in less than two months. Could we please have a
minimal xml for reproduction, or even better verification of the package
in wily-proposed? I would simply recommend closing this bug unsolved,
except the fix is wrapped up with several others which were already verified.

Revision history for this message
Mathew Hodson (mhodson) wrote :

This was reopened because of a regression. That regression was handled in Bug #1529319, so I'm resetting this bug to its previous status.

Changed in ceph (Ubuntu):
importance: Undecided → High
status: Invalid → Fix Released
Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.