AppArmor

BUG: unable to handle kernel NULL pointer dereference (aa_label_merge)

Bug #1448912 reported by Roger Lehmann on 2015-04-27

This bug affects 6 people

	Status	Importance	Assigned to
AppArmor	Confirmed	Medium	John Johansen
linux (Ubuntu)	Fix Released	Undecided	John Johansen
Trusty	Confirmed	Undecided	John Johansen
Vivid	Won't Fix	Undecided	Unassigned
Wily	Won't Fix	Undecided	John Johansen

Bug Description

I had a kernel problem with one of my high availability VMs. This was not the first occurence of the problem, it seems to appear every now and then.
The VM froze because of the Kernel bug appended as a text file. I had to hard reset it.
I am running Ubuntu 14.04.2 Kernel 3.16.0-34-generic with libapache2-mod-apparmor 2.8.95~2430-0ubuntu5.1 and apache 2.4.7-1ubuntu4.4 (mpm_prefork).
Unfortunately I am unable to reproduce the bug at will. This is a serious problem for high availability VMs.
Looking at the trace, I assume this is a bug in AppArmor.
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 15 14:13 seq
crw-rw---- 1 root audio 116, 33 May 15 14:13 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=UUID=69e1aff7-c3d1-4bf4-8685-bd3fc2942f60
InstallationDate: Installed on 2015-03-25 (61 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
IwConfig:
eth0 no wireless extensions.

lo no wireless extensions.
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
LANGUAGE=en_US:en
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-37-generic root=UUID=fea3075f-a9a3-414b-992a-b812d86cd04e ro
ProcVersionSignature: Ubuntu 3.16.0-37.51~14.04.1-generic 3.16.7-ckt9
RelatedPackageVersions:
linux-restricted-modules-3.16.0-37-generic N/A
linux-backports-modules-3.16.0-37-generic N/A
linux-firmware 1.127.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.16.0-37-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

WifiSyslog:

_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-2.2
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.2:cvnQEMU:ct1:cvrpc-i440fx-2.2:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.2
dmi.sys.vendor: QEMU

See original description

Tags:

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-04-27:

Trace dump of the kernel NULL pointer derefecene bug Edit (77.9 KiB, text/plain)

Tyler Hicks (tyhicks) on 2015-05-22

Changed in apparmor:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

Kees Cook (kees) wrote on 2015-05-22:

warn and oops Edit (7.1 KiB, text/plain)

I saw the same thing this morning.

Ubuntu 14.04.2 LTS
Kernel linux-image-3.16.0-30-generic
libapache2-mod-apparmor 2.8.95~2430-0ubuntu5.1
apache2-mpm-prefork 2.4.7-1ubuntu4.4

Attached is first the warning (like in bug 1447530), and then the Oops, 9 seconds later.

Kees Cook (kees) on 2015-05-22

summary:

- BUG: unable to handle kernel NULL pointer dereference
+ BUG: unable to handle kernel NULL pointer dereference (aa_label_merge)

Revision history for this message

Brad Figg (brad-figg) wrote on 2015-05-22: Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1448912

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete
tags:	added: utopic

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: BootDmesg.txt

BootDmesg.txt Edit (35.1 KiB, text/plain)

apport information

tags:	added: apport-collected trusty
description:	updated

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: CurrentDmesg.txt

CurrentDmesg.txt Edit (3.0 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: Lspci.txt

Lspci.txt Edit (4.6 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: ProcCpuinfo.txt

ProcCpuinfo.txt Edit (1.2 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: ProcInterrupts.txt

ProcInterrupts.txt Edit (1.4 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: ProcModules.txt

ProcModules.txt Edit (1.0 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: UdevDb.txt

#10

UdevDb.txt Edit (67.7 KiB, text/plain)

apport information

Revision history for this message

Roger Lehmann (roger-lehmann-u) wrote on 2015-05-26: UdevLog.txt

#11

UdevLog.txt Edit (161.0 KiB, text/plain)

apport information

Changed in linux (Ubuntu):
status:	Incomplete → Confirmed

Revision history for this message

Kees Cook (kees) wrote on 2015-07-15:

#12

one.txt Edit (3.6 KiB, text/plain)

Three more crashes today, one after the other (the trigger must be some kind of Apache access pattern, still trying to figure that out.)

Revision history for this message

Kees Cook (kees) wrote on 2015-07-15:

#13

two.txt Edit (3.9 KiB, text/plain)

Revision history for this message

Kees Cook (kees) wrote on 2015-07-15:

#14

three.txt Edit (3.5 KiB, text/plain)

Tim Gardner (timg-tpi) on 2015-07-15

Changed in linux (Ubuntu):
assignee:	nobody → John Johansen (jjohansen)

Revision history for this message

Kees Cook (kees) wrote on 2015-07-17:

#15

debugging-oops.txt Edit (3.4 KiB, text/plain)

Here is the Oops from a jj-special kernel. similar, but different.

Revision history for this message

John Johansen (jjohansen) wrote on 2015-07-18:

#16

Yep, thanks. I think I've got it now. I am working on a fix

Revision history for this message

Aidan Hobson Sayers (aidanhs) wrote on 2015-07-22:

#17

syslog (1) Edit (10.3 KiB, text/plain)

Not sure if this is the same issue or not, my null pointer is *not* in aa_label_merge - "BUG: unable to handle kernel NULL pointer dereference (aa_file_perm)".

It does have some significant similarities in the trace though. I've had this on 14.04 kernel 3.16 and 15.04 kernel 3.19.

Revision history for this message

Aidan Hobson Sayers (aidanhs) wrote on 2015-07-22:

#18

I should note that the above is using AUFS (via docker) - https://github.com/docker/docker/issues/14816

Revision history for this message

Kees Cook (kees) wrote on 2015-08-01:

#19

The debug kernels reduced the frequency of the Oopsing, but this has made AppArmor unusable with Apache for me.

Changed in linux (Ubuntu Trusty):
status:	New → Confirmed
Changed in linux (Ubuntu Vivid):
status:	New → Confirmed

Revision history for this message

John Johansen (jjohansen) wrote on 2015-11-13:

#20

there are new test images in
http://people.canonical.com/~jj/lp1448912/

there has been some testing of these and so far they appear to fix the issue

Revision history for this message

Kees Cook (kees) wrote on 2015-11-17:

#21

It's been 3 days running the test kernel and I've seen no problems. Very encouraging!

Revision history for this message

Kees Cook (kees) wrote on 2015-12-05:

#22

Still no issues for me. Yay! :)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-03-02:

#23

Download full text (7.6 KiB)

This bug was fixed in the package linux - 4.4.0-9.24

---------------
linux (4.4.0-9.24) xenial; urgency=low

[ Tim Gardner ]

* Release Tracking Bug
- LP: #1551319

* AppArmor logs denial for when the device path is ENOENT (LP: #1482943)
- SAUCE: apparmor: fix log of apparmor audit message when kern_path() fails

  * BUG: unable to handle kernel NULL pointer dereference (aa_label_merge) (LP:
    #1448912)
    - SAUCE: apparmor: Fix: insert race between label_update and label_merge
    - SAUCE: apparmor: Fix: ensure aa_get_newest will trip debugging if the
      replacedby is not setup
    - SAUCE: apparmor: Fix: label merge handling of marking unconfined and stale
    - SAUCE: apparmor: Fix: refcount race between locating in labelset and get
    - SAUCE: apparmor: Fix: ensure new labels resulting from merge have a
      replacedby
    - SAUCE: apparmor: Fix: label_vec_merge insertion
    - SAUCE: apparmor: Fix: deadlock in aa_put_label() call chain
    - SAUCE: apparmor: Fix: add required locking of __aa_update_replacedby on
      merge path
    - SAUCE: apparmor: Fix: convert replacedby update to be protected by the
      labelset lock
    - SAUCE: apparmor: Fix: update replacedby allocation to take a gfp parameter

  * apparmor kernel BUG kills firefox (LP: #1430546)
    - SAUCE: apparmor: Disallow update of cred when then subjective != the
      objective cred
    - SAUCE: apparmor: rework retrieval of the current label in the profile update
      case

* sleep from invalid context in aa_move_mount (LP: #1539349)
- SAUCE: apparmor: fix sleep from invalid context

* s390x: correct restore of high gprs on signal return (LP: #1550468)
- s390/compat: correct restore of high gprs on signal return

* missing SMAP support (LP: #1550517)
- x86/entry/compat: Add missing CLAC to entry_INT80_32

  * Floating-point exception handler receives empty Data-Exception Code in
    Floating Point Control register (LP: #1548414)
    - s390/fpu: signals vs. floating point control register

* kvm fails to boot GNU Hurd kernels with 4.4 Xenial kernel (LP: #1550596)
- KVM: x86: fix conversion of addresses to linear in 32-bit protected mode

  * Surelock GA2 SP1: capiredp01: cxl_init_adapter fails for CAPI devices
    0000:01:00.0 and 0005:01:00.0 after upgrading to 840.10 Platform firmware
    build fips840/b1208b_1604.840 (LP: #1532914)
    - cxl: Fix PSL timebase synchronization detection

* [Feature]EDAC support for Knights Landing (LP: #1519631)
- EDAC, sb_edac: Set fixed DIMM width on Xeon Knights Landing

  * Various failures of kernel_security suite on Xenial kernel on s390x arch
    (LP: #1531327)
    - [config] s390x -- CONFIG_DEFAULT_MMAP_MIN_ADDR=65536

* Unable to install VirtualBox Guest Service in 15.04 (LP: #1434579)
- [Config] Provides: virtualbox-guest-modules when appropriate

  * linux is missing provides for virtualbox-guest-modules [i386 amd64 x32] (LP:
    #1507588)
    - [Config] Provides: virtualbox-guest-modules when appropriate

  * Backport more recent driver for SKL, KBL and BXT graphics (LP: #1540390)
    - SAUCE: i915_bpo: Provide a backport driver for SKL, KBL & BXT graphics
    - SA...

This bug was fixed in the package linux - 4.4.0-9.24

---------------
linux (4.4.0-9.24) xenial; urgency=low

[ Tim Gardner ]

* Release Tracking Bug
    - LP: #1551319

* AppArmor logs denial for when the device path is ENOENT (LP: #1482943)
    - SAUCE: apparmor: fix log of apparmor audit message when kern_path() fails

* sleep from invalid context in aa_move_mount (LP: #1539349)
    - SAUCE: apparmor: fix sleep from invalid context

* s390x: correct restore of high gprs on signal return (LP: #1550468)
    - s390/compat: correct restore of high gprs on signal return

* missing SMAP support (LP: #1550517)
    - x86/entry/compat: Add missing CLAC to entry_INT80_32

* Floating-point exception handler receives empty Data-Exception Code in
    Floating Point Control register (LP: #1548414)
    - s390/fpu: signals vs. floating point control register

* kvm fails to boot GNU Hurd kernels with 4.4 Xenial kernel (LP: #1550596)
    - KVM: x86: fix conversion of addresses to linear in 32-bit protected mode

* [Feature]EDAC support for Knights Landing (LP: #1519631)
    - EDAC, sb_edac: Set fixed DIMM width on Xeon Knights Landing

* Various failures of kernel_security suite on Xenial kernel on s390x arch
    (LP: #1531327)
    - [config] s390x -- CONFIG_DEFAULT_MMAP_MIN_ADDR=65536

* Unable to install VirtualBox Guest Service in 15.04 (LP: #1434579)
    - [Config] Provides: virtualbox-guest-modules when appropriate

* linux is missing provides for virtualbox-guest-modules [i386 amd64 x32] (LP:
    #1507588)
    - [Config] Provides: virtualbox-guest-modules when appropriate

* Backport more recent driver for SKL, KBL and BXT graphics (LP: #1540390)
    - SAUCE: i915_bpo: Provide a backport driver for SKL, KBL & BXT graphics
    - SAUCE: i915_bpo: Update intel_ips.h file location
    - SAUCE: i915_bpo: Rename the backport driver to i915_bpo
    - SAUCE: i915_bpo: Add i915_bpo_*() calls for ubuntu/i915
    - drm/i915: remove an extra level of indirection in PCI ID list
    - drm/i915/kbl: Add Kabylake PCI ID
    - drm/i915/kbl: Add Kabylake GT4 PCI ID
    - mm: Export nr_swap_pages
    - async: export current_is_async()
    - drm: fix potential dangling else problems in for_each_ macros
    - dp/mst: add SDP stream support
    - drm: Implement drm_modeset_lock_all_ctx()
    - drm: Add "prefix" parameter to drm_rect_debug_print()
    - drm/i915: Set connector_state->connector using the helper.
    - drm/atomic: add connector mask to drm_crtc_state.
    - drm/i915: Report context GTT size
    - drm/i915: Add get_eld audio component
    - SAUCE: Backport I915_PARAM_HAS_EXEC_SOFTPIN and EXEC_OBJECT_PINNED
    - SAUCE: i915_bpo: Revert passing plane/encoder name
    - SAUCE: sound/hda: Load i915_bpo from the hda driver on SKL/KBL/BXT
    - SAUCE: i915_bpo: Support only SKL, KBL and BXT with the backport driver
    - drm/i915/bxt: update list of PCIIDs
    - drm/i915/skl: Add missing SKL ids
    - SAUCE: i915_bpo: Revert "drm/i915: Defer probe if gmux is present but its
      driver isn't"
    - SAUCE: uapi/drm/i915: Backport I915_EXEC_BSD_MASK
    - drm/atomic: Do not unset crtc when an encoder is stolen
    - drm/i915: Update connector_mask during readout, v2.
    - drm/atomic: Add encoder_mask to crtc_state, v3.
    - SAUCE: drm/core: Add drm_encoder_index.
    - SAUCE: i915_bpo: Revert "drm/i915: Switch DDC when reading the EDID"
    - i915_bpo: [Config] Enable CONFIG_DRM_I915_BPO=m

* arm64: guest hangs when ntpd is running (LP: #1549494)
    - hrtimer: Add support for CLOCK_MONOTONIC_RAW
    - hrtimer: Catch illegal clockids
    - KVM: arm/arm64: timer: Switch to CLOCK_MONOTONIC_RAW

* Miscellaneous Ubuntu changes
    - [Debian] git-ubuntu-log -- wrap long bug and commit titles
    - [Config] CONFIG_ARM_SMMU=y on arm64
    - rebase to v4.4.3
    - [Debian] git-ubuntu-log -- ensure we get the last commit
    - [Config] fix up spelling of probably again
    - [Debian] perf -- build in the context of the full generated local headers
    - SAUCE: tools: lib/bpf -- add generated headers to search path
    - SAUCE: proc: Always set super block owner to init_user_ns
    - SAUCE: fix-up: kern_mount fail path should not be doing put_buffers()
    - SAUCE: apparmor: Fix: oops do to invalid null ptr deref in label print fns
    - SAUCE: apparmor: debug: POISON label and replaceby pointer on free
    - SAUCE: apparmor: add underscores to indicate aa_label_next_not_in_set() use
      needs locking
    - SAUCE: apparmor: Fix: refcount leak in aa_label_merge
    - SAUCE: apparmor: ensure that repacedby sharing is done correctly
    - SAUCE: apparmor Fix: refcount bug in pivotroot mediation
    - SAUCE: apparmor: Fix: now that insert can force replacement use it instead
      of remove_and_insert
    - SAUCE: apparmor: Fix: refcount bug when inserting label update that
      transitions ns
    - SAUCE: apparmor: Fix: break circular refcount for label that is directly
      freed.
    - SAUCE: apparmor: Don't remove label on rcu callback if the label has already
      been removed
    - SAUCE: apparmor: Fix: query label file permission
    - SAUCE: apparmor: fix: ref count leak when profile sha1 hash is read
    - SAUCE: fixup: cleanup return handling of labels
    - SAUCE: fix: replacedby forwarding is not being properly update when ns is
      destroyed
    - SAUCE: fixup: make __share_replacedby private to get rid of build warning
    - SAUCE: fixup: 20/23 locking issue around in __label_update
    - SAUCE: fixup: get rid of unused var build warning
    - SAUCE: fixup: cast poison values to remove warnings
    - SAUCE: apparmor: fix refcount race when finding a child profile
    - SAUCE: fixup: warning about aa_label_vec_find_or_create not being static
    - SAUCE: fix: audit "no_new_privs" case for exec failure
    - SAUCE: Fixup: __label_update() still doesn't handle some cases correctly.
    - SAUCE: Move replacedby allocation into label_alloc
    - [Debian] supply zfs dkms Provides: based on do_zfs
    - [Config] supply zfs dkms Provides: based on do_zfs
    - [Config] drop linux-image-3.0 provides

* Miscellaneous upstream changes
    - x86/mpx: Fix off-by-one comparison with nr_registers

[ Upstream Kernel Changes ]

* rebase to v4.4.3

-- Tim Gardner <tim.gardner@canonical.com>  Thu, 25 Feb 2016 19:47:55 -0700

Changed in linux (Ubuntu):
status:	Confirmed → Fix Released

Revision history for this message

Tyler Hicks (tyhicks) wrote on 2017-05-09:

#24

I've marked the releases which have went end-of-life as "Won't Fix". I'm not sure if John is planning on SRU'ing the fix to Trusty so I'll leave that bug task alone.

Changed in linux (Ubuntu Vivid):
status:	Confirmed → Won't Fix
Changed in linux (Ubuntu Wily):
status:	Confirmed → Won't Fix
Changed in linux (Ubuntu Trusty):
assignee:	nobody → John Johansen (jjohansen)
Changed in apparmor:
assignee:	nobody → John Johansen (jjohansen)