After upgrading to 4.4.0-65-generic all of our Jenkins test runners are dying every 10 minutes or so. They don't answer on the network, on the console or through serial console.
The kernel backtraces we got are:
```
buildd04 login: [ 1443.707658] BUG: unable to handle kernel paging request at 2d5e501d
[ 1443.707969] IP: [<c11fb2ef>] mntget+0xf/0x20
[ 1443.708086] *pdpt = 0000000024056001 *pde = 0000000000000000
[ 1443.708237] Oops: 0002 [#1] SMP
[ 1443.708325] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables xt_comment veth ebtable_filter ebtables dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables zram lz4_compress bridge stp llc kvm_intel ppdev kvm irqbypass crc32_pclmul aesni_intel aes_i586 xts lrw gf128mul ablk_helper cryptd joydev input_leds serio_raw parport_pc 8250_fintek i2c_piix4 mac_hid lp parport autofs4 btrfs xor raid6_pq psmouse virtio_scsi pata_acpi floppy
[ 1443.710365] CPU: 1 PID: 14167 Comm: apparmor_parser Not tainted 4.4.0-65-generic #86-Ubuntu
[ 1443.710505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 1443.710651] task: f5920a00 ti: e63f2000 task.ti: e63f2000
[ 1443.710776] EIP: 0060:[<c11fb2ef>] EFLAGS: 00010286 CPU: 1
[ 1443.710875] EIP is at mntget+0xf/0x20
[ 1443.710946] EAX: f57e4d90 EBX: 00000000 ECX: c1d333cc EDX: 0002801d
[ 1443.711088] ESI: c1d36404 EDI: c1d36408 EBP: e63f3de8 ESP: e63f3de8
[ 1443.711228] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1443.711334] CR0: 80050033 CR2: 2d5e501d CR3: 35072440 CR4: 001406f0
[ 1443.711471] Stack:
[ 1443.711593] e63f3e04 c1203752 c13b7f71 c1d333cc eebb5980 e59d71e0 000041ed e63f3e30
[ 1443.711822] c130546b e59d7230 1a628dcf 00000003 ffffffff e63f3e58 6c0a010a e53b6800
[ 1443.712044] 000000de eebb5980 e63f3e44 c13055be 00000000 00000000 00000000 e63f3e6c
[ 1443.712264] Call Trace:
[ 1443.712314] [<c1203752>] simple_pin_fs+0x32/0xa0
[ 1443.712421] [<c13b7f71>] ? vsnprintf+0x321/0x420
[ 1443.712516] [<c130546b>] securityfs_create_dentry+0x5b/0x150
[ 1443.712632] [<c13055be>] securityfs_create_dir+0x2e/0x30
[ 1443.712729] [<c133a3c6>] __aa_fs_profile_mkdir+0x46/0x3c0
[ 1443.712826] [<c1345000>] aa_replace_profiles+0x4c0/0xbc0
[ 1443.712927] [<c10798c5>] ? ns_capable_common+0x55/0x80
[ 1443.713022] [<c1338ee7>] policy_update+0x97/0x230
[ 1443.713122] [<c1302189>] ? security_file_permission+0x39/0xc0
[ 1443.713247] [<c1339118>] profile_replace+0x98/0xe0
[ 1443.713346] [<c1339080>] ? policy_update+0x230/0x230
[ 1443.713445] [<c11dd99f>] __vfs_write+0x1f/0x50
[ 1443.713535] [<c11ddf7c>] vfs_write+0x8c/0x1b0
[ 1443.713633] [<c11de971>] SyS_write+0x51/0xb0
[ 1443.713738] [<c100385d>] do_fast_syscall_32+0x8d/0x150
[ 1443.713838] [<c17bcd9c>] sysenter_past_esp+0x3d/0x61
[ 1443.713938] Code: c0 74 09 83 42 10 01 89 d0 5b 5d c3 3b 5b 10 b8 fe ff ff ff 75 e3 eb eb 8d 74 26 00 55 89 e5 3e 8d 74 26 00 85 c0 74 06 8b 50 14 <64> ff 02 5d c3 8d b6 00 00 00 00 8d bf 00 00 00 00 55 89 e5 3e
[ 1443.715713] EIP: [<c11fb2ef>] mntget+0xf/0x20 SS:ESP 0068:e63f3de8
[ 1443.715852] CR2: 000000002d5e501d
```
```
buildd07 login: [ 1262.522071] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[ 1262.522339] IP: [<ffffffff8122fbd8>] mntput_no_expire+0x68/0x180
[ 1262.522464] PGD 439912067 PUD 43997f067 PMD 0
[ 1262.522556] Oops: 0002 [#1] SMP
[ 1262.522760] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables xt_comment veth ebtable_filter ebtables dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc zram lz4_compress zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ppdev ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd input_leds joydev i2c_piix4 serio_raw 8250_fintek parport_pc mac_hid lp parport autofs4 btrfs xor raid6_pq psmouse virtio_scsi pata_acpi floppy
[ 1262.535658] CPU: 10 PID: 163332 Comm: apparmor_parser Tainted: P O 4.4.0-65-generic #86-Ubuntu
[ 1262.536544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 1262.536666] task: ffff88043c3fd400 ti: ffff88044fed0000 task.ti: ffff88044fed0000
[ 1262.536773] RIP: 0010:[<ffffffff8122fbd8>] [<ffffffff8122fbd8>] mntput_no_expire+0x68/0x180
[ 1262.536949] RSP: 0018:ffff88044fed3d70 EFLAGS: 00010206
[ 1262.537046] RAX: 0000000000000000 RBX: ffff88046a74e480 RCX: 0000000000000000
[ 1262.537205] RDX: 0000000000000000 RSI: 0000000000000200 RDI: ffffffff81f38c40
[ 1262.537354] RBP: ffff88044fed3d88 R08: 0000000000000000 R09: 0000000000000fff
[ 1262.537512] R10: ffff88043aeedb00 R11: 0000000000000246 R12: ffffffff821dbde8
[ 1262.537787] R13: ffff88046a74e4d0 R14: ffff880626c8ca38 R15: ffff880626c91388
[ 1262.538021] FS: 00007fe98994c740(0000) GS:ffff880627480000(0000) knlGS:0000000000000000
[ 1262.538324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1262.538492] CR2: 0000000000000008 CR3: 0000000446b88000 CR4: 00000000001406e0
[ 1262.538688] Stack:
[ 1262.538749] ffffffff821dbde0 ffffffff821dbde8 ffff88043c12d730 ffff88044fed3d98
[ 1262.539034] ffffffff8122fd14 ffff88044fed3db8 ffffffff81236ea5 ffff88043b269600
[ 1262.539372] ffff88043b269000 ffff88044fed3dd8 ffffffff8134aae9 ffff88043c12d758
[ 1262.539657] Call Trace:
[ 1262.539717] [<ffffffff8122fd14>] mntput+0x24/0x40
[ 1262.539888] [<ffffffff81236ea5>] simple_release_fs+0x45/0x50
[ 1262.540041] [<ffffffff8134aae9>] securityfs_remove+0x99/0xb0
[ 1262.540198] [<ffffffff81382892>] __aa_fs_profile_rmdir+0x72/0xd0
[ 1262.540358] [<ffffffff8138c9b0>] __remove_profile+0x40/0xd0
[ 1262.540494] [<ffffffff8138e9e0>] aa_remove_profiles+0xe0/0x370
[ 1262.540654] [<ffffffff813819a4>] profile_remove+0x144/0x2e0
[ 1262.540833] [<ffffffff8120e668>] __vfs_write+0x18/0x40
[ 1262.540945] [<ffffffff8120eff9>] vfs_write+0xa9/0x1a0
[ 1262.541082] [<ffffffff8120df8f>] ? do_sys_open+0x1bf/0x2a0
[ 1262.541233] [<ffffffff8120fcb5>] SyS_write+0x55/0xc0
[ 1262.541359] [<ffffffff8183c672>] entry_SYSCALL_64_fastpath+0x16/0x71
[ 1262.541521] Code: ff 85 c0 0f 85 f6 00 00 00 8b 43 30 a9 00 00 00 01 0f 85 e8 00 00 00 0d 00 00 00 01 48 8b 53 70 4c 8d 6b 50 89 43 30 48 8b 43 78 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 70
[ 1262.544762] RIP [<ffffffff8122fbd8>] mntput_no_expire+0x68/0x180
[ 1262.545050] RSP <ffff88044fed3d70>
[ 1262.545142] CR2: 0000000000000008
```
```
buildd06 login: [ 1330.969096] BUG: unable to handle kernel paging request at 2d5b2000
[ 1330.971286] IP: [<c11fb2ef>] mntget+0xf/0x20
[ 1330.972781] *pdpt = 00000000298be001 *pde = 0000000000000000
[ 1330.980802] Oops: 0002 [#1] SMP
[ 1330.982401] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables xt_comment dm_snapshot dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c veth ebtable_filter ebtables binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc zram lz4_compress kvm_intel ppdev kvm irqbypass crc32_pclmul aesni_intel aes_i586 xts lrw gf128mul joydev input_leds ablk_helper cryptd serio_raw i2c_piix4 8250_fintek parport_pc mac_hid lp parport autofs4 btrfs xor raid6_pq psmouse virtio_scsi floppy pata_acpi
[ 1330.987438] CPU: 0 PID: 21383 Comm: lxd Not tainted 4.4.0-65-generic #86-Ubuntu
[ 1330.987629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 1330.987827] task: f565e400 ti: e6c22000 task.ti: e6c22000
[ 1330.987952] EIP: 0060:[<c11fb2ef>] EFLAGS: 00210286 CPU: 0
[ 1330.988080] EIP is at mntget+0xf/0x20
[ 1330.988174] EAX: d1dbd190 EBX: 00000000 ECX: c1d333cc EDX: 00004000
[ 1330.988332] ESI: c1d36404 EDI: c1d36408 EBP: e6c23ec8 ESP: e6c23ec8
[ 1330.988493] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1330.988625] CR0: 80050033 CR2: 2d5b2000 CR3: 35762380 CR4: 001406f0
[ 1330.988787] Stack:
[ 1330.988851] e6c23ee4 c1203752 e6c23eec c1d333cc eec950c0 00000000 eec950dc e6c23eec
[ 1330.989186] c130566c e6c23f20 c1338784 eebd6dc0 00000003 e3127100 e6c23f14 eebd6e30
[ 1330.989509] 000001ed e3127100 eebd6dc0 eebd6dc0 c17e7140 e3127100 e6c23f48 c11e8a2e
[ 1330.989832] Call Trace:
[ 1330.989902] [<c1203752>] simple_pin_fs+0x32/0xa0
[ 1330.990048] [<c130566c>] securityfs_pin_fs+0x1c/0x20
[ 1330.990176] [<c1338784>] ns_mkdir_op+0xe4/0x370
[ 1330.990306] [<c11e8a2e>] vfs_mkdir+0xee/0x1d0
[ 1330.990436] [<c1349ec0>] ? wrap_apparmor_path_mkdir+0x20/0x30
[ 1330.990595] [<c11edc2f>] SyS_mkdirat+0xcf/0x110
[ 1330.990725] [<c1003773>] do_syscall_32_irqs_on+0x53/0xb0
[ 1330.990855] [<c17bcdf1>] entry_INT80_32+0x31/0x31
[ 1330.990982] Code: c0 74 09 83 42 10 01 89 d0 5b 5d c3 3b 5b 10 b8 fe ff ff ff 75 e3 eb eb 8d 74 26 00 55 89 e5 3e 8d 74 26 00 85 c0 74 06 8b 50 14 <64> ff 02 5d c3 8d b6 00 00 00 00 8d bf 00 00 00 00 55 89 e5 3e
[ 1330.993186] EIP: [<c11fb2ef>] mntget+0xf/0x20 SS:ESP 0068:e6c23ec8
[ 1330.993386] CR2: 000000002d5b2000
```
```
buildd05 login: [ 1400.626415] BUG: unable to handle kernel paging request at 2d62201d
[ 1400.634202] IP: [<c11fc961>] mntput_no_expire+0x11/0x160
[ 1400.634805] *pdpt = 00000000350d5001 *pde = 0000000000000000
[ 1400.634971] Oops: 0002 [#1] SMP
[ 1400.635078] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables xt_comment veth ebtable_filter ebtables binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc zram lz4_compress kvm_intel kvm irqbypass crc32_pclmul aesni_intel aes_i586 xts input_leds joydev lrw gf128mul i2c_piix4 ppdev 8250_fintek ablk_helper mac_hid cryptd serio_raw parport_pc lp parport autofs4 btrfs xor raid6_pq psmouse virtio_scsi pata_acpi floppy
[ 1400.638087] CPU: 4 PID: 10296 Comm: lxd Not tainted 4.4.0-65-generic #86-Ubuntu
[ 1400.638300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
[ 1400.638488] task: e5369e00 ti: e1072000 task.ti: e1072000
[ 1400.638615] EIP: 0060:[<c11fc961>] EFLAGS: 00210282 CPU: 4
[ 1400.638743] EIP is at mntput_no_expire+0x11/0x160
[ 1400.638866] EAX: f546a180 EBX: c1d36404 ECX: 00000000 EDX: 0002801d
[ 1400.639040] ESI: c1d36408 EDI: e2f4e6d0 EBP: e1073e64 ESP: e1073e54
[ 1400.639206] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 1400.639371] CR0: 80050033 CR2: 2d62201d CR3: 352eb1a0 CR4: 001406f0
[ 1400.639620] Stack:
[ 1400.639692] f4ad0848 c1d36404 c1d36408 e2f4e6d0 e1073e6c c11fcad0 e1073e7c c12037f7
[ 1400.641243] eb808e00 f4ad0800 e1073e8c c1305634 e2f4e73c e2f4e724 e1073eb4 c133a892
[ 1400.641582] c1af6740 00000001 00000000 00000000 e2f4e708 00200286 e2f4e6c0 e2f4e71c
[ 1400.641920] Call Trace:
[ 1400.641989] [<c11fcad0>] mntput+0x20/0x40
[ 1400.642083] [<c12037f7>] simple_release_fs+0x37/0x40
[ 1400.642221] [<c1305634>] securityfs_remove+0x74/0x90
[ 1400.642359] [<c133a892>] __aa_fs_ns_rmdir+0x152/0x1d0
[ 1400.642487] [<c1359c20>] destroy_ns+0x90/0xa0
[ 1400.642622] [<c1359b64>] __aa_remove_ns+0x24/0x50
[ 1400.642929] [<c133966f>] ns_rmdir_op+0x15f/0x3a0
[ 1400.643030] [<c11e8161>] vfs_rmdir+0xa1/0x110
[ 1400.643099] [<c11ed0dd>] do_rmdir+0x1cd/0x1f0
[ 1400.643169] [<c11edddd>] SyS_unlinkat+0x2d/0x40
[ 1400.643239] [<c1003773>] do_syscall_32_irqs_on+0x53/0xb0
[ 1400.643307] [<c17bcdf1>] entry_INT80_32+0x31/0x31
[ 1400.643373] Code: f0 e8 34 3e 1c 00 3b 05 74 27 b9 c1 7c dc 89 d8 5b 5e 5f 5d c3 90 8d 74 26 00 55 89 e5 57 56 53 83 ec 04 3e 8d 74 26 00 8b 50 24 <64> ff 0a 8b 50 70 85 d2 74 0d 83 c4 04 5b 5e 5f 5d c3 90 8d 74
[ 1400.644544] EIP: [<c11fc961>] mntput_no_expire+0x11/0x160 SS:ESP 0068:e1073e54
[ 1400.644675] CR2: 000000002d62201d
```
We can reproduce this very easily by triggering a LXD testsuite run which causes a lot of apparmor profiles and namespaces creation/deletion, causing this issue. A busy LXD host would also hit this eventually (if the similar BUG we had before is any indication).