audit: improve audit queue handling when "audit=1" on cmdline

Bug #1965723 reported by gerald.yang
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Committed
Critical
gerald.yang
Bionic
Fix Released
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned

Bug Description

SRU Justification

[Impact]
When an admin enables audit at early boot via the "audit=1" kernel
command line the audit queue behavior is slightly different; the
audit subsystem goes to greater lengths to avoid dropping records,
which unfortunately can result in problems when the audit daemon is
forcibly stopped for an extended period of time.

[Fix]
upstream discussion:
https://lore.kernel<email address hidden>/T/
upstream commit:
f26d04331360d42dbd6b58448bd98e4edbfbe1c5

[Test]
configurations:
    auditctl -b 64
    auditctl --backlog_wait_time 60000
    auditctl -r 0
    auditctl -w /root/aaa -p wrx
shell scripts:
    #!/bin/bash
    i=0
    while [ $i -le 66 ]
    do
        touch /root/aaa
     let i++
    done
mandatory conditions:
    add "audit=1" to the cmdline, and kill -19 pid_number(for /sbin/auditd).

As long as we keep the audit_hold_queue non-empty, flush the hold queue will fall into an infinite loop.

This could also trigger soft lockup when it drops into a infinite loop, e.g.
kernel: [ 94.186433] watchdog: BUG: soft lockup - CPU#2 stuck for 11s! [kauditd:34]
kernel: [ 94.187736] Modules linked in: xfs iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_
conntrack libcrc32c iptable_filter isofs xt_cgroup xt_tcpudp iptable_mangle ip_tables x_tables sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 pp
dev crypto_simd glue_helper joydev vmwgfx ttm cryptd vmw_balloon drm_kms_helper intel_rapl_perf input_leds psmouse drm fb_sys_fops syscopyarea vmxnet3 sysfillrect parport_pc parport m
ac_hid shpchp i2c_piix4 vmw_vsock_vmci_transport vsock sysimgblt vmw_vmci serio_raw mptspi mptscsih mptbase scsi_transport_spi pata_acpi floppy autofs4
kernel: [ 94.187757] CPU: 2 PID: 34 Comm: kauditd Not tainted 4.15.0-171-generic #180~16.04.1-Ubuntu
kernel: [ 94.187757] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS
 6.00 11/12/2020
kernel: [ 94.187800] skb_queue_head+0x47/0x50
kernel: [ 94.187803] kauditd_rehold_skb+0x18/0x20
kernel: [ 94.187805] kauditd_send_queue+0xcd/0x100
kernel: [ 94.187806] ? kauditd_retry_skb+0x20/0x20
kernel: [ 94.187808] ? kauditd_send_multicast_skb+0x80/0x80
kernel: [ 94.187809] kauditd_thread+0xa7/0x240
kernel: [ 94.187812] ? wait_woken+0x80/0x80
kernel: [ 94.187815] kthread+0x105/0x140
kernel: [ 94.187817] ? auditd_reset+0x90/0x90
kernel: [ 94.187818] ? kthread_bind+0x40/0x40
kernel: [ 94.187820] ret_from_fork+0x35/0x40

[Other Info]
SF: #00330803

CVE References

Changed in linux (Ubuntu):
assignee: nobody → gerald.yang (gerald-yang-tw)
importance: Undecided → Critical
status: New → In Progress
Changed in linux (Ubuntu Impish):
status: New → In Progress
Changed in linux (Ubuntu Focal):
status: New → In Progress
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Changed in linux (Ubuntu Impish):
importance: Undecided → Critical
Changed in linux (Ubuntu Focal):
importance: Undecided → Critical
Changed in linux (Ubuntu Bionic):
importance: Undecided → Critical
Changed in linux (Ubuntu Impish):
assignee: nobody → gerald.yang (gerald-yang-tw)
Changed in linux (Ubuntu Focal):
assignee: nobody → gerald.yang (gerald-yang-tw)
Changed in linux (Ubuntu Bionic):
assignee: nobody → gerald.yang (gerald-yang-tw)
tags: added: sts
no longer affects: linux (Ubuntu Jammy)
Revision history for this message
gerald.yang (gerald-yang-tw) wrote :

This one has been merged in the last upstream stable patch set

no longer affects: linux (Ubuntu Impish)
no longer affects: linux (Ubuntu Focal)
no longer affects: linux (Ubuntu Bionic)
Changed in linux (Ubuntu):
status: In Progress → Invalid
Revision history for this message
Kenyon Ralph (kralph) wrote (last edit ):

Focal is LP: #1964634. LP: #1964422 is for Impish.

Revision history for this message
gerald.yang (gerald-yang-tw) wrote (last edit ):

the fix in Impish is still in progress here LP: #1966021

Changed in linux (Ubuntu Impish):
status: New → Confirmed
Changed in linux (Ubuntu Focal):
status: New → Confirmed
Changed in linux (Ubuntu Bionic):
status: New → Confirmed
Changed in linux (Ubuntu Impish):
status: Confirmed → Fix Committed
Changed in linux (Ubuntu Focal):
status: Confirmed → Fix Committed
Changed in linux (Ubuntu Bionic):
status: Confirmed → Fix Committed
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

For Impish, I have cherry-picked the fix from the pull-request for bug 1966021 and applied to the master-next branch and it will be included in the current kernel SRU cycle re-spin.

Note that for Focal the patch was already applied for a re-spin and I wasn't able to include the additional BugLink to the commit message, therefore this bug report won't receive automatic updates about the fix.

Revision history for this message
gerald.yang (gerald-yang-tw) wrote :

Thanks a lot for the help Kleber

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/4.15.0-176.185 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.13.0-40.45 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-impish' to 'verification-done-impish'. If the problem still exists, change the tag 'verification-needed-impish' to 'verification-failed-impish'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-impish
Revision history for this message
John Lewis (jlewis-johnlewis-deactivatedaccount) wrote :

I've reproduced the original problem then been running proposed kernels in bionic and focal now for at least 2 hours, and problem seems fixed:

root@machine-1:~# uname -r
4.15.0-176-generic
root@machine-1:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
root@machine-1:~#

5.4.0-108-generic
root@machine-2:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
root@machine-2:~#

Marking as verified in bionic - should we be doing so in focal too?

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hello John,

Thank you for verifying the fix. The verification-needed tag was not added automatically for Focal because the commit contains only the link to the upstream stable update https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964634. I'll add the tag manually to flag that it has been verified.

Thank you!

tags: added: verification-done-focal
Revision history for this message
Simon Mijolovic (smijolovic) wrote :

Identified this issue in Focal starting with the update to 105 kernel. Constant soft lockups then freezing when kauditd/auditd exceeded backlog limit. Due to DoD requirements, all systems set to audit=1 in kernel boot options and have stringent list of syscalls/binaries/files that are audited.

Added focal-proposed repo and updated to 109 kernel. Few hours of compiling and heavy static code analysis loads on system and no lockups as of yet.

This is a really scary bug - seems to be present on all linux kernels and a huge threat as a DDoS.

Changed in linux (Ubuntu):
status: Invalid → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (53.1 KiB)

This bug was fixed in the package linux - 5.13.0-40.45

---------------
linux (5.13.0-40.45) impish; urgency=medium

  * impish/linux: 5.13.0-40.45 -proposed tracker (LP: #1966701)

  * CVE-2022-1016
    - netfilter: nf_tables: initialize registers in nft_do_chain()

  * CVE-2022-1015
    - netfilter: nf_tables: validate registers coming from userspace.

  * audit: improve audit queue handling when "audit=1" on cmdline
    (LP: #1965723) // Impish update: upstream stable patchset 2022-03-22
    (LP: #1966021)
    - audit: improve audit queue handling when "audit=1" on cmdline

  * PS/2 Keyboard wakeup from s2idle not functioning on AMD Yellow Carp platform
    (LP: #1961739)
    - PM: s2idle: ACPI: Fix wakeup interrupts handling

  * Low RX performance for 40G Solarflare NICs (LP: #1964512)
    - SAUCE: sfc: The size of the RX recycle ring should be more flexible

  * [UBUNTU 20.04] Fix SIGP processing on KVM/s390 (LP: #1962578)
    - KVM: s390: Simplify SIGP Set Arch handling
    - KVM: s390: Add a routine for setting userspace CPU state

  * Move virtual graphics drivers from linux-modules-extra to linux-modules
    (LP: #1960633)
    - [Packaging] Move VM DRM drivers into modules

  * Impish update: upstream stable patchset 2022-03-09 (LP: #1964422)
    - bnx2x: Utilize firmware 7.13.21.0
    - bnx2x: Invalidate fastpath HSI version for VFs
    - rcu: Tighten rcu_advance_cbs_nowake() checks
    - select: Fix indefinitely sleeping task in poll_schedule_timeout()
    - drm/amdgpu: Use correct VIEWPORT_DIMENSION for DCN2
    - arm64/bpf: Remove 128MB limit for BPF JIT programs
    - Bluetooth: refactor malicious adv data check
    - net: sfp: ignore disabled SFP node
    - net: stmmac: skip only stmmac_ptp_register when resume from suspend
    - s390/hypfs: include z/VM guests with access control group set
    - bpf: Guard against accessing NULL pt_regs in bpf_get_task_stack()
    - scsi: zfcp: Fix failed recovery on gone remote port with non-NPIV FCP
      devices
    - udf: Restore i_lenAlloc when inode expansion fails
    - udf: Fix NULL ptr deref when converting from inline format
    - efi: runtime: avoid EFIv2 runtime services on Apple x86 machines
    - PM: wakeup: simplify the output logic of pm_show_wakelocks()
    - tracing/histogram: Fix a potential memory leak for kstrdup()
    - tracing: Don't inc err_log entry count if entry allocation fails
    - ceph: properly put ceph_string reference after async create attempt
    - ceph: set pool_ns in new inode layout for async creates
    - fsnotify: fix fsnotify hooks in pseudo filesystems
    - Revert "KVM: SVM: avoid infinite loop on NPF from bad address"
    - perf/x86/intel/uncore: Fix CAS_COUNT_WRITE issue for ICX
    - drm/etnaviv: relax submit size limits
    - KVM: x86: Update vCPU's runtime CPUID on write to MSR_IA32_XSS
    - netfilter: nft_payload: do not update layer 4 checksum when mangling
      fragments
    - serial: 8250: of: Fix mapped region size when using reg-offset property
    - serial: stm32: fix software flow control transfer
    - tty: n_gsm: fix SW flow control encoding/handling
    - tty: Add support for Brainboxes UC cards.
    - usb-storage: Add unusual-devs...

Changed in linux (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.1 KiB)

This bug was fixed in the package linux - 4.15.0-176.185

---------------
linux (4.15.0-176.185) bionic; urgency=medium

  * bionic/linux: 4.15.0-176.185 -proposed tracker (LP: #1966771)

  * Bionic update: upstream stable patchset 2022-03-04 (LP: #1963717)
    - can: bcm: fix UAF of bcm op
    - net: bridge: clear bridge's private skb space on xmit
    - s390/hypfs: include z/VM guests with access control group set
    - scsi: zfcp: Fix failed recovery on gone remote port with non-NPIV FCP
      devices
    - udf: Restore i_lenAlloc when inode expansion fails
    - udf: Fix NULL ptr deref when converting from inline format
    - PM: wakeup: simplify the output logic of pm_show_wakelocks()
    - netfilter: nft_payload: do not update layer 4 checksum when mangling
      fragments
    - serial: stm32: fix software flow control transfer
    - tty: n_gsm: fix SW flow control encoding/handling
    - tty: Add support for Brainboxes UC cards.
    - usb-storage: Add unusual-devs entry for VL817 USB-SATA bridge
    - usb: common: ulpi: Fix crash in ulpi_match()
    - usb: gadget: f_sourcesink: Fix isoc transfer for USB_SPEED_SUPER_PLUS
    - USB: core: Fix hang in usb_kill_urb by adding memory barriers
    - usb: typec: tcpm: Do not disconnect while receiving VBUS off
    - net: sfp: ignore disabled SFP node
    - powerpc/32: Fix boot failure with GCC latent entropy plugin
    - lkdtm: Fix content of section containing lkdtm_rodata_do_nothing()
    - i40e: Increase delay to 1 s after global EMP reset
    - i40e: fix unsigned stat widths
    - rpmsg: char: Fix race between the release of rpmsg_ctrldev and cdev
    - rpmsg: char: Fix race between the release of rpmsg_eptdev and cdev
    - scsi: bnx2fc: Flush destroy_work queue before calling bnx2fc_interface_put()
    - ipv6_tunnel: Rate limit warning messages
    - net: fix information leakage in /proc/net/ptype
    - ping: fix the sk_bound_dev_if match in ping_lookup
    - ipv4: avoid using shared IP generator for connected sockets
    - hwmon: (lm90) Reduce maximum conversion rate for G781
    - NFSv4: Handle case where the lookup of a directory fails
    - NFSv4: nfs_atomic_open() can race when looking up a non-regular file
    - net-procfs: show net devices bound packet types
    - drm/msm: Fix wrong size calculation
    - drm/msm/dsi: invalid parameter check in msm_dsi_phy_enable
    - ibmvnic: don't spin in tasklet
    - yam: fix a memory leak in yam_siocdevprivate()
    - ipv4: raw: lock the socket in raw_bind()
    - ipv4: tcp: send zero IPID in SYNACK messages
    - netfilter: nat: remove l4 protocol port rovers
    - netfilter: nat: limit port clash resolution attempts
    - ipheth: fix EOVERFLOW in ipheth_rcvbulk_callback
    - net: amd-xgbe: ensure to reset the tx_timer_active flag
    - net: amd-xgbe: Fix skb data length underflow
    - rtnetlink: make sure to refresh master_dev/m_ops in __rtnl_newlink()
    - af_packet: fix data-race in packet_setsockopt / packet_setsockopt
    - ASoC: ops: Reject out of bounds values in snd_soc_put_volsw()
    - ASoC: ops: Reject out of bounds values in snd_soc_put_volsw_sx()
    - ASoC: ops: Reject out of bounds values in snd_soc_put_xr_sx()
    - drm/n...

Read more...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Manually marking focal/linux as 'Fix Released' as the fix was released under bug 1964634.

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.