udev fails to make prctl() syscall with apparmor=0 (as used by maas by default)

From the logs:

cloud-initramfs-dyn-netconf: did no find a nic with 00:18:2d:04:00:c0

This kernel is failing to initialize the NICs

I can confirm this also happens in a LXD VM running the latest Lunar image from http://images.maas.io/ephemeral-v3/candidate/

Begin: Running /scripts/init-premount ... cloud-initramfs-dyn-netconf: did no find a nic with 00:16:3e:c7:66:a0

I see no mention of virtio_net being initialized in the logs

The root of the problem is that the kernel's networking modules are not being loaded, but no idea why this is yet. When comparing logs from an earlier maas deployment that loads the squashfs, I see this:

Starting systemd-udevd version 252.5-2ubuntu2
[ 24.561035] IPMI message handler: version 39.2
[ 24.579284] ipmi device interface
[ 24.589680] xhci_hcd 0004:03:00.0: Adding to iommu group 37
[ 24.594182] ACPI: bus type drm_connector registered
[ 24.596816] xhci_hcd 0004:03:00.0: failed to load firmware renesas_usb_fw.mem, fallback to ROM
[ 24.601479] ipmi_ssif: IPMI SSIF Interface driver
[ 24.608865] xhci_hcd 0004:03:00.0: xHCI Host Controller
[ 24.617397] ipmi_si: IPMI System Interface driver
[ 24.618771] xhci_hcd 0004:03:00.0: new USB bus registered, assigned bus number 1
[ 24.626843] ipmi_si: Unable to find any System Interface(s)
[ 24.626990] ipmi_ssif i2c-AMPC0004:00: ipmi_ssif: Trying ACPI-specified SSIF interface at i2c address 0x10, adapter Synopsys DesignWare I2C adapter, slave address 0x20
[ 24.630860] xhci_hcd 0004:03:00.0: Zeroing 64bit base registers, expecting fault
[ 24.768420] ipmi_ssif i2c-AMPC0004:00: IPMI message handler: Found new BMC (man_id: 0x00cd3a, prod_id: 0x0082, dev_id: 0x20)
[ 24.800956] xhci_hcd 0004:03:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x0000001100000410
[ 24.814231] xhci_hcd 0004:03:00.0: xHCI Host Controller
[ 24.819491] xhci_hcd 0004:03:00.0: new USB bus registered, assigned bus number 2
[ 24.826888] xhci_hcd 0004:03:00.0: Host supports USB 3.0 SuperSpeed
...
[ 24.960323] mlx5_core 0000:01:00.0: Adding to iommu group 39
[ 24.967051] mlx5_core 0000:01:00.0: firmware version: 14.26.1040
[ 24.973085] mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
...
+ many many more lines
Begin: Loading essential drivers ... [ 28.602841] raid6: neonx8 gen() 15741 MB/s

Within that section of logs is where I see the network device being recognized (mlx5_core in this case). From the failing case, we don't see any of this:

[ 21.660837] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.5-2ubuntu3
Begin: Loading essential drivers ... [ 25.605516] raid6: neonx8 gen() 13160 MB/s
[ 25.677515] raid6: neonx4 gen() 11952 MB/s

I did notice that there is a newer version of udev 252.5-2ubuntu3 over 252.5-2ubuntu2, so I'm suspicious of the udev or the udev rules being a possible source of the issue. I ran an experiment of copying all of the files in the udev package from an older lunar install and repacked them into the boot-initrd. This didn't help, although I systemd-udevd reported the version from the system I copied from (further indicating that the boot-initrd is being loaded and used).

I'm going to continue experimenting with this. My first attempt only copied the udev rules present in the udev package. There are other rules installed by other packages that I didn't copy yet.

do you have a full last working log?

As in the failing one i see a lot of firmware corruption, and ACPI bugs, because it seems as if the system doesn't declare that it has a bus, or network devices attached to it.

1) alexsander-souza - last working full serial/console log from lxd-vm
2) alexsander-souza - non-working serial/console log from lxd-vm
3) fginther - last working full serial/console log from that working machine

i am assuming that things did work since 28th of March, when bootloaders were updated in the stream.

qemu-system-x86_64 -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append break=bottom -nic user,model=virtio-net-pci

using boot-initrd & boot-kernel from the stream, and using the above command, i do get virtio-net loaded and usable.

one can test that things work by booting with option `break=top` and then replicate that init-top will do:

SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-udevd --daemon --resolve-names=never
udevadm trigger --type=subsystems --action=add
udevadm trigger --type=devices --action=add
udevadm settle || true

After these things (during the --type=devices --action=add) virtio-net is loaded, in a VM.

If it doesn't.... something is borked before kernel. I.e. grub, shim, memory, firmware.

given the large jump in the serials it does jump a kernel; userspace components of initrd; and bootloaders too.

it doesn't help to triange things.

If it is a kernel regression, it likely to be severe - as basically we are failing to detect and correctly enumerate things in ACPI.

I don't think it is a userspace regression, as in small trials it is working correctly.

Console log of "fili" deployed using the 20230319 kernel and initrd.

Console log of "fili" deployed using the 20230419 kernel and initrd (fails on networking).

For francis case, indeed there is like everything missing:

Broken:

Loading, please wait...
Starting systemd-udevd version 252.5-2ubuntu3
Begin: Loading essential drivers ... [ 11.584699] raid6: avx2x4 gen() 24389 MB/s
[ 11.664701] raid6: avx2x2 gen() 23865 MB/s
[ 11.744685] raid6: avx2x1 gen() 21744 MB/s
[ 11.759413] raid6: using algorithm avx2x4 gen() 24389 MB/s
[ 11.744685] raid6: avx2x1 gen() 21744 MB/s
[ 11.759413] raid6: using algorithm avx2x4 gen() 24389 MB/s
[ 11.840704] raid6: .... xor() 7897 MB/s, rmw enabled
[ 11.855975] raid6: using avx2x2 recovery algorithm
[ 11.872387] xor: automatically using best checksumming function avx
[ 11.891201] async_tx: api initialized (async)
done.

whereas the working case has:

Loading, please wait...
Starting systemd-udevd version 252.5-2ubuntu2
[ 9.990796] dca service started, version 1.12.1
[ 10.005835] i801_smbus 0000:00:1f.3: enabling device (0001 -> 0003)
[ 10.022075] i801_smbus 0000:00:1f.3: SMBus using PCI interrupt
[ 10.037480] i801_smbus 0000:00:11.1: enabling device (0000 -> 0003)
[ 10.053433] cryptd: max_cpu_qlen set to 1000
....
(all the things being triggered by udev and being loaded).

The firmware level stuff looks the same, in both cases all the EFI passed tables are in the same locations.

Boot of the 20230319 kernel (6.1) and initrd with udev debug enabled.

Boot of the 20230418 kernel (6.2) and initrd with udev debug enabled.

Observations from udev debug logs:

* The 0418 log has lots of "Failed to set death signal: Invalid argument" not present in 0319
* In the 0418 log, the udev rules files are read, but I so no indication of them being used.
* In 0418, worker threads are "exiting with return code 1":
$ grep "block:" udev-debug-0418.log
block: Device is queued (SEQNUM=2697, ACTION=add)
block: Device ready for processing (SEQNUM=2697, ACTION=add)
block: Worker [690] is forked for processing SEQNUM=2697.
block: Worker [690] exited with return code 1.
block: sd-device-monitor(manager): Passed 212 byte to netlink monitor.

* In 0319, these appear to be successful:
$ grep "block: " udev-debug-0319.log
block: Device is queued (SEQNUM=2719, ACTION=add)
block: Device ready for processing (SEQNUM=2719, ACTION=add)
block: sd-device-monitor(manager): Passed 165 byte to netlink monitor.
block: Processing device (SEQNUM=2719, ACTION=add)
block: /usr/lib/udev/rules.d/60-block.rules:5 ATTR '/sys/module/block/parameters/events_dfl_poll_msecs' writing '2000'
block: Device processed (SEQNUM=2719, ACTION=add)
block: sd-device-monitor(worker): Passed 165 byte to netlink monitor.

So, it appears all of the workers that udev spawns to process all of these events are failing. The "Failed to set death signal: Invalid argument" message comes from here:

        if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT))
                if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ? SIGINT : SIGTERM) < 0) {
                        log_full_errno(prio, errno, "Failed to set death signal: %m");
                        _exit(EXIT_FAILURE);
                }

and the error exit then leads to the "exiting with return code 1" message.

Why is this failing? To get an "Invalid argument" (if that can be trusted), either PR_SET_PDEATHSIG, SIGINT or SIGTERM would appear to need to be out of the valid range of values.

Kernel side:

kernel/sys.c::SYSCALL_DEFINE5(prctl, int, option,...):

{
        struct task_struct *me = current;
        unsigned char comm[sizeof(me->comm)];
        long error;

        error = security_task_prctl(option, arg2, arg3, arg4, arg5);
        if (error != -ENOSYS)
                return error;

        error = 0;
        switch (option) {
        case PR_SET_PDEATHSIG:
                if (!valid_signal(arg2)) {
                        error = -EINVAL;
                        break;
                }
                me->pdeath_signal = arg2;
                break;

...
        return error;
}

and include/linux/signal.h::valid_signal():

/* Test if 'sig' is valid signal. Use this instead of testing _NSIG directly */
static inline int valid_signal(unsigned long sig)
{
        return sig <= _NSIG ? 1 : 0;
}

and arch/x86/include/asm/signal.h:#define _NSIG 64

I wonder about security_task_prctl() though.

I am annoyed that i cannot reproduce this locally outside of MAAS.

We discussed fixes to the maas bootloader builds and as a result, we have a new maas bootloader which should have the latest shim and grub:

http://images.maas.io/ephemeral-v3/candidate/bootloaders/uefi/amd64/20230420.0/

I manually copied these to my test maas, it did not resolve the issue (or have any other visible change).

Paolo built a kernel to try and debug this from the prctl interface. With this (and udev with -debug), I see these two additional console log entries:

[ 18.029479] __do_sys_prctl::2375
[ 18.042840] __do_sys_prctl::2380

There are lots of these, but only the same two lines.

horay i managed to reproduce it locally.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2016908

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I can confirm @xnox's findings with my maas server deploying lunar. Adding `apparmor=1` to the settings/configuration/kernel-parameters allows for a successful deployment with the lunar 6.2.0-20.20 kernel.

MAAS started to set `apparmor=0` to fix https://bugs.launchpad.net/maas/+bug/1677336 due to https://bugs.launchpad.net/maas/+bug/1408106

This seems to no longer affect Lunar (and probably a few releases before that)

I confirm this dirty hack makes MAAS successfully deploy Lunar:

--- a/src/provisioningserver/kernel_opts.py
+++ b/src/provisioningserver/kernel_opts.py
@@ -111,9 +111,6 @@ def compose_purpose_opts(params):
         "cc:{'datasource_list': ['MAAS']}end_cc",
         # Read by cloud-init.
         "cloud-config-url=%s" % params.preseed_url,
- # Disable apparmor in the ephemeral environment. This addresses
- # MAAS bug LP: #1677336 due to LP: #1408106
- "apparmor=0",
     ]
     return kernel_params

We need to make this conditional on the Ubuntu release.

Now about those bugs, it is true that apparmor and overlayfs used to not play along.

Depending on support matrix we can attempt to turn apparmor back on.

Equally it is buggy that Ubuntu kernel does not work with apparmor turned off.

It would be nice to investigate if we can at least enable apparmor for some target series.

Lunar kernel will need SRU to be fixed up.

And separately, we could check if we can get rid of apparmor=0 for all supported releases or not, in next mass release.

The plan is for MAAS to enable apparmor for Lunar onwards. Older releases were always deployed without apparmor and I think changing this would be dangerous, as we don't know how many customers are relying on the current behaviour.

alexsander-souza - if you can make this on per-distro basis that would be great. Indeed empty (thus apparmor=1) should work on jammy and up, but yes we can never know. And having it for lunar onwards would be super nice, because yes overlayfs apparmor things got fixed a while back and are expected to work from now on. And there are more and more things that rely on apparmor to be there.

vmlinuz-6.2.0-18-generic is good, so regression introduced in 6.2.0-19 abi, suspecting new apparmor stack https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2012136

prctl behavior was changed by

c2350a7eca5c UBUNTU: SAUCE: Stacking v38: LSM: Specify which LSM to display

it introduces a short circuit to protect against 2 new lsm prctl commands being invoked without a major lsm, and unfortunately makes the mistake that using lsm_slot == 0 means there are no LSMs present and the default value should be returned.

This however is no longer true as lsm_slot is now only used to track LSMs that need access to secid mappings (major LSMs). Attached is the patch sent to kt to restore behavior

The attachment "0001-UBUNTU-SAUCE-no-up-Stacking-v38-Fix-prctl-syscall-wi.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Status changed to 'Confirmed' because the bug affects multiple users.

My understanding is that nothing needs to be done for systemd. Please re-open if I am mistaken.

This bug is awaiting verification that the linux-hwe-6.2/6.2.0-23.23~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-6.2/6.2.0-1003.3~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux - 6.2.0-23.23

---------------
linux (6.2.0-23.23) lunar; urgency=medium

  * lunar/linux: 6.2.0-23.23 -proposed tracker (LP: #2019845)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/2023.05.15)

  * Fix flicker display problem on some panels which support PSR2 (LP: #2002968)
    - drm/i915/psr: Add continuous full frame bit together with single

  * Kernel 6.1 bumped the disk consumption on default images by 15%
    (LP: #2015867)
    - [Packaging] introduce a separate linux-lib-rust package

  * Update I915 PSR calculation on Linux 6.2 (LP: #2018655)
    - drm/i915: Fix fast wake AUX sync len
    - drm/i915: Explain the magic numbers for AUX SYNC/precharge length

  * Computer with Intel Atom CPU will not boot with Kernel 6.2.0-20
    (LP: #2017444)
    - [Config]: Disable CONFIG_INTEL_ATOMISP

  * udev fails to make prctl() syscall with apparmor=0 (as used by maas by
    default) (LP: #2016908)
    - SAUCE: (no-up) Stacking v38: Fix prctl() syscall with apparmor=0

  * CVE-2023-32233
    - netfilter: nf_tables: deactivate anonymous set from preparation phase

  * CVE-2023-2612
    - SAUCE: shiftfs: prevent lock unbalance in shiftfs_create_object()

  * CVE-2023-31436
    - net: sched: sch_qfq: prevent slab-out-of-bounds in qfq_activate_agg

  * CVE-2023-1380
    - wifi: brcmfmac: slab-out-of-bounds read in brcmf_get_assoc_ies()

  * 5.19 not reporting cgroups v1 blkio.throttle.io_serviced (LP: #2016186)
    - SAUCE: blk-throttle: Fix io statistics for cgroup v1

  * LSM stacking and AppArmor for 6.2: additional fixes (LP: #2017903)
    - SAUCE: (no-up) apparmor: fix policy_compat perms remap for file dfa
    - SAUCE: (no-up) apparmor: fix profile verification and enable it
    - SAUCE: (no-up) apparmor: fix: add missing failure check in
      compute_xmatch_perms
    - SAUCE: (no-up) apparmor: fix: kzalloc perms tables for shared dfas

  * Lunar update: v6.2.12 upstream stable release (LP: #2017219)
    - Revert "pinctrl: amd: Disable and mask interrupts on resume"
    - drm/amd/display: Pass the right info to drm_dp_remove_payload
    - drm/i915: Workaround ICL CSC_MODE sticky arming
    - ALSA: emu10k1: fix capture interrupt handler unlinking
    - ALSA: hda/sigmatel: add pin overrides for Intel DP45SG motherboard
    - ALSA: i2c/cs8427: fix iec958 mixer control deactivation
    - ALSA: hda: patch_realtek: add quirk for Asus N7601ZM
    - ALSA: hda/realtek: Add quirks for Lenovo Z13/Z16 Gen2
    - ALSA: firewire-tascam: add missing unwind goto in
      snd_tscm_stream_start_duplex()
    - ALSA: emu10k1: don't create old pass-through playback device on Audigy
    - ALSA: hda/sigmatel: fix S/PDIF out on Intel D*45* motherboards
    - ALSA: hda/hdmi: disable KAE for Intel DG2
    - Bluetooth: L2CAP: Fix use-after-free in l2cap_disconnect_{req,rsp}
    - Bluetooth: Fix race condition in hidp_session_thread
    - bluetooth: btbcm: Fix logic error in forming the board name.
    - Bluetooth: Free potentially unfreed SCO connection
    - Bluetooth: hci_conn: Fix possible UAF
    - btrfs: restore the thread_pool=...

This bug was fixed in the package linux - 6.3.0-7.7

---------------
linux (6.3.0-7.7) mantic; urgency=medium

  * mantic/linux: 6.3.0-7.7 -proposed tracker (LP: #2023297)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/master)

 -- Paolo Pisati <email address hidden> Thu, 08 Jun 2023 16:44:41 +0200

This bug is awaiting verification that the linux-azure/6.2.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-oem-6.5/6.5.0-1002.2 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-done-jammy-linux-oem-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-oem-6.5' to 'verification-failed-jammy-linux-oem-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-azure-6.5/6.5.0-1007.7~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-done-jammy-linux-azure-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-failed-jammy-linux-azure-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws-6.5/6.5.0-1008.8~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-done-jammy-linux-aws-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-failed-jammy-linux-aws-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-6.5/6.5.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-done-jammy-linux-nvidia-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-6.5' to 'verification-failed-jammy-linux-nvidia-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-lowlatency-hwe-6.5/6.5.0-14.14.1~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.5' to 'verification-done-jammy-linux-lowlatency-hwe-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency-hwe-6.5' to 'verification-failed-jammy-linux-lowlatency-hwe-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!