Raspberry Pi 400 Kernel Panic

Status changed to 'Confirmed' because the bug affects multiple users.

This also affects me running Ubuntu 20.10 MATE on a pi400

I'm new to this. Si, this bug is invalid? What does this mean? I'm affected. At the raspberry pi forum others too, so was my report a waste of time? It took 1 day for me to try reporting this as correct as possible.

So, what kind of support do I need? I just installed ubuntu as described, on an SSD. I think this was my first and my lasr bug report. I think that this procedure us a mess. Bye!

Sorry, muscle memory kicked in. This problem is confirmed and on my list of things to look at.

Quick note on a workaround: I've found on my pi400 that reverting to kernel 1007 works around the issue. If you've still got 1007 installed (a quick "ls /boot" should confirm that if you see lots of *-1007-raspi files lying around) you can temporarily switch to it like so:

$ sudo flash-kernel --force 5.8.0-1007-raspi

This will force flash-kernel to copy the older kernel (and dtbs etc.) to the boot partition instead of the latest (obviously you need to reboot afterward to use the 1007 kernel). flash-kernel will warn in its output that installing another kernel will just overwrite this, but that's *probably* what you want anyway (assuming the next upgrade fixes this).

Is this what you are seeing?

[ 77.591737] systemd-shutdown[1]: All filesystems, swaps, loop devices and DM devices detached.
[ 77.631493] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 77.638423] systemd-shutdown[1]: Powering off.
[ 77.656068] kvm: exiting hardware virtualization
[ 77.724718] bcmgenet fd580000.ethernet eth0: Link is Down
[ 77.748495] reboot: Power down
[ 100.222654] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 100.228912] rcu: 1-O..!: (1 GPs behind) idle=dd6/1/0x4000000000000002 softirq=17079/17079 fqs=0
[ 100.237953] rcu: 3-O..!: (8 GPs behind) idle=c7a/0/0x1 softirq=15956/15956 fqs=0
[ 100.245665] (detected by 0, t=5620 jiffies, g=14373, q=6)
[ 100.251249] Task dump for CPU 1:
[ 100.254533] rcu_preempt R running task 0 10 2 0x00000028
[ 100.261716] Call trace:
[ 100.264213] __switch_to+0xfc/0x12c
[ 100.267766] rcu_state+0x258/0x400
[ 100.271225] Task dump for CPU 3:
[ 100.274506] swapper/3 R running task 0 0 1 0x00000028
[ 100.281688] Call trace:
[ 100.284176] __switch_to+0xfc/0x12c
[ 100.287727] 0xffff8000100a3f10
[ 100.290924] rcu: rcu_preempt kthread starved for 5620 jiffies! g14373 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
[ 100.301378] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[ 100.310680] rcu: RCU grace-period kthread stack dump:
[ 100.315821] rcu_preempt R running task 0 10 2 0x00000028
[ 100.323001] Call trace:
[ 100.325489] __switch_to+0xfc/0x12c
[ 100.329038] rcu_state+0x258/0x400
[ 164.350653] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 164.356864] rcu: 1-O..!: (1 GPs behind) idle=dd6/1/0x4000000000000002 softirq=17079/17079 fqs=0
[ 164.365904] rcu: 3-O..!: (8 GPs behind) idle=c7a/0/0x1 softirq=15956/15956 fqs=0
[ 164.373615] (detected by 0, t=21652 jiffies, g=14373, q=6)
[ 164.379288] Task dump for CPU 1:
[ 164.382570] rcu_preempt R running task 0 10 2 0x00000028
[ 164.389750] Call trace:
[ 164.392237] __switch_to+0xfc/0x12c
[ 164.395788] rcu_state+0x258/0x400
[ 164.399247] Task dump for CPU 3:
[ 164.402530] swapper/3 R running task 0 0 1 0x00000028
[ 164.409710] Call trace:
[ 164.412197] __switch_to+0xfc/0x12c
[ 164.415745] 0xffff8000100a3f10
[ 164.418941] rcu: rcu_preempt kthread starved for 21652 jiffies! g14373 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1
[ 164.429483] rcu: Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
[ 164.438785] rcu: RCU grace-period kthread stack dump:
[ 164.443925] rcu_preempt R running task 0 10 2 0x00000028
[ 164.451104] Call trace:
[ 164.453592] __switch_to+0xfc/0x12c
[ 164.457140] rcu_state+0x258/0x400
[ 228.478652] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 228.484863] rcu: 1-O..!: (1 GPs behind) idle=dd6/1/0x4000000000000002 softirq=17079/17079 fqs=0
[ 228.493904] rcu: 3-O..!: (8 GPs behind) idle=c7a/0/0x1 softirq=15956/15956 fqs=0
[ 228.502965] (detected by 0, t=37684 jiffies, g=14373, q=6)
[ 228.509989] Task dump for CPU 1:
[ 228.51...

@Juergh, no. In your dump the wording "Kernel Panic" never appears.

A pity... I cannot log this. If I could, I'd able to do include a copy of it.

@waveform, I'll try it.

@waveform: I'll get an error-message:

sudo flash-kernel --force 5.8.0-1007-raspi
Using DTB: bcm2711-rpi-4-b.dtb
Couldn't find DTB bcm2711-rpi-4-b.dtb on the following paths: /etc/flash-kernel/dtbs /usr/lib/linux-image-5.8.0-1007-raspi /lib/firmware/5.8.0-1007-raspi/device-tree/
Installing into /boot/dtbs/5.8.0-1007-raspi/./bcm2711-rpi-4-b.dtb

Maybe this kernel was never installed before(?).

So, I took 1007 from the repository and repeated the force-command suggested by @waveform. Thx for the workaround, yes, this Kernel does not lead to a Panic. Thx.

@Juerg, the output is starting with the "shutting down" line, and after about 10 minutes, the non blinking cursor is replaced by a few of your lines which are shown up above. Also with "Kernel Panic". I don't know how to log this.

I did some photos of it, but the content is not readable, a pity.. For the moment, I try to avoid 5.8.0-1008-raspi. I crashed lots of sd-cards in past, because of switching on/off with power-supply-only.

But, I'm ready to support. If I have a way to log everything till after kernel panic, I can submit further logs/files.

@juergh I've seen both the "bare" reboot: Power down followed by a long-ish delay and a kernel panic, and the repeated rcu_preempt messages that you posted in comment 10 under 1008. Neither appear to occur in 1007 so I'm guessing they might both be symptoms of the same issue? I'll see if I can hook up a serial console to a 400 this weekend and see if I can goad it into producing the kernel panic to get some legible output.

My rough observations are the rcu_preempt error occurs most of the time (I haven't seen the "bare" reboot: Power down error in a while), but I've definitely seen both under 1008 with the former the more common one.

Maybe this: https://github.com/raspberrypi/linux/issues/3941
I'll produce a test kernel without the mentioned commit.

Can you give this a try? https://kernel.ubuntu.com/~juergh/lp1907432/
Seems to do it for me, meaning the Pi 400 powers fully off.

Yes, I'll try.

I just download all files and then I install it with the package manager?

Hmmm... Software Center is saying, that none of the files is compatible. Could you please tell me what I can do?

I think that I have to do this:
dpkg -i linux-*.deb.

Installation done.
First test: shutdown. success.
Second test: shutdown, after changing some settings. success.

WOOOHOOO!!! It seems to work! Yes, it shuts down immediately!

👏👏👏👏👏👏👏 💗💗💗💗💗

This bug was fixed in the package linux-raspi - 5.8.0-1013.16

---------------
linux-raspi (5.8.0-1013.16) groovy; urgency=medium

  * groovy/linux-raspi: 5.8.0-1013.16 -proposed tracker (LP: #1911336)

  * rpi3b+ wifi doesn't get used if ethernet disconnected (LP: #1890487)
    - net: lan78xx: Ack pending PHY ints when resetting

  * groovy/linux-raspi: Upstream raspberrypi patchset 2020-11-05 (LP: #1903541)
    - brcmfmac: Increase power saving delay to 2s
    - overlays: Update i2c0 overlay to disable the i2c0mux.
    - dt: Remove duplicate assignment for i2c0 pinctrl config
    - overlays: Add option for composite to vc4-kms-v3d-pi4.
    - overlays: minor typo in instructions
    - overlays: Regenerate upstream-pi4 overlay
    - overlays: Add parameters to adafruit18, sainsmart18
    - rpivid_h265: Fix width/height typo
    - overlays: Add extra CMA sizes (up to 512M)
    - overlays: Add note to BCM2711 overlays
    - configs: Remove I2C_BRCMSTB from the kernel
    - net: bcmgenet: Reset RBUF on first open
    - ASoC: cs42xx8: Only define cs42xx8_of_match once
    - configs: Regenerate the defconfigs
    - staging: bcm2835-codec: Use a define the completion timeout
    - staging: bcm2835-codec: Correct buffer number change on start streaming
    - USB: gadget: f_hid: avoid crashes and log spam
    - Update hy28b-overlay.dts
    - overlays: Update display GPIO declarations
    - configs: Add CONFIG_RTC_DRV_SD3078=m
    - overlays: Add sd3078 to the i2c-rtc overlay
    - configs: Restore SND_PCM_OSS=m
    - dwc_otg: initialise sched_frame for periodic QHs that were parked
    - arm64: configs: Enable Unicam support
    - arm64: configs: Enable V4L2 test module support
    - staging: bcm2835-camera: Replace deprecated V4L2_PIX_FMT_BGR32
    - staging: bcm2835-codec: Replace deprecated V4L2_PIX_FMT_BGR32
    - ARM: bcm2711-rpi.dts: Unlock DMA channels 9 & 10
    - gpio: Add gpio-fsm driver
    - configs: Add CONFIG_GPIO_FSM=m
    - overlays: Add fsm-demo overlay
    - overlays: Add ghost-amp overlay
    - Fixes a problem when module probes before i2c module is available
    - [Config] raspi: updateconfigs after update to rpi-5.8.y 2020-11-05

  * Raspberry Pi 400 Kernel Panic (LP: #1907432)
    - SAUCE: Revert "mailbox: avoid timer start from callback"

  [ Ubuntu: 5.8.0-39.44 ]

  * groovy/linux: 5.8.0-39.44 -proposed tracker (LP: #1911350)
  * overlay: permission regression in 5.4.0-51.56 due to patches related to
    CVE-2020-16120 (LP: #1900141)
    - ovl: do not fail because of O_NOATIME

  [ Ubuntu: 5.8.0-38.43 ]

  * groovy/linux: 5.8.0-38.43 -proposed tracker (LP: #1911143)
  * CVE-2020-28374
    - SAUCE: target: fix XCOPY NAA identifier lookup
  * Packaging resync (LP: #1786013)
    - update dkms package versions

  [ Ubuntu: 5.8.0-36.40 ]

  * debian/scripts/file-downloader does not handle positive failures correctly
    (LP: #1878897)
    - [Packaging] file-downloader not handling positive failures correctly

  [ Ubuntu: 5.8.0-35.39 ]

  * Packaging resync (LP: #1786013)
    - update dkms package versions
  * CVE-2021-1052 // CVE-2021-1053
    - [Packaging] NVIDIA -- Add the NVIDIA 460 driver

 -- Ian May <email address hidden> W...

This bug was fixed in the package linux-raspi - 5.11.0-1003.3

---------------
linux-raspi (5.11.0-1003.3) hirsute; urgency=medium

  * hirsute/linux-raspi: 5.11.0-1003.3 -proposed tracker (LP: #1917734)

  * Packaging resync (LP: #1786013)
    - [Packaging] update variants
    - [Packaging] update update.conf

  * Miscellaneous Ubuntu changes
    - [Packaging] raspi: Initial version of linux-raspi for Hirsute
    - [Config] raspi: updateconfigs after rebase to Ubuntu-5.11.0-11.12
    - [Packaging] raspi: Set disable_d_i=true in hooks.mk
    - [Packaging] raspi: Add dctrl-tools to Build-Depends
    - [Packaging] raspi: Correctly implement noudeb build profiles
    - [Packaging] raspi: remove Provides: aufs-dkms

  [ Ubuntu: 5.11.0-11.12 ]

  * hirsute/linux: 5.11.0-11.12 -proposed tracker (LP: #1917335)
  * Packaging resync (LP: #1786013)
    - update dkms package versions
    - [Packaging] update variants
  * Support no udeb profile (LP: #1916095)
    - [Packaging] replace custom filter script with dctrl-tools
    - [Packaging] correctly implement noudeb build profiles.
  * Miscellaneous Ubuntu changes
    - [Packaging] dkms-versions -- remove nvidia-graphics-drivers-440-server
    - [Debian] run ubuntu-regression-suite for linux-unstable
    - [Packaging] remove Provides: aufs-dkms
    - [Packaging] Change source package name to linux
    - [Config] update gcc version in config due to toolchain update
  * Miscellaneous upstream changes
    - Revert "UBUNTU: [Config] disable nvidia and nvidia_server builds"
    - Xen/x86: don't bail early from clear_foreign_p2m_mapping()
    - Xen/x86: also check kernel mapping in set_foreign_p2m_mapping()
    - Xen/gntdev: correct dev_bus_addr handling in gntdev_map_grant_pages()
    - Xen/gntdev: correct error checking in gntdev_map_grant_pages()
    - xen/arm: don't ignore return errors from set_phys_to_machine
    - xen-blkback: don't "handle" error by BUG()
    - xen-netback: don't "handle" error by BUG()
    - xen-scsiback: don't "handle" error by BUG()
    - xen-blkback: fix error handling in xen_blkbk_map()
    - tty: protect tty_write from odd low-level tty disciplines
    - Bluetooth: btusb: Always fallback to alt 1 for WBS
    - media: pwc: Use correct device for DMA
    - bpf: Fix truncation handling for mod32 dst reg wrt zero
    - HID: make arrays usage and value to be the same
    - USB: quirks: sort quirk entries
    - usb: quirks: add quirk to start video capture on ELMO L-12F document camera
      reliable
    - ntfs: check for valid standard information attribute
    - Bluetooth: btusb: Some Qualcomm Bluetooth adapters stop working
    - arm64: tegra: Add power-domain for Tegra210 HDA
    - hwmon: (dell-smm) Add XPS 15 L502X to fan control blacklist
    - KVM: x86: Zap the oldest MMU pages, not the newest
    - KVM: do not assume PTE is writable after follow_pfn
    - mm: provide a saner PTE walking API for modules
    - KVM: Use kvm_pfn_t for local PFN variable in hva_to_pfn_remapped()

 -- Juerg Haefliger <email address hidden> Thu, 04 Mar 2021 14:42:19 +0100