rcu_sched detected stalls on CPUs/tasks

apport information

efi: Make efi_rts_work accessible to efi page fault handler
https://<email address hidden>/T/#mcbd236122ec0ae61f1ee9c22b8d7910ffc2276a0
addressed a similar issue on x86.

I have a workaround for this:

    UBUNTU: SAUCE: riscv: Disable VMAP_STACK since it fails with efi

    When VMAP_STACK is enabled, kernel threads have their stacks in the vmalloc
    region.

    So when The kworker responsible for handling efi work queue (efi_call_rts) calls
    efi_virtmap_load and then switch_mm, if the stack of the worker is in a vmalloc
    area not yet synchronized with efi_mm (since RISC-V lazily populates vmalloc
    area), an attempt to access this stack will trigger a fault which can't be
    resolved since when trying to save the context, a new trap will be triggered and
    so on.

    So disable VMAP_STACK for now until we figure out the correct fix.

And I'm working on the proper fix which consists in synchronizing the efi page table with the page table of the calling thread before switching to efi mm.

The workaround built successfully in my PPA: https://launchpad.net/~alexghiti/+archive/ubuntu/riscv/+sourcepub/13413205/+listing-archive-extra

I tested it successfully with the RISC-V live installer (where this problem was seen).

I did not have time yet to work on the proper fix so I think we should go for this workaround: @xnox what do you think?

Thanks

VMAP_STACK is new on riscv since v5.14 kernel, meaning we haven't shipped any releases on riscv with VMAP_STACK enabled.

At the time when VMAP_STACK was enabled in the kernel nobody had EFI for riscv yet either.

So it makes sense that this issue is being experienced by us just now, once we got all the pieces landing together.

I would be in favour to disable VMAP_STACK, on riscv64 only, until we have more fixes to make it work better.

This bug was fixed in the package linux-riscv - 5.15.0-1007.7

---------------
linux-riscv (5.15.0-1007.7) jammy; urgency=medium

  * jammy/linux-riscv: 5.15.0-1007.7 -proposed tracker (LP: #1968740)

  * rcu_sched detected stalls on CPUs/tasks (LP: #1967130)
    - [Config] Disable VMAP_STACK due to CPU stalls on EFI

  * Fix unmatched ASMedia ASM2824 PCIe link training (LP: #1964796)
    - PCI: fu740: Force 2.5GT/s for initial device probe

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Apr 2022 15:44:58 +0100

There is more information in LP: #1992458

This bug is awaiting verification that the linux-hwe-5.19/5.19.0-24.25~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This should now be fixed by the upstream commit:
3f105a742725 ("riscv: Sync efi page table's kernel mappings before switching")

This bug is awaiting verification that the linux-azure-6.5/6.5.0-1007.7~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-done-jammy-linux-azure-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure-6.5' to 'verification-failed-jammy-linux-azure-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws-6.5/6.5.0-1008.8~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-done-jammy-linux-aws-6.5'. If the problem still exists, change the tag 'verification-needed-jammy-linux-aws-6.5' to 'verification-failed-jammy-linux-aws-6.5'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!