ubuntu-kernel-tests

ubuntu_boot test failures on b/linux-gcp-5.4 on c3a-standard instance

Bug #2051184 reported by Nathan Sweetman
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ubuntu-kernel-tests
New
Undecided
Unassigned
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Confirmed
Undecided
Unassigned

Bug Description

Boot testing fails for b/gcp-5.4 on new c3a-standard instance as below:

Causes WARNING at: arch/arm64/kernel/cpu_errata.c:1387 spectre_bhb_enable_mitigation+0x148

[ 0.060607] WARNING: CPU: 3 PID: 29 at /build/linux-gcp-5.4-FHHMzo/linux-gcp-5.4-5.4.0/arch/arm64/kernel/cpu_errata.c:1387 spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060612] WARNING: CPU: 2 PID: 23 at /build/linux-gcp-5.4-FHHMzo/linux-gcp-5.4-5.4.0/arch/arm64/kernel/cpu_errata.c:1387 spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060613] Modules linked in:
[ 0.060615] Modules linked in:
[ 0.060621] CPU: 3 PID: 29 Comm: migration/3 Not tainted 5.4.0-1122-gcp #131~18.04.1-Ubuntu
[ 0.060622] CPU: 2 PID: 23 Comm: migration/2 Not tainted 5.4.0-1122-gcp #131~18.04.1-Ubuntu
[ 0.060628] WARNING: CPU: 1 PID: 17 at /build/linux-gcp-5.4-FHHMzo/linux-gcp-5.4-5.4.0/arch/arm64/kernel/cpu_errata.c:1387 spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060629] pstate: 20400085 (nzCv daIf +PAN -UAO)
[ 0.060630] Modules linked in:
[ 0.060632] pc : spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060635] lr : spectre_bhb_enable_mitigation+0x120/0x228
[ 0.060636] CPU: 1 PID: 17 Comm: migration/1 Not tainted 5.4.0-1122-gcp #131~18.04.1-Ubuntu
[ 0.060637] sp : ffff80001012bce0
[ 0.060639] pstate: 20400085 (nzCv daIf +PAN -UAO)
[ 0.060639] x29: ffff80001012bce0 x28: 0000000000000002
[ 0.060642] pc : spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060645] lr : spectre_bhb_enable_mitigation+0x120/0x228
[ 0.060646] x27: ffff80001003bd10
[ 0.060647] sp : ffff8000100fbce0
[ 0.060647] x26: 0000000000000000
[ 0.060648] x29: ffff8000100fbce0
[ 0.060650] x28: 0000000000000002
[ 0.060652] x25: 0000000000000060
[ 0.060653] x24: 0000000000000000
[ 0.060655] x27: ffff80001003bd10 x26: 0000000000000000
[ 0.060657] x23: 0000000000000001
[ 0.060661] x22: ffffdca01ea9a3c0
[ 0.060663] x25: 0000000000000060 x24: 0000000000000000
[ 0.060665] x21: ffffdca01dda4970
[ 0.060668] x20: 0000000000000001
[ 0.060669] x23: 0000000000000001 x22: ffffdca01ea9a3c0
[ 0.060671] x19: ffffdca01ecca000
[ 0.060675] x18: ffffdca01dc56a10
[ 0.060680] x21: ffffdca01dda4970 x20: 0000000000000001
[ 0.060684] x17: 00000000bc8dcec3
[ 0.060687] x16: 00000000b1e1b9b5
[ 0.060688] x19: ffffdca01ecca000 x18: ffffdca01dc56a10
[ 0.060694] x17: 000000007e50fa5f
[ 0.060694] x15: 0000000000000001
[ 0.060695] x16: 000000009cd7320e
[ 0.060696] x14: ffff74e6bd9c01c8
[ 0.060699] x13: 0000000000000004
[ 0.060701] x15: 0000000000000001
[ 0.060702] x12: 0000000000000228
[ 0.060703] x14: ffff74e6bd99f1c8
[ 0.060705] x11: 0000000000000000
[ 0.060711] x13: 0000000000000004
[ 0.060712] x10: 0000000000000b00
[ 0.060713] x12: 0000000000000228
[ 0.060716] x11: 0000000000000000
[ 0.060722] x9 : ffff80001012bd20
[ 0.060723] x10: 0000000000000b00
[ 0.060724] x8 : ffff74e6ba3c66e0
[ 0.060726] x7 : 0000000000000000 x6 : ffffdca01d822000
[ 0.060727] x9 : ffff8000100fbd20
[ 0.060729] x8 : ffff74e6ba33a9e0
[ 0.060729] x5 : 00000000c00fac30
[ 0.060730] x4 : 0000000000000000
[ 0.060731] x7 : 0000000000000000
[ 0.060734] x6 : ffffdca01d822000
[ 0.060735] x3 : 000000000000000b
[ 0.060736] x2 : ffffdca01ecca508
[ 0.060738] x5 : 00000000c00fac30 x4 : 0000000000000000
[ 0.060740] x1 : 000000000000000b
[ 0.060742] x0 : 000000000000000b
[ 0.060746] x3 : 000000000000000b x2 : ffffdca01ecca508
[ 0.060749] Call trace:
[ 0.060751] x1 : 000000000000000b
[ 0.060753] spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060755] x0 : 000000000000000b
[ 0.060757] cpu_enable_non_boot_scope_capabilities+0x6c/0xa8
[ 0.060761] multi_cpu_stop+0xac/0x188
[ 0.060762] Call trace:
[ 0.060763] cpu_stopper_thread+0xe4/0x1a0
[ 0.060766] spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060769] smpboot_thread_fn+0x1a8/0x1d8
[ 0.060771] cpu_enable_non_boot_scope_capabilities+0x6c/0xa8
[ 0.060773] kthread+0x114/0x118
[ 0.060775] multi_cpu_stop+0xac/0x188
[ 0.060778] ret_from_fork+0x10/0x18
[ 0.060779] cpu_stopper_thread+0xe4/0x1a0
[ 0.060780] ---[ end trace 82b1638f0eea221b ]---
[ 0.060783] smpboot_thread_fn+0x1a8/0x1d8
[ 0.060785] kthread+0x114/0x118
[ 0.060787] ret_from_fork+0x10/0x18
[ 0.060789] ---[ end trace 82b1638f0eea221c ]---
[ 0.060797] pstate: 20400085 (nzCv daIf +PAN -UAO)
[ 0.060799] pc : spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060800] lr : spectre_bhb_enable_mitigation+0x120/0x228
[ 0.060801] sp : ffff80001015bce0
[ 0.060802] x29: ffff80001015bce0 x28: 0000000000000002
[ 0.060804] x27: ffff80001003bd10 x26: 0000000000000000
[ 0.060805] x25: 0000000000000060 x24: 0000000000000000
[ 0.060806] x23: 0000000000000001 x22: ffffdca01ea9a3c0
[ 0.060807] x21: ffffdca01dda4970 x20: 0000000000000001
[ 0.060808] x19: ffffdca01ecca000 x18: ffffdca01dc56a10
[ 0.060809] x17: 000000002041ff62 x16: 0000000057377f7c
[ 0.060811] x15: 0000000000000001 x14: ffff74e6bd9e11c8
[ 0.060812] x13: 0000000000000004 x12: 0000000000000228
[ 0.060813] x11: 0000000000000000 x10: 0000000000000000
[ 0.060814] x9 : 0000000000000000 x8 : 0000000000000000
[ 0.060815] x7 : ffff74e6bd9f5bc0 x6 : ffffdca01d822000
[ 0.060816] x5 : 00000000c00fac30 x4 : 0000000000000000
[ 0.060817] x3 : 000000000000000b x2 : ffffdca01ecca508
[ 0.060818] x1 : 000000000000000b x0 : 000000000000000b
[ 0.060820] Call trace:
[ 0.060822] spectre_bhb_enable_mitigation+0x148/0x228
[ 0.060823] cpu_enable_non_boot_scope_capabilities+0x6c/0xa8
[ 0.060826] multi_cpu_stop+0xac/0x188
[ 0.060827] cpu_stopper_thread+0xe4/0x1a0
[ 0.060829] smpboot_thread_fn+0x1a8/0x1d8
[ 0.060830] kthread+0x114/0x118
[ 0.060832] ret_from_fork+0x10/0x18
[ 0.060833] ---[ end trace 82b1638f0eea221d ]---

See original description
Nathan Sweetman (nathan-sweetman)
description: updated
Po-Hsu Lin (cypressyew)
tags: added: 5.4 bionic gcp sru-20240108 ubuntu-boot
Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: New → Fix Released
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Here is my investigation when discussing this issue with Jesse:

This warning is triggered in spectre_bhb_enable_mitigation() from arch/arm64/kernel/cpu_errata.c, inside this if statement:
    } else if (spectre_bhb_loop_affected(SCOPE_LOCAL_CPU)) {
            switch (spectre_bhb_loop_affected(SCOPE_SYSTEM)) {
            case 8:
                    /*
                     * A57/A72-r0 will already have selected the
                     * spectre-indirect vector, which is sufficient
                     * for BHB too.
                     */
                    if (!__this_cpu_read(bp_hardening_data.fn))
                            kvm_setup_bhb_slot(__spectre_bhb_loop_k8_start);
                    break;
            case 24:
                    kvm_setup_bhb_slot(__spectre_bhb_loop_k24_start);
                    break;
            case 32:
                    kvm_setup_bhb_slot(__spectre_bhb_loop_k32_start);
                    break;
            default:
                    WARN_ON_ONCE(1);
            }
            this_cpu_set_vectors(EL1_VECTOR_BHB_LOOP);

This is because the case value (spectre_bhb_loop_affected(SCOPE_SYSTEM)) does not match, and falls into the "default" case.

This piece of code came from a SAUCE patch 14d45ef5 (UBUNTU: SAUCE: arm64: Mitigate spectre style branch history side channels), in which it moves the code that exist in arch/arm64/kernel/proton-pack.c to cpu_errata.c as mentioned in the commit message:
    [ modified for stable, moved code to cpu_errata.c removed bitmap of
      mitigations, use kvm template infrastructure ]

This issue does not exist in 5.15 kernel, and there is a new commit that added a new "case 11" to the switch-case, 0e5d5ae8 (arm64: Add AMPERE1 to the Spectre-BHB affected list).

According to Google [1]:
> C3A powered by AmpereOne processors
 So I think this 0e5d5ae8 is very likely the fix we need for 5.4.

[1] https://cloud.google.com/blog/products/compute/whats-new-in-googles-workload-optimized-computing

tags: added: arm64
Revision history for this message
Manuel Diewald (diewald) wrote :

From LP: #2052632

log_check/kernel_tainted tests will fail on AmpereOne based machines like c3a-standard-4 for focal:linux 5.4.0-168.186 and newer.

The warning was introduced with upstream commit

f41cab7a4653 arm64: Add AMPERE1 to the Spectre-BHB affected list

that came with upstream stable release v5.4.253 as part of Ubuntu-5.4.0-168.186: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2038652

Looking at the code, this warning will always be issued on AMPERE1 based machines but Spectre-BHB mitigation should be guaranteed regardless. The warning is issued because there is no case for loop iteration count 11 necessary for AMPERE1 CPUs in spectre_bhb_enable_mitigation().

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu Focal):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.