Ubuntu
linux package

[Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure Encrypted Virtualization(SEV) is enabled

Bug #1956575 reported by Louis Bouchard on 2022-01-06

This bug affects 3 people

	Status	Importance	Assigned to
linux (Ubuntu)	Confirmed	Undecided	Unassigned
Xenial	Invalid	Undecided	Unassigned
Bionic	Confirmed	High	Unassigned
Focal	Fix Released	High	Unassigned
Hirsute	Invalid	Undecided	Unassigned

Bug Description

[Impact]

The latest Focal kernel (linux-image-5.4.0-92-generic) fails to boot when SEV is enabled.

The kernel panics with the following backtrace :

[ 1.531125] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.531760] EFI Variables Facility v0.08 2004-May-17
[ 1.532575] general protection fault: 0000 [#1] SMP NOPTI
[ 1.533116] CPU: 11 PID: 1 Comm: swapper/0 Not tainted 5.4.157-debug6 #15
[ 1.533788] Hardware name: Scaleway SCW-ENT1-L, BIOS 0.0.0 02/06/2015
[ 1.534429] RIP: 0010:efi_mokvar_sysfs_init+0x9d/0x184
[ 1.534949] Code: 00 48 85 c0 0f 85 b3 00 00 00 48 c7 c7 b0 db e2 a1 41 bd f4 ff ff ff e8 98 90 16 ff e9 e6 00 00 00 48 85 d2 0f 85 a5 00 00 00 <80> 3b 00 0f 84 b5 00 00 00 48 85 db 0f 84 ac 00 00 00 48 8b 3d 85
[ 1.535120] RSP: 0018:ffffb96e4001bdf8 EFLAGS: 00010202
[ 1.535120] RAX: b6262a23e510e179 RBX: b625e392251db281 RCX: 0000000000000000
[ 1.535120] RDX: ffffb96e400cd000 RSI: ffff978437e5dc38 RDI: ffffffffa2121860
[ 1.535120] RBP: ffffb96e4001be10 R08: 0000000000000000 R09: 0000000000000228
[ 1.538947] R10: 0000000000000001 R11: 0000000000000000 R12: ffff978437e555a0
[ 1.538947] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.538947] FS: 0000000000000000(0000) GS:ffff97843f6c0000(0000) knlGS:0000000000000000
[ 1.538947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.538947] CR2: 00000000ffffffff CR3: 00080011a060a001 CR4: 0000000000360ee0
[ 1.538947] Call Trace:
[ 1.538947] ? efi_rci2_sysfs_init+0x29a/0x29a
[ 1.538947] do_one_initcall+0x4a/0x200
[ 1.538947] kernel_init_freeable+0x1c0/0x263
[ 1.538947] ? rest_init+0xb0/0xb0
[ 1.538947] kernel_init+0xe/0x110
[ 1.538947] ret_from_fork+0x22/0x40
[ 1.538947] Modules linked in:
[ 1.545871] ---[ end trace 815dc8177e65da02 ]---
[ 1.546328] RIP: 0010:efi_mokvar_sysfs_init+0x9d/0x184
[ 1.546872] Code: 00 48 85 c0 0f 85 b3 00 00 00 48 c7 c7 b0 db e2 a1 41 bd f4 ff ff ff e8 98 90 16 ff e9 e6 00 00 00 48 85 d2 0f 85 a5 00 00 00 <80> 3b 00 0f 84 b5 00 00 00 48 85 db 0f 84 ac 00 00 00 48 8b 3d 85
[ 1.548703] RSP: 0018:ffffb96e4001bdf8 EFLAGS: 00010202
[ 1.549218] RAX: b6262a23e510e179 RBX: b625e392251db281 RCX: 0000000000000000
[ 1.549916] RDX: ffffb96e400cd000 RSI: ffff978437e5dc38 RDI: ffffffffa2121860
[ 1.550617] RBP: ffffb96e4001be10 R08: 0000000000000000 R09: 0000000000000228
[ 1.551337] R10: 0000000000000001 R11: 0000000000000000 R12: ffff978437e555a0
[ 1.552036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.552737] FS: 0000000000000000(0000) GS:ffff97843f6c0000(0000) knlGS:0000000000000000
[ 1.553529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.554093] CR2: 00000000ffffffff CR3: 00080011a060a001 CR4: 0000000000360ee0
[ 1.554818] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 1.555335] Kernel Offset: 0x1fa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1.555335] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---
-

The previous kernel (linux-image-5.4.0-91-generic) boots correctly with SEV.

Bisection of the kernels b/w 5.4.0-91 and 5.4.0-92 identified the following commit as the source of regresssion :

# git bisect good
7ca05228f713c24eb55574b36e32d9b54c5a1b76 is the first bad commit
commit 7ca05228f713c24eb55574b36e32d9b54c5a1b76
Author: Lenny Szubowicz <email address hidden>
Date: Fri Sep 4 21:31:05 2020 -0400

efi: Support for MOK variable config table

BugLink: https://bugs.launchpad.net/bugs/1928679

...

Since the panic happens in very early stages of boot, no trace of the panic is present in the log files and apparently it is not possible to collect a kernel crash dump, the crashkernel has not been enabled yet.

[Test case]

The kernel need to boot properly with SEV enabled.

[Potential regression]

The fix is very specific and restricted and should cause regressions, however it touches ioremap that is widely used.

See original description

Tags:

Revision history for this message

Louis Bouchard (louis) wrote on 2022-01-06 (last edit on 2022-01-06):

Details of the bisected commit :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/commit/?id=7ca05228f713c24eb55574b36e32d9b54c5a1b76

commit 7ca05228f713c24eb55574b36e32d9b54c5a1b76
Author: Lenny Szubowicz <email address hidden>
Date: Fri Sep 4 21:31:05 2020 -0400

efi: Support for MOK variable config table

BugLink: https://bugs.launchpad.net/bugs/1928679

    Because of system-specific EFI firmware limitations, EFI volatile
    variables may not be capable of holding the required contents of
    the Machine Owner Key (MOK) certificate store when the certificate
    list grows above some size. Therefore, an EFI boot loader may pass
    the MOK certs via a EFI configuration table created specifically for
    this purpose to avoid this firmware limitation.

    An EFI configuration table is a much more primitive mechanism
    compared to EFI variables and is well suited for one-way passage
    of static information from a pre-OS environment to the kernel.

    This patch adds initial kernel support to recognize, parse,
    and validate the EFI MOK configuration table, where named
    entries contain the same data that would otherwise be provided
    in similarly named EFI variables.

    Additionally, this patch creates a sysfs binary file for each
    EFI MOK configuration table entry found. These files are read-only
    to root and are provided for use by user space utilities such as
    mokutil.

A subsequent patch will load MOK certs into the trusted platform
key ring using this infrastructure.

    Signed-off-by: Lenny Szubowicz <email address hidden>
    Link: https://<email address hidden>
    Signed-off-by: Ard Biesheuvel <email address hidden>
    (cherry picked from commit 58c909022a5a56cd1d9e89c8c5461fd1f6a27bb5)
    Signed-off-by: Dimitri John Ledkov <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Acked-by: Tim Gardner <email address hidden>
    Signed-off-by: Stefan Bader <email address hidden>

arch/x86/kernel/setup.c | 1 +
arch/x86/platform/efi/efi.c | 3 +
drivers/firmware/efi/Makefile | 1 +
drivers/firmware/efi/arm-init.c | 1 +
drivers/firmware/efi/efi.c | 6 +
drivers/firmware/efi/mokvar-table.c | 360 ++++++++++++++++++++++++++++++++++++
include/linux/efi.h | 34 ++++
7 files changed, 406 insertions(+)
create mode 100644 drivers/firmware/efi/mokvar-table.c

Details of the bisected commit :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/commit/?id=7ca05228f713c24eb55574b36e32d9b54c5a1b76

commit 7ca05228f713c24eb55574b36e32d9b54c5a1b76
Author: Lenny Szubowicz <lszubowi@redhat.com>
Date:   Fri Sep 4 21:31:05 2020 -0400

efi: Support for MOK variable config table
    
    BugLink: https://bugs.launchpad.net/bugs/1928679
    
    Because of system-specific EFI firmware limitations, EFI volatile
    variables may not be capable of holding the required contents of
    the Machine Owner Key (MOK) certificate store when the certificate
    list grows above some size. Therefore, an EFI boot loader may pass
    the MOK certs via a EFI configuration table created specifically for
    this purpose to avoid this firmware limitation.
    
    An EFI configuration table is a much more primitive mechanism
    compared to EFI variables and is well suited for one-way passage
    of static information from a pre-OS environment to the kernel.
    
    This patch adds initial kernel support to recognize, parse,
    and validate the EFI MOK configuration table, where named
    entries contain the same data that would otherwise be provided
    in similarly named EFI variables.
    
    Additionally, this patch creates a sysfs binary file for each
    EFI MOK configuration table entry found. These files are read-only
    to root and are provided for use by user space utilities such as
    mokutil.
    
    A subsequent patch will load MOK certs into the trusted platform
    key ring using this infrastructure.
    
    Signed-off-by: Lenny Szubowicz <lszubowi@redhat.com>
    Link: https://lore.kernel.org/r/20200905013107.10457-2-lszubowi@redhat.com
    Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
    (cherry picked from commit 58c909022a5a56cd1d9e89c8c5461fd1f6a27bb5)
    Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com>
    Acked-by: Stefan Bader <stefan.bader@canonical.com>
    Acked-by: Tim Gardner <tim.gardner@canonical.com>
    Signed-off-by: Stefan Bader <stefan.bader@canonical.com>

arch/x86/kernel/setup.c             |   1 +
 arch/x86/platform/efi/efi.c         |   3 +
 drivers/firmware/efi/Makefile       |   1 +
 drivers/firmware/efi/arm-init.c     |   1 +
 drivers/firmware/efi/efi.c          |   6 +
 drivers/firmware/efi/mokvar-table.c | 360 ++++++++++++++++++++++++++++++++++++
 include/linux/efi.h                 |  34 ++++
 7 files changed, 406 insertions(+)
 create mode 100644 drivers/firmware/efi/mokvar-table.c

Changed in linux (Ubuntu Focal):
status:	New → Confirmed
Changed in linux (Ubuntu):
status:	New → Confirmed
summary:	[Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure - Encrypted Virtualization(SEV) is enagbled + Encrypted Virtualization(SEV) is enabled

Louis Bouchard (louis) on 2022-01-06

description:

updated

Revision history for this message

Louis Bouchard (louis) wrote on 2022-01-06:

According to the following advisory[1] from HP, this is a known issue which is fixed upstream :

"Advisory: Red Hat Enterprise Linux 8.3 - Guest Stops Responding at efi_mokvar_sysfs_init+0xa9/0x19d with AMD Secure Encrypted Virtualization (SEV) Enabled"

The upstream fix cited in the article is the following :

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b

x86/ioremap: Map EFI-reserved memory as encrypted for SEV

Adding this commit on top of kernel 5.4.0-92.103 does fix the booting issue.

[1] https://support.hpe.com/hpesc/public/docDisplay?docId=a00119071en_us&docLocale=en_US

Revision history for this message

Marcelo Cerri (mhcerri) wrote on 2022-01-06:

The 5.11 hirsute kernel already contains the fix, so I'm marking it as invalid:

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/hirsute/commit/?id=929ece70ad54afc1ca090ea1914b79def67137c1

description:	updated
Changed in linux (Ubuntu Focal):
status:	Confirmed → In Progress
Changed in linux (Ubuntu Hirsute):
status:	New → Invalid
Changed in linux (Ubuntu Focal):
importance:	Undecided → High

Revision history for this message

Marcelo Cerri (mhcerri) wrote on 2022-01-06:

I also prepared a bionic/linux-gcp-5.4 test kernel with the fix and I confirmed it also fixed the same boot problem in confidential computing instances.

Revision history for this message

Khaled El Mously (kmously) wrote on 2022-01-06:

Hello @Louis,

Are you able to confirm whether the 4.15 kernels are affected by this problem?

Revision history for this message

Marcelo Cerri (mhcerri) wrote on 2022-01-06:

4.4 doesn't have the offending commit so marking it as invalid.

Changed in linux (Ubuntu Xenial):
status:	New → Invalid

Revision history for this message

Louis Bouchard (louis) wrote on 2022-01-07:

Hello @kmously, yes I am able to reproduce the same panic on 4.15.0-166.

Kleber Sacilotto de Souza (kleber-souza) on 2022-01-07

Changed in linux (Ubuntu Bionic):
status:	New → Confirmed
importance:	Undecided → High

Kleber Sacilotto de Souza (kleber-souza) on 2022-01-07

Changed in linux (Ubuntu Focal):
status:	In Progress → Fix Committed

Revision history for this message

Louis Bouchard (louis) wrote on 2022-01-07:

Download full text (4.9 KiB)

Here is the result of a similar bisect on the bionic kernel :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?id=091554e4a5b2a7647830a1c7beea781148b51509

091554e4a5b2a7647830a1c7beea781148b51509 is the first bad commit
commit 091554e4a5b2a7647830a1c7beea781148b51509
Author: Lenny Szubowicz <email address hidden>
Date: Tue Nov 30 12:04:00 2021 +0100

efi: Support for MOK variable config table

BugLink: https://bugs.launchpad.net/bugs/1928679

    An EFI configuration table is a much more primitive mechanism
    compared to EFI variables and is well suited for one-way passage
    of static information from a pre-OS environment to the kernel.

Additio...

Here is the result of a similar bisect on the bionic kernel :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?id=091554e4a5b2a7647830a1c7beea781148b51509

091554e4a5b2a7647830a1c7beea781148b51509 is the first bad commit
commit 091554e4a5b2a7647830a1c7beea781148b51509                                                          
Author: Lenny Szubowicz <lszubowi@redhat.com>                                                            
Date:   Tue Nov 30 12:04:00 2021 +0100                                                                   
                                                                                                         
    efi: Support for MOK variable config table                                                           
                                                                                                         
    BugLink: https://bugs.launchpad.net/bugs/1928679                 
                                                                                                         
    Because of system-specific EFI firmware limitations, EFI volatile 
    variables may not be capable of holding the required contents of  
    the Machine Owner Key (MOK) certificate store when the certificate  
    list grows above some size. Therefore, an EFI boot loader may pass                                   
    the MOK certs via a EFI configuration table created specifically for
    this purpose to avoid this firmware limitation.                                                                                                                                                                
                                                                                                         
    An EFI configuration table is a much more primitive mechanism                                                                                                                                                  
    compared to EFI variables and is well suited for one-way passage                                                                                                                                               
    of static information from a pre-OS environment to the kernel.             
                                                                                                         
    This patch adds initial kernel support to recognize, parse,             
    and validate the EFI MOK configuration table, where named                                            
    entries contain the same data that would otherwise be provided                                                                                                                                                 
    in similarly named EFI variables.                                                                                                                                                                              
                                                                                                                                                                                                                   
    Additionally, this patch creates a sysfs binary file for each
    EFI MOK configuration table entry found. These files are read-only
    to root and are provided for use by user space utilities such as
    mokutil.                                 
                                      
    A subsequent patch will load MOK certs into the trusted platform
    key ring using this infrastructure.       
     
    Signed-off-by: Lenny Szubowicz <lszubowi@redhat.com>
    Link: https://lore.kernel.org/r/20200905013107.10457-2-lszubowi@redhat.com
    Signed-off-by: Ard Biesheuvel <ardb@kernel.org>                                                      
    (cherry picked from commit 58c909022a5a56cd1d9e89c8c5461fd1f6a27bb5)
    Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> 
    Acked-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>  
    Acked-by: Tim Gardner <tim.gardner@canonical.com>                   
    Signed-off-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
                                                    
 arch/x86/kernel/setup.c             |   1 +                                                             
 drivers/firmware/efi/Makefile       |   1 +                                                             
 drivers/firmware/efi/arm-init.c     |   1 +                                                             
 drivers/firmware/efi/efi.c          |   9 +
 drivers/firmware/efi/mokvar-table.c | 360 ++++++++++++++++++++++++++++++++++++
 include/linux/efi.h                 |  34 ++++                                                          
 6 files changed, 406 insertions(+)                                                                      
 create mode 100644 drivers/firmware/efi/mokvar-table.c

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2022-01-08:

This bug is awaiting verification that the linux/5.4.0-94.106 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags:

added: verification-needed-focal

Louis Bouchard (louis) on 2022-01-10

tags:

added: verification-done-focal
removed: verification-needed-focal

Revision history for this message

Launchpad Janitor (janitor) wrote on 2022-01-11:

#10

This bug was fixed in the package linux - 5.4.0-94.106

---------------
linux (5.4.0-94.106) focal; urgency=medium

* focal/linux: 5.4.0-94.106 -proposed tracker (LP: #1956628)

  * [Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure Encrypted
    Virtualization(SEV) is enabled (LP: #1956575)
    - x86/ioremap: Map EFI-reserved memory as encrypted for SEV

-- Khalid Elmously <email address hidden> Thu, 06 Jan 2022 16:56:46 -0500

Changed in linux (Ubuntu Focal):
status:	Fix Committed → Fix Released

Revision history for this message

Kleber Sacilotto de Souza (kleber-souza) wrote on 2022-01-11:

#11

Hi Louis,

Thank you very much for reporting the issue and helping us debug and verify the fix!

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

[Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure Encrypted Virtualization(SEV) is enabled

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
linux package