Ubuntu
grub2-signed package

GCE instances drop to a grub prompt when GRUB_DISTRIBUTOR=Debian is set

Bug #1928040 reported by Matthew Ruffell
24
This bug affects 2 people
Affects Status Importance Assigned to Milestone
grub2-signed (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Fix Released
Critical
Unassigned

Bug Description

[Impact]

GCE cloud instances started with images released prior to 2020-11-11 will fail to reboot when the newest grub2 2.02~beta2-36ubuntu3.32 packages are installed from -updates.

Upon reboot, the instance drops down to a grub prompt, and ceases to boot any further.

The output displayed is:

BdsDxe: loading Boot0003 "debian" from HD(15,GPT,<UUID>,0x2800,0x35000)/\EFI\debian\shimx64.efi
BdsDxe: starting Boot0003 "debian" from HD(15,GPT,<UUID>,0x2800,0x35000)/\EFI\debian\shimx64.efi

UEFI: Attempting to start image.
Description: debian
FilePath: HD(15,GPT,36903981-58D9-4718-A4EC-3D3E6CF6AF42,0x2800,0x35000)/\EFI\debian\shimx64.efi
OptionNumber: 3.

GNU GRUB version 2.04
grub>

Now, on GCE cloud images prior to 2020-11-11, /etc/default/grub.d/50-cloudimg-settings.cfg contained this line:

GRUB_DISTRIBUTOR=Debian

In images after 2020-11-11, this line was REMOVED from /etc/default/grub.d/50-cloudimg-settings.cfg and instead, images fell back to the below line in /etc/default/grub

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

The above line always returns "Ubuntu".

The new grub2 version 2.04-1ubuntu44 now always looks for the grub.cfg in /EFI/ubuntu, with no fallback to look in the directory the EFI grub executable was booted from. Since GRUB_DISTRIBUTOR=Debian means the grub config is only written to /EFI/debian/grub.cfg, the new grub fails to find its config despite being booted from /EFI/debian/grubx64.efi, and hence we drop to a grub shell.

[Testcase]

You can start an instance up on Google Cloud with an affected image with the below command:

gcloud compute instances create test-xenial-1 --image=ubuntu-1604-xenial-v20200429 --image-project=ubuntu-os-cloud

From there:

$ sudo apt update
$ sudo apt install grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed grub-pc-bin grub2-common
$ sudo reboot

The instance will not come back up, and you will see a grub shell in the logs on GCP.

You can also reproduce in KVM. Simply add:

GRUB_DISTRIBUTOR=Debian

to a file in /etc/default/grub.d/50-cloudimg-settings.cfg then:

$ sudo update-grub
$ sudo reboot

Test packages are available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/lp1928040-test

If you install these test packages, you should be able to upgrade grub and reboot without issue.

[Where problems could occur]

We will be changing the grub configuration for every Google cloud instance started with an image produced before 2020-11-11, and there is risk that we could make a change which prevents instances from booting. We should proceed with caution and make sure to test older and newer images.

The fix will be targeted to the /etc/default/grub.d/50-cloudimg-settings.cfg file only, so only cloud instances would get the change, and only if the grub versions match particular versions.

[Other info]

A workaround is to remove the below line from /etc/default/grub.d/50-cloudimg-settings.cfg before installing the new grub packages.

GRUB_DISTRIBUTOR=Debian

See original description
Matthew Ruffell (mruffell)
Changed in grub2 (Ubuntu):
status: New → Fix Released
Changed in grub2 (Ubuntu Xenial):
importance: Undecided → Critical
status: New → In Progress
tags: added: sts xenial
Matthew Ruffell (mruffell)
description: updated
Steve Langasek (vorlon)
affects: grub2 (Ubuntu) → grub2-signed (Ubuntu)
Changed in grub2-signed (Ubuntu):
status: Fix Released → Invalid
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Attached is a debdiff for grub2-signed which issues a sed to remove the GRUB_DISTRIBUTOR=Debian line from /etc/default/grub.d/50-cloudimg-settings.cfg

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Looked at the debdiff - looks sane. One thing I pointed out and discussed with Matthew already is the versioning used in the grub-efi-amd64-signed.postinst addition. The grub2-signed binaries are getting binary versions that are glued together as <grub2-signed version>+<grub2 version>, so normally we'd like to check for that instead of only the source version (so in this case 1.167~16.04.2+2.04-1ubuntu44 I suppose?).

That being said, we decided just to leave it as is since we basically only care about the grub2-signed part to change. Since it's the first part of the version number, all upgrade scenarios should cause this check to be avoided (also 1.167~16.04.2 < 1.167~16.04.2+2.04-1ubuntu44 so it's all good). Documenting this here so we have some history regarding the decision making.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Note to self: after discussions with Steve, Matthew and Phil, it seems the GRUB_DISTRIBUTOR=Debian line is only present in the xenial images, so we shouldn't need any fixes for bionic and newer.

Changed in grub2-signed (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Matthew, or anyone else affected,

Accepted grub2-signed into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.167~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Philip Roche (philroche) wrote :

@sil2100 We have now successfully tested reboot with grub-efi-amd64-signed 1.167~16.04.2+2.04-1ubuntu44 from xenial-proposed with image ubuntu-1604-xenial-v20200429 in GCE

And I confirmed the bug with grub-efi-amd64-signed 1.66.23+2.02~beta2-36ubuntu3.23 prior to testing

Marking verification-done and verification-done-xenial

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Steve Langasek (vorlon)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2-signed - 1.167~16.04.2

---------------
grub2-signed (1.167~16.04.2) xenial; urgency=medium

  * Remove the line GRUB_DISTRIBUTOR=Debian from
    /etc/default/grub.d/50-cloudimg-settings.cfg on select cloud images,
    namely those produced for GCE before 2020-11-11. This ensures that UEFI
    instances can locate efi executables and grub.cfg in correct directories
    since grub 2.04 seems to enforce pedantic locations. (LP: #1928040)

 -- Matthew Ruffell <email address hidden> Tue, 11 May 2021 19:38:29 +1200

Changed in grub2-signed (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for grub2-signed has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.