Xen Dom0 linux boot fails with multiple initrd files

Bug #1987567 reported by Mauricio Faria de Oliveira
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Fix Released
Medium
Mauricio Faria de Oliveira
Focal
Fix Released
Medium
Mauricio Faria de Oliveira
Jammy
Fix Released
Medium
Mauricio Faria de Oliveira
Kinetic
Fix Released
Medium
Mauricio Faria de Oliveira

Bug Description

[Impact]

The linux_xen template seems to be broken for multiple initrd files.

Linux fails to boot when it needs a real initrd but early/microcode
initrd(s) are found by grub-mkconfig.

The 2 patches (merged upstream [2]) allow more than one initrd to be
actually loaded, and the real/non-microcode initrd get used by Linux.

More details in the patchset cover letter [1].

[Test Plan]

    $ sudo apt install microcode-initrd

    $ du --bytes /boot/...
    13660416 /boot/vmlinuz-5.4.0-122-generic
    33062542 /boot/initrd.img-5.4.0-122-generic
    5045248 /boot/microcode.cpio

    $ sudo reboot

- Original:

    [ 2.505207] Kernel panic - not syncing: VFS: Unable to mount root
    fs on unknown-block(0,0)

- Patch 1:

    [ 1.890498] Freeing initrd memory: 4928K
    ...
    [ 2.710948] Kernel panic - not syncing: VFS: Unable to mount root
    fs on unknown-block(0,0)

- Patch 2:

    [ 1.968578] Freeing initrd memory: 32288K
    ...
    [ 2.844889] Run /init as init process
    [ 2.916532] systemd-udevd[148]: Starting version 245.4-4ubuntu3.17

[Where problems could occur]

Regressions would be likely to manifest when booting Linux under Xen
with more than one initrd file (changes contained in this code path).

[Other Info]

[1] https://lists.gnu.org/archive/html/grub-devel/2022-08/msg00137.html
[PATCH 0/2] templates/linux_xen: Properly handle multiple initrd files

grub.git commits:
18d8eafdea23 templates/linux_xen: Properly order the multiple initrd files
b4b4acaf4ec7 templates/linux_xen: Properly load multiple initrd files

Related branches

Changed in grub2 (Ubuntu Focal):
status: New → Triaged
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in grub2 (Ubuntu Jammy):
status: New → Triaged
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in grub2 (Ubuntu Kinetic):
status: New → In Progress
assignee: nobody → Mauricio Faria de Oliveira (mfo)
Changed in grub2 (Ubuntu Focal):
importance: Undecided → Medium
Changed in grub2 (Ubuntu Jammy):
importance: Undecided → Medium
Changed in grub2 (Ubuntu Kinetic):
importance: Undecided → Medium
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Merge Request for Kinetic/devel series:
https://code.launchpad.net/~mfo/grub/+git/grub/+merge/428882

description: updated
description: updated
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Changed in grub2 (Ubuntu Kinetic):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.06-2ubuntu12

---------------
grub2 (2.06-2ubuntu12) kinetic; urgency=medium

  * ubuntu-zfs-enhance-support.patch: Fix missing lines (LP: #1990143)

 -- Julian Andres Klode <email address hidden> Mon, 19 Sep 2022 16:00:47 +0200

Changed in grub2 (Ubuntu Kinetic):
status: Fix Committed → Fix Released
tags: added: foundations-todo
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

testing on jammy:

Launch/Setup:
---

$ multipass launch -c 2 -d 8g -m 2g -n xj jammy
$ multipass shell xj

$ sudo apt update && sudo apt install -y xen-system-amd64

$ sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg
# ... GRUB_TERMINAL=serial, GRUB_TIMEOUT=5, GRUB_TIMEOUT_STYLE=''
$ sudo update-grub && sudo reboot

$ virsh console xj

Before:
---

$ dpkg -s grub-common | grep Version:
Version: 2.06-2ubuntu7

$ sudo apt install -y microcode-initrd

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.15.0-56-generic

$ du -k /boot/initrd.img-* /boot/microcode.cpio
36092 /boot/initrd.img-5.15.0-56-generic
4936 /boot/microcode.cpio

$ sudo reboot # fails
...
[ 2.100482] Freeing initrd memory: 4936K
...
[ 2.799346] VFS: Cannot open root device "UUID=..." or unknown-block(0,0): error -6
...

After:
---

$ sudo add-apt-repository -y ppa:mfo/build-proposed
$ sudo apt install -y grub-common

$ dpkg -s grub-common | grep Version:
Version: 2.06-2ubuntu7.2

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/initrd.img-5.15.0-56-generic
 module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-56-generic
   module2 --nounzip /boot/microcode.cpio

$ sudo reboot # works
...
[ 2.675755] Freeing initrd memory: 36092K
...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

testing on focal:

Launch/Setup:
---

$ multipass launch -c 2 -d 8g -m 2g -n xf focal
$ multipass shell xf

$ sudo apt update && sudo apt install -y xen-system-amd64

$ sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg
# ... GRUB_TERMINAL=serial, GRUB_TIMEOUT=5, GRUB_TIMEOUT_STYLE=''
$ sudo update-grub && sudo reboot

$ virsh console xf

Before:
---

$ dpkg -s grub-common | grep Version:
Version: 2.04-1ubuntu26.15

$ sudo apt install -y microcode-initrd

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio /boot/initrd.img-5.4.0-135-generic

$ du -k /boot/initrd.img-* /boot/microcode.cpio
32296 /boot/initrd.img-5.4.0-135-generic
4936 /boot/microcode.cpio

$ sudo reboot # fails
...
[ 2.097349] Freeing initrd memory: 4936K
...
[ 2.954304] VFS: Cannot open root device "UUID=..." or unknown-block(0,0): error -6
...

After:
---

$ sudo add-apt-repository -y ppa:mfo/build-proposed
$ sudo apt install -y grub-common

$ dpkg -s grub-common | grep Version:
Version: 2.04-1ubuntu26.17

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/initrd.img-5.4.0-135-generic
 module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-135-generic
   module2 --nounzip /boot/microcode.cpio

$ sudo reboot # works
...
[ 1.381813] Freeing SMP alternatives memory: 40K
...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

uploaded to jammy/focal.

Changed in grub2 (Ubuntu Jammy):
status: Triaged → In Progress
Changed in grub2 (Ubuntu Focal):
status: Triaged → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Mauricio, or anyone else affected,

Accepted grub2 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.06-2ubuntu7.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

description: updated
Changed in grub2 (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-jammy
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Mauricio, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Verification done on jammy.
---

$ sudo apt update && sudo apt install -y xen-system-amd64 microcode-initrd

$ sudo add-apt-repository -yp proposed
$ sudo apt install -y grub-common

$ apt policy grub2-common
grub2-common:
  Installed: 2.06-2ubuntu7.2
  Candidate: 2.06-2ubuntu7.2
  Version table:
 *** 2.06-2ubuntu7.2 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
...

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/initrd.img-5.15.0-57-generic
 module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-57-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-57-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-57-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.15.0-57-generic
   module2 --nounzip /boot/microcode.cpio

$ sudo reboot

# it works!

$ dmesg | grep 'Hypervisor detected'
[ 0.000000] Hypervisor detected: Xen PV

$ sudo dmesg | grep initrd
[ 2.250454] Freeing initrd memory: 36148K

$ du -k /boot/initrd.img-* /boot/microcode.cpio
36148 /boot/initrd.img-5.15.0-57-generic
4936 /boot/microcode.cpio

tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Verification done on focal.
---

$ sudo apt update && sudo apt install -y xen-system-amd64 microcode-initrd

$ sudo add-apt-repository -y 'deb http://archive.ubuntu.com/ubuntu focal-proposed main'
sudo apt install -y grub-common microcode-initrd

$ apt policy grub2-common
grub2-common:
  Installed: 2.04-1ubuntu26.17
  Candidate: 2.04-1ubuntu26.17
  Version table:
 *** 2.04-1ubuntu26.17 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
...

$ grep module2 /boot/grub/grub.cfg | grep -v vmlinuz
 module2 --nounzip /boot/initrd.img-5.4.0-137-generic
 module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-137-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-137-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-137-generic
   module2 --nounzip /boot/microcode.cpio
   module2 --nounzip /boot/initrd.img-5.4.0-137-generic
   module2 --nounzip /boot/microcode.cpio

$ sudo reboot

# it works!

$ dmesg | grep 'Hypervisor detected'
[ 0.000000] Hypervisor detected: Xen PV

$ dmesg | grep -i initrd
[ 2.441382] Freeing initrd memory: 32296K

$ du -k /boot/initrd.img-* /boot/microcode.cpio
32296 /boot/initrd.img-5.4.0-137-generic
4936 /boot/microcode.cpio

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.06-2ubuntu7.2

---------------
grub2 (2.06-2ubuntu7.2) jammy; urgency=medium

  * linux_xen: Properly handle multiple initrd files (LP: #1987567)
    - d/p/linux_xen-Properly-load-multiple-initrd-files.patch
    - d/p/linux_xen-Properly-order-multiple-initrd-files.patch

 -- Mauricio Faria de Oliveira <email address hidden> Sun, 18 Dec 2022 18:21:26 -0300

Changed in grub2 (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.04-1ubuntu26.17

---------------
grub2 (2.04-1ubuntu26.17) focal; urgency=medium

  * linux_xen: Properly handle multiple initrd files (LP: #1987567)
    - d/p/linux_xen-Properly-load-multiple-initrd-files.patch
    - d/p/linux_xen-Properly-order-multiple-initrd-files.patch

 -- Mauricio Faria de Oliveira <email address hidden> Sun, 18 Dec 2022 18:29:03 -0300

Changed in grub2 (Ubuntu Focal):
status: Fix Committed → Fix Released
Benjamin Drung (bdrung)
tags: removed: foundations-todo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.