grub 2.12~rc1 fails to load files from large directories on XFS

As we don't have a fix for this in grub yet, we want ubuntu-release-upgrader to block upgrades to mantic in the meantime if it detects that /boot is on xfs.

--> if /boot is on xfs AND the system is not booted under UEFI, since we do not expect this to impact bootability when using the unified grub.efi image.

the check for whether the system is booted under UEFI is [ -d /sys/firmware/efi ]

Hello Julian, or anyone else affected,

Accepted ubuntu-release-upgrader into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:23.10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This has been fixed upstream: https://git.savannah.gnu.org/cgit/grub.git/commit/?id=07318ee7e11a00b9c1dea4c6b4edf62af35a511a

This bug was fixed in the package grub2 - 2.12~rc1-12ubuntu2

---------------
grub2 (2.12~rc1-12ubuntu2) noble; urgency=medium

  * Merge from Debian unstable; remaining changes:
    - Add Ubuntu sbat data
    - build-efi-images: do not produce -installer.efi.signed. LP: 1863994
    - grub-common: Install canonical-uefi-ca.crt
    - Check signatures
    - Support installing to multiple ESP (LP: 1871821)
    - Disable various bits on i386
    - Split out unsigned artefacts into grub2-unsigned
    - Vcs-Git: Point to ubuntu packaging branch
    - Relax dependencies on grub-common and grub2-common
    - grub-pc: Avoid the possibility of breaking grub on SRU update due
      to ABI change
    - UBUNTU: Default timeout changes
    - Revert "Add jfs module to signed UEFI images. Closes: #950959"
    - Revert "Add f2fs module to signed UEFI images"
    - Install grub-initrd-fallback.service again
    - Build using -O1 on s390x to avoid misoptimization
    - grub-check-signatures: Support gzip compressed kernels (LP: #1954683)
    - grub-multi-install: Reset partition type between partitions (LP: #1997795)
    - Drop i386 from grub-efi-amd64* (LP: #2020907)
    - Turn depends on grub-efi-amd64/arm64 unversioned
    - forward port fix for LP: #1926748
    - Make the grub2/no_efi_extra_removable setting work correctly
    - Forward port the fix for LP: #1930742 and make it conditional (xenial/bionic only)
    - Build grub2-unsigned packages with xz compression
    - Revert: "Have -bin packages Break pre-2.12 -signed packages.", this is not
      compatible with our versioning schemes.
    - Install a /usr/lib/grub/grub-sort-version and use that to sort versions as
      it respects GRUB_FLAVOUR_ORDER. Depend on python3 to do so.
    - rules: Add DPKG_BUILDPACKAGE_OPTIONS to generate-grub2-unsigned
    - Replaced patches:
      - installe-signed.patched
      - grub-install-extra-removable.patch
      - grub-install-removable-shim.patch
    - Added patches:
      + rhboot-f34-dont-use-int-for-efi-status.patch
      + rhboot-f34-make-exit-take-a-return-code.patch
      + suse-grub.texi-add-net_bootp6-document.patch
      + ubuntu-add-devicetree-command-support.patch
      + ubuntu-add-initrd-less-boot-fallback.patch
      + ubuntu-add-initrd-less-boot-messages.patch
      + ubuntu-boot-from-multipath-dependent-symlink.patch
      + ubuntu-dont-verify-loopback-images.patch
      + ubuntu-fix-lzma-decompressor-objcopy.patch
      + ubuntu-grub-install-extra-removable.patch
      + ubuntu-install-signed.patch
      + ubuntu-mkconfig-leave-breadcrumbs.patch
      + ubuntu-os-prober-auto.patch
      + ubuntu-recovery-dis_ucode_ldr.patch
      + ubuntu-resilient-boot-boot-order.patch
      + ubuntu-resilient-boot-ignore-alternative-esps.patch
      + ubuntu-shorter-version-info.patch
      + ubuntu-speed-zsys-history.patch
      + ubuntu-support-initrd-less-boot.patch
      + ubuntu-verifiers-last.patch
      + ubuntu-zfs-enhance-support.patch
      + ubuntu-zfs-gfxpayload-dynamic.patch
      + ubuntu-zfs-gfxpayload-keep-default.patch
      + ubuntu-zfs-insmod-xzio-and-lzopio-on-xen.patch
      + ubuntu-zfs-mkconfig-recovery-title.patch
      + ubuntu-zfs-mkco...

This bug was fixed in the package ubuntu-release-upgrader - 1:24.04.3

---------------
ubuntu-release-upgrader (1:24.04.3) noble; urgency=medium

  * DistUpgradeQuirk: do not use XFS /boot quirk for arm64 UEFI
  * Run pre-build.sh: updating po files, mirrors, and demotions.

 -- Nick Rosbrook <email address hidden> Sun, 05 Nov 2023 01:47:58 -0400

I have verified this fix using ubuntu-release-upgrader 1:23.10.14 from mantic-proposed:

(1) On a BIOS system with XFS /boot partition:

ubuntu@ubuntu:~$ mount | grep /boot
/dev/sda1 on /boot type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
ubuntu@ubuntu:~$ ls /sys/firmware/
acpi dmi memmap qemu_fw_cfg
ubuntu@ubuntu:~$ do-release-upgrade --proposed
Checking for a new Ubuntu release

= Welcome to Ubuntu 23.10 'Mantic Minotaur' =

The Ubuntu team is proud to announce Ubuntu 23.10 'Mantic Minotaur'.

To see what's new in this release, visit:
  https://wiki.ubuntu.com/ManticMinotaur/ReleaseNotes

Ubuntu is a Linux distribution for your desktop or server, with a fast
and easy install, regular releases, a tight selection of excellent
applications installed by default, and almost any other software you
can imagine available through the network.

We hope you enjoy Ubuntu.

== Feedback and Helping ==

If you would like to help shape Ubuntu, take a look at the list of
ways you can participate at

  http://www.ubuntu.com/community/participate/

Your comments, bug reports, patches and suggestions will help ensure
that our next release is the best release of Ubuntu ever. If you feel
that you have found a bug please read:

  http://help.ubuntu.com/community/ReportingBugs

Then report bugs using apport in Ubuntu. For example:

  ubuntu-bug linux

will open a bug report in Launchpad regarding the linux package.

If you have a question, or if you think you may have found a bug but
aren't sure, first try asking on the #ubuntu or #ubuntu-bugs IRC
channels on Libera.Chat, on the Ubuntu Users mailing list, or on the
Ubuntu forums:

  http://help.ubuntu.com/community/InternetRelayChat
  http://lists.ubuntu.com/mailman/listinfo/ubuntu-users
  http://www.ubuntuforums.org/

== More Information ==

You can find out more about Ubuntu on our website, IRC channel and wiki.
If you're new to Ubuntu, please visit:

  http://www.ubuntu.com/

To sign up for future Ubuntu announcements, please subscribe to Ubuntu's
very low volume announcement list at:

  http://lists.ubuntu.com/mailman/listinfo/ubuntu-announce

Continue [yN] y
Get:1 Upgrade tool signature [833 B]
Get:2 Upgrade tool [1,267 kB]
Fetched 1,268 kB in 0s (0 B/s)
authenticate 'mantic.tar.gz' against 'mantic.tar.gz.gpg'
extracting 'mantic.tar.gz'

Reading cache

Checking package manager
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Hit http://us.archive.ubuntu.com/ubuntu lunar InRelease
Hit http://us.archive.ubuntu.com/ubuntu lunar-updates InRelease
Hit http://us.archive.ubuntu.com/ubuntu lunar-backports InRelease
Hit http://us.archive.ubuntu.com/ubuntu lunar-security InRelease ...

Hi Nick,

Thanks for performing the verification and providing the verification logs.

The changes for Mantic also include a check for arm64/UEFI, but that isn't mentioned in this bug.
Could you or someone please clarify that and add a test case for it?

The verification steps cover BIOS with/without XFS on /boot, but do not cover an unaffected UEFI system (eg, amd64), as the test plan suggests:

```
We should also test upgrades on "normal"/unaffected system to make sure it is not unnecessarily prevented.
```

Could that be verified as well, please?
Thanks!

This bug has been reused for both the grub and ubuntu-release-upgrader SRUs, however the test case only covers the ubuntu-release-upgrader handling. Please provide a test case for verifying this fix.

> The changes for Mantic also include a check for arm64/UEFI, but that isn't mentioned in this bug.

I do not see that, you must have been looking at something else?

>> We should also test upgrades on "normal"/unaffected system to make sure it is not unnecessarily prevented.
> Could that be verified as well, please?

I converted a lunar VM to xfs /boot, ran do-release-upgrade -p and it worked fine.

Hello Julian, or anyone else affected,

Accepted grub2-unsigned into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.12~rc1-10ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

> The changes for Mantic also include a check for arm64/UEFI, but that isn't mentioned in this bug.
> Could you or someone please clarify that and add a test case for it?

This is not the case for the ubuntu-release-upgrader in mantic-proposed.

This bug was fixed in the package ubuntu-release-upgrader - 1:23.10.14

---------------
ubuntu-release-upgrader (1:23.10.14) mantic; urgency=medium

  * DistUpgradeQuirks: prevent upgrades on BIOS systems with XFS /boot
    (LP: #2039172)
  * Run pre-build.sh: updating po files, mirrors, and demotions.

 -- Nick Rosbrook <email address hidden> Sun, 05 Nov 2023 01:39:16 -0400

Hello Julian, or anyone else affected,

Accepted grub2 into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.12~rc1-10ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted grub2 (2.12~rc1-10ubuntu4.1) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

ubiquity/23.10.8 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#grub2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Please note that the autopkgtest failure above has now been resolved.

Verification is okay, grub-pc now boots from XFS /boot, however the caveat is that it needs to be manually reinstalled first with dpkg-reconfigure grub-pc because of LP#1889556 and only the LP#2043995 upload to noble removes that workaround

On an fstransformed XFS filesystem it spams the console with: `error: not a correct XFS inode` messages before booting.

On a new install of Ubuntu Server 23.10 on XFS where I reinstalled and configured GRUB via chrooting in from a live environment it works without errors.

Hey,

I got redirected here from an error message while trying to upgrade my system from 23.04 to 23.10.
It looks like the issue has been resolved, but I'm still stuck with this message:

>Cannot upgrade this system
>
>Due to a bug in grub, this system cannot be safely upgraded at this
>time.
>
>See https://launchpad.net/bugs/2039172.

Any ideas on how to fix this, beside reformatting /boot?

Thank you!

We're still waiting for the grub update to be released.