Debootstrap fails for Noble with base-files 13ubuntu7

Steve I'm sorry but I cannot reproduce this. I have verified this countless of times during working on this and I am also unable to verify this here:

# apt policy debootstrap
debootstrap:
  Installed: 1.0.134ubuntu1
  Candidate: 1.0.134ubuntu1
  Version table:
 *** 1.0.134ubuntu1 500
        500 https://debian.charite.de/ubuntu noble/main amd64 Packages
        500 https://debian.charite.de/ubuntu noble/main i386 Packages
        500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu noble/main i386 Packages
        100 /var/lib/dpkg/status

# rm -r /tmp/nobletest/ && sudo debootstrap --variant=minbase noble /tmp/nobletest &> debootstrap.log
# grep base-files debootstrap.log
I: Retrieving base-files 13ubuntu7
I: Validating base-files 13ubuntu7
I: Extracting base-files...
I: Unpacking base-files...
I: Configuring base-files...
# tail debootstrap.log
I: Configuring libtasn1-6:amd64...
I: Configuring libstdc++6:amd64...
I: Configuring libhogweed6:amd64...
I: Configuring libidn2-0:amd64...
I: Configuring libapt-pkg6.0:amd64...
I: Configuring libp11-kit0:amd64...
I: Configuring libgnutls30:amd64...
I: Configuring apt...
I: Configuring libc-bin...
I: Base system installed successfully.

Your debootstrap log looks fine, it extracts base-files first. At this point, /tmp/nobletest should be empty. Unless I suppose you use a old broken version of debootstrap that used to create the symlinks before extracting the packages?

debootstrap (1.0.130) unstable; urgency=medium

  [ Helmut Grohne ]
  * implement merged-/usr by post-merging. Closes: #1049898

 -- Luca Boccassi <email address hidden> Fri, 18 Aug 2023 16:02:59 +0100

Let me move this to debootstrap tasks instead, I guess we should fix the LTS debootstraps.

I wrote instead but of course it's in addition, I think that's probably the easier outcome than trying to workaround it in the base-files preinst. Consulting with helmut

I forgot we can't rely on preinst in bootstrap so backporting the debootstrap change it will have to be.

As a workaround, --no-merged-usr probably should work fine in old releases. It might be easier to add this to the script as a special case for > noble than backporting new logic we don't need anymore.

Apologies Julian, I completely forgot to include details of my (Jammy / 22.04) build host which would have avoided the confusion. Adding it to the report now (although I know it is sort of too late at this point).

Also, many thanks for the workaround. I can confirm it works in my case.

TL;DR: The fix is not needed on Mantic.

This is just to expand a bit on the brief note in '[Where problems could occur]' ("moved to merging post-extraction"), for clarity.

The code in `scripts/gutsy` (used for noble) is different between Mantic and Jammy with regards to the order of package extraction and usr-merge setup, in first_stage_install().

The issue happens in Jammy because usr-merge setup runs _before_ extraction (thus extraction of base-files fails as the /{bin,lib,lib64,sbin} symlinks already exist):

        setup_merged_usr
        extract $required

But it doesn't happen in Mantic because usr-merge setup runs _after_ extraction (thus extraction of base-files works as the symlinks don't yet exist):

        extract $required
        merge_usr

Slrry I can make this clearer too by adding a mantic task and marking it as fix released.

I confirmed that the resulting target dir of debootstrap in Jammy (with patch/MERGED_USR=no) and Mantic is the same, with a comparison of `ls -lR` (normalized for Month/Day/Time fields).

...

Test command for Jammy (patched) and Mantic:

# debootstrap --arch=amd64 --variant=minbase --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg noble /tmp/nobletest http://archive.ubuntu.com/ubuntu

Dir listing for Jammy (patched) and Mantic:

deboostrap-jammy # ls -lR /tmp/nobletest/ > list.jammy
deboostrap-mantic # ls -lR /tmp/nobletest/ > list.mantic

Normalization and comparison:

$ sed -i 's/[A-Z][a-z][a-z] .[0-9] [0-9][0-9]:[0-9][0-9]/DATETIME/' list.jammy list.mantic

$ diff -U0 list.jammy list.mantic
--- list.jammy 2024-02-26 19:39:49.102864808 -0300
+++ list.mantic 2024-02-26 19:39:49.114864569 -0300
@@ -76 +76 @@
--rw-r--r-- 1 root root 18 DATETIME hostname
+-rw-r--r-- 1 root root 19 DATETIME hostname
@@ -7093 +7093 @@
--rw-r--r-- 1 root root 60238 DATETIME bootstrap.log
+-rw-r--r-- 1 root root 60045 DATETIME bootstrap.log

Hello Åka, or anyone else affected,

Accepted debootstrap into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debootstrap/1.0.126+nmu1ubuntu0.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted debootstrap (1.0.126+nmu1ubuntu0.7) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

debuerreotype/0.14-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#debootstrap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

debootstrap 1.0.126+nmu1ubuntu0.7 tested and verified successfully on my amd64 Jammy install. Thanks!

Now, with the above ppc64el regression warning from ubuntu-sru-bot in mind, I will refrain from changing the tag to verification-done until devs address it and/or advise.

Building an noble image with ubuntu-image, using debootstrap 1.0.126+nmu1ubuntu0.7, worked (on a amd64 Jammy).

The regression in debuerreotype is expected due to the included change from the previous SRU (LP: #1990856), packages are now extracted in a different order for the test case, hence some time stamps shifted.

On

root@known-seal:~# apt policy debootstrap
debootstrap:
  Installed: 1.0.126+nmu1ubuntu0.7
  Candidate: 1.0.126+nmu1ubuntu0.7
  Version table:
 *** 1.0.126+nmu1ubuntu0.7 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.126+nmu1ubuntu0.5 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1.0.126+nmu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

I ran
- debootstrap {focal,jammy,mantic,noble}
- mk-sbuild {focal,jammy,mantic,noble}
- pbuilder-dist {focal,jammy,mantic,noble} create

and with the ubuntu-image run from upils I think we can call that verified.

This bug was fixed in the package debootstrap - 1.0.126+nmu1ubuntu0.7

---------------
debootstrap (1.0.126+nmu1ubuntu0.7) jammy; urgency=medium

  * Disable usrmerge code on noble+ as they ship symlinks in base-files,
    and the code is incompatible (and no longer needed) (LP: #2054925)

debootstrap (1.0.126+nmu1ubuntu0.6) jammy; urgency=medium

  * Support Packages files which are not ordered alphabetically by Package
    field, by backporting upstream commit
    86ca8bcc736ceba53ad4a7d9b10b4c2ab65d739d (LP: #1990856)

 -- Julian Andres Klode <email address hidden> Mon, 26 Feb 2024 10:41:03 +0100

The verification of the Stable Release Update for debootstrap has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The changes for Focal had a syntax error (double ';;' lines, not present in Jammy) which causes debootstrap to fail early and silently, even for a previously working series.

Original:

 $ sudo debootstrap --variant=minbase focal /tmp/debootstrap.focal http://br.archive.ubuntu.com/ubuntu
 ...
 I: Base system installed successfully.
 $ echo $?
 0

Patched:

 $ sudo debootstrap --variant=minbase focal /tmp/debootstrap.focal http://br.archive.ubuntu.com/ubuntu
 $ echo $?
 2

Addressed this issue; verified the diff only changed in that regard; and re-uploaded.

 $ diff -U0 debootstrap_1.0.118ubuntu1.12_1.0.118ubuntu1.13.diff debdiff | grep -v '^@@' | sort | uniq -c | sort
       1 +++ debdiff 2024-03-04 21:01:43.500570118 +0000
       1 --- debootstrap_1.0.118ubuntu1.12_1.0.118ubuntu1.13.diff 2024-02-26 15:24:26.000000000 +0000
      32 +@@ -78,7 +78,14 @@
      32 -+ ;;
      32 -@@ -78,7 +78,15 @@

Testing with the change applied:

 $ sudo debootstrap --variant=minbase noble /tmp/debootstrap.noble http://br.archive.ubuntu.com/ubuntu
 ...
 I: Base system installed successfully.
 $ echo $?
 0

 $ ls -l /tmp/debootstrap.noble/ | grep usr/
 lrwxrwxrwx 1 root root 7 Feb 22 14:39 bin -> usr/bin
 lrwxrwxrwx 1 root root 7 Feb 22 14:39 lib -> usr/lib
 lrwxrwxrwx 1 root root 9 Feb 22 14:39 lib64 -> usr/lib64
 lrwxrwxrwx 1 root root 8 Feb 22 14:39 sbin -> usr/sbin

Hello Åka, or anyone else affected,

Accepted debootstrap into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debootstrap/1.0.118ubuntu1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Apologies, I saw the same issue locally fixed it but must have forgotten to build a new .dsc:/

> Apologies, I saw the same issue locally fixed it but must have forgotten to build a new .dsc:/

No worries; thanks for mentioning!

All autopkgtests for the newly accepted debootstrap (1.0.118ubuntu1.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

debuerreotype/0.10-1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#debootstrap

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!