Boot error: libgcc_s.so.1 must be installed for pthread_exit to work

As per https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=950254, the "libpthread copies libgcc" snippet in the copy_exec hook function may be needed for other packages such as open-iscsi, btrfs-progs. In this case these may also be affected by this issue. If this is the case, a solution should lie in the hook-functions here.

Status changed to 'Confirmed' because the bug affects multiple users.

Accidentally changed the status... sorry about that. Can anyone please remove the "Fix Released" status? Not sure how to do that...

In case you are running Jammy and gets affected by this issue:

The workaround described consists of booting from a live usb, decrypting the disk, getting a chroot, and only then installing thin-provisioning-tools and running update-initramfs.

This is the script I was using to decrypt the disk and get the chroot (you will need to adjust it for your disk) [1].

In special, the name used with luksOpen should match the one described in /etc/crypttab

[1]:

#!/bin/bash
set -eux
sudo cryptsetup luksOpen /dev/nvme0n1p3 crypt_dev_nvme0n1p3
sudo vgscan --mknodes
sudo vgchange -ay
sudo mkdir /media/disk
sudo mount /dev/mapper/system-root /media/disk/
sudo mount /dev/nvme0n1p1 /media/disk/boot/
sudo cp -L /etc/resolv.conf /media/disk/etc/resolv.conf
cd /
sudo mount -t proc proc /media/disk/proc
sudo mount -t sysfs sys /media/disk/sys
sudo mount -o bind /dev /media/disk/dev
sudo chroot /media/disk /bin/bash

It feels like we should just copy libgcc_s.so.1 and libpthread always. It is tiny in size and something is bound to use them.

Yes, that's my take as well. I was going to proposed a patch to do just that but I decided to see if there was another way to just selectively copy lib{gcc_s,pthread}. I don't think it's worth the hassle, though.

Are you guaranteed to be able to know the set of architecture(s) of libgcc_s needed? That's what's prevented me from suggesting a patch like that.

We have been bitten by this before. And Last time I did proposed to do this:

- create a stub binary tha tis linked with pthread and libgcc_s
- copy_exec that into the initramfs

https://code.launchpad.net/~xnox/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/385243

This ensures that at least one binary wants pthreads and libgcc_s and those things get copied into the initramfs correctly and always for the host architecture.

I had to do this with a binary, because precisely like raof says we never know which architecture we need, and which hw optimized version of those libraries is needed.

I feel like I should upload the above (rebased on jammy) into jammy to get this done once and for all. Kind of sad that I didn't do that 2 years ago. Which would have avoided this issue today.

Dimitri,

I don't know if this will suffice. With glibc's decision to remove libpthread, I believe we will need to explicitly copy libpthread.so, at least temporarily until everything else doesn't explicitly link against it.

Taking the stub binary your MP is generating as an example (running on Jammy):

# gcc -Wl,--no-as-needed -shared -pthread -l:libgcc_s.so.1 -o bla
# ldd bla
        linux-vdso.so.1 (0x00007ffdba94d000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f581d651000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f581d429000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f581d678000)

We see that it's not linked against pthread.

With that branch rebased, and typpos fixed up testing locally does produce this:

Adding binary /usr/lib/initramfs-tools/bin/gcc_s1-stub
Adding binary /lib/x86_64-linux-gnu/libgcc_s.so.1
Adding binary /lib/x86_64-linux-gnu/libc.so.6
Adding binary-link /usr/lib64/ld-linux-x86-64.so.2
Adding binary /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2

Let me see if i can force linkage with pthreads too.

$ gcc -Wl,--no-as-needed -shared -l:libpthread.so.0 -l:libgcc_s.so.1 -o bla

$ ldd bla
 linux-vdso.so.1 (0x00007ffe0e7e6000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007feeeaa32000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007feeeaa18000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feeea7f0000)
 /lib64/ld-linux-x86-64.so.2 (0x00007feeeaa7f000)

might do it.

With above, when testing I now get:

Adding binary /usr/lib/initramfs-tools/bin/gcc_s1-stub
Adding binary /lib/x86_64-linux-gnu/libgcc_s.so.1
Adding binary /lib/x86_64-linux-gnu/libc.so.6
Adding binary-link /usr/lib64/ld-linux-x86-64.so.2
Adding binary /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
Adding binary /lib/x86_64-linux-gnu/libpthread.so.0

which seems alright.

To be fair we don't need the empty pthread in the initramfs, we only need the non-empty dynamically opened libgcc_s, but it shouldn't hurt either.

This bug was fixed in the package initramfs-tools - 0.140ubuntu12

---------------
initramfs-tools (0.140ubuntu12) jammy; urgency=medium

  * Force copy pthread and libgcc_s libraries via a stub binary.

    To avoid circular shared library dependencies between compiler and
    libc, libpthreads chooses to dlopen libgcc_s, instead of linking
    against it. However, we have no way to parse/know what it
    needs. And even when we hardcode to include libgcc_s1, we don't
    have a way to know which libgcc_s1 is needed on a given system and
    from which path to include it. But there is a hook-function to
    copy a binary and all of its shared library dependencies. Thus
    create a stub empty executable, with shared linking against
    pthreads and gcc_s1 and always include it in the initrd. This way
    at initrd creation time, ldd is used to correctly resolve these
    shared library dependencies and correctly copy them into the
    initrd. This removes hardcoding paths as to where these libraries
    must be copied from on the host. LP: #1958594 LP: #1880853

  * For net boot include most of net/usb modules, except niche modems. LP:
    #1950996

  * Cherry-pick updated merge request !26 to support xz compressed
    firmware. Uncompressed firmware is preffered, with a fallback to use
    compressed firmware files. LP: #1942260

  * Cherry-pick updated merge request !51 to decompress compressed kernel
    modules for boot speed performance. LP: #1932329

 -- Dimitri John Ledkov <email address hidden> Tue, 08 Feb 2022 20:43:02 +0000

we are still observing the issue even with initramfs-tools/now 0.140ubuntu13.1 pacakge with python 3.7 version.
do we need to upgrade any other packages to fix this issue?

No, updating initramfs-tools should be enough. Can you open a new bug report to investigate and attach the output of lsinitramfs on the failing initramfs?