glibc 2.34 upgrade will break some essential services

Investigating a bit more, here's what we see when we check the status of docker.socket:

# systemctl status docker.socket
× docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-09-01 02:05:47 UTC; 14s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)

Sep 01 02:05:47 test-docker systemd[1]: Starting Docker Socket for the API.
Sep 01 02:05:47 test-docker systemd[2488]: docker.socket: Failed to resolve group docker: No such process
Sep 01 02:05:47 test-docker systemd[1]: docker.socket: Control process exited, code=exited, status=216/GROUP
Sep 01 02:05:47 test-docker systemd[1]: docker.socket: Failed with result 'exit-code'.
Sep 01 02:05:47 test-docker systemd[1]: Failed to listen on Docker Socket for the API.

What caught my attention is the following line:

Sep 01 02:05:47 test-docker systemd[2488]: docker.socket: Failed to resolve group docker: No such process

It doesn't make sense to me. As can be seen in the bug description, the "docker" group was properly created *before* the service/socket was (tentatively) started.

If we inspect the socket, we see that its group is indeed wrong (it should be "docker", but it's "root"):

# ls -la /var/run/docker.sock
srw-rw---- 1 root root 0 Sep 1 02:05 /var/run/docker.sock

What's strange is that systemd should be responsible for changing the ownership of the socket when the service is started, but it can't (because it fails to "resolve" the "docker" group). strace wasn't very helpful to determine what's going on here.

It's also interesting to note that I can't reproduce the problem with docker.io 20.10.7-0ubuntu1 (from impish-release). The installation finishes just fine.

I noticed that the last upload was about shipping libnetwork into the golang-github-docker-docker-dev package. Initially I don't see how this could have impacted the docker installation here. Maybe libnetwork is messing with the socket creation somehow?

I can reproduce, but I can even reproduce lots of failures by only upgrading "libc6" and "libc-bin" (which come in with "docker.io"), without Docker even installed or being installed. Lots of other services then try to restart and fail to do so, and "apt update" even starts failing to resolve DNS.

After some discussion with mwhudson, I tried the following:

- add proposed
- install the libc6/libc-bin updates (which breaks a lot of stuff)
- restart the container
- stuff is working again
- install docker.io (from proposed)
- profit

And what exactly should we do here to unblock this version from impish-proposed? Should we change debian/tests/docker-in-lxd to enable -proposed in the lxd container?

Indeed, I can confirm that if we enable -proposed, upgrade everything and reboot the container, then the docker.io installation succeeds.

I'm not sure what the best course of action is here, but I'm experimenting a few things and will file a PR if/when I have something workable.

FWIW, I was able to reproduce this problem even inside a VM, which means that this is not related specifically to LXD containers.

Another interesting point here is the fact that a lot of important systemd services are unable to restart after the libc upgrade. We end up with a system without internet connectivity, for example.

I'm adding a glibc task to this bug and a block-proposed tag, as well as retitling it.

What appears to be going on here is that systemd is not restarted as part of the upgrade of glibc so it is still running glibc 2.33. When starting a service that does anything even slightly funky with users and groups (so things that use DynamicUser= like systemd-resolved but also things like docker which just uses Group= on a socket) it forks itself and calls Name Service Switch apis which dlopen nss modules like /lib/x86_64-linux-gnu/libnss_files.so.2. But these now come from the glibc 2.34 package and are not compatible with the libc already loaded into the forked process and so the nss calls all fail.

I don't know why this didn't bite us for other glibc upgrades -- nss modules are basically never cross version compatible afaik. Maybe systemd has changed and used to have an execve between the fork and any access to nss apis?

So my previous comment isn't quite right.

For one thing, I *don't* think this affects DynamicUser=, only User= and Group=. And https://sourceware.org/legacy-ml/libc-help/2016-12/msg00006.html says that nss modules are supposed to be ABI compatible between releases but it appears that for whatever reason, nss_files from 2.34 is not compatible with glibc 2.33. Now to look into why that might be.

Wait a minute...

root@upgrade-testing:~# readelf --wide -s /usr/lib/x86_64-linux-gnu/libnss_files.so.2

Symbol table '.dynsym' contains 6 entries:
   Num: Value Size Type Bind Vis Ndx Name
     0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
     1: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterTMCloneTable
     2: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__
     3: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMCloneTable
     4: 0000000000000000 0 FUNC WEAK DEFAULT UND __cxa_finalize@GLIBC_2.2.5 (3)
     5: 0000000000000000 0 OBJECT GLOBAL DEFAULT ABS GLIBC_PRIVATE

This doesn't look even close to correct!

https://sourceware.org/pipermail/libc-alpha/2021-June/127273.html

stuff from nss_files was moved into glibc proper.
for libnss-db, we addressed this by no longer explicitly linking against -lnss_files.

LP: #1939918

Oh right! Yes that explains everything. I filed an upstream bug https://sourceware.org/bugzilla/show_bug.cgi?id=28300 although I'm not really sure what I expect them to do about it. The obvious thing to do would be to reexec systemd on upgrade but as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=753725 explains that's not completely safe. I guess we could revert that patch series. I can't really see how to fix this without another glibc upload though :/

OK, with thanks to Julian and Dimitri, I think this patch https://paste.ubuntu.com/p/Bf9tDggMvv/ to glibc will avoid the issue and avoid the worst part of the referenced debian bug (the kernel panic). It means another glibc upload though :(

Status changed to 'Confirmed' because the bug affects multiple users.

I've uploaded a glibc fix now and it appears to fix the bug so I'll go ahead and remove the block-proposed tag.

This bug was fixed in the package glibc - 2.34-0ubuntu2

---------------
glibc (2.34-0ubuntu2) impish; urgency=medium

  * d/patches/ubuntu/Fix-close_range-closefrom-tests.patch: Patch from
    upstream to fix test failures in autopkgtest environment (which has a
    pair of fds open that the test suite did not cope with).
  * d/debhelper.in/libc.postinst: go back to restarting systemd on libc6
    upgrade, but carefully. LP: #1942276

 -- Michael Hudson-Doyle <email address hidden> Fri, 03 Sep 2021 09:26:51 +1200

This bug was fixed in the package docker.io - 20.10.7-0ubuntu3

---------------
docker.io (20.10.7-0ubuntu3) impish; urgency=medium

  * d/t/docker-in-lxd:
    Perform a full upgrade and restart of the container before attempting
    to install docker.io. (LP: #1942276)

 -- Sergio Durigan Junior <email address hidden> Wed, 01 Sep 2021 18:58:31 -0400