nspawn on some 32-bit archs blocks _time64 syscalls, breaks upgrade to focal in containers

Bug #1883447 reported by Steve Dodd
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Low
Dan Streetman
Focal
Fix Released
Undecided
Unassigned

Bug Description

[impact]

nspawn fails on armhf

[test case]

setup a bionic armhf system (note that if lxd is used to setup armhf container under arm64 system, the armhf container must have 'security.nesting' set to true) and get a focal img/filesystem to use with systemd-nspawn, e.g.

$ wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-armhf-root.tar.xz
$ mkdir f
$ cd f
$ tar xvf ../focal-server-cloudimg-armhf-root.tar.xz

install systemd-container, and start nspawn; then test anything that uses the time, e.g. just run python:

$ systemd-nspawn
Spawning container f on /root/f.
Press ^] three times within 1s to kill container.
root@f:~# python3
Fatal Python error: pyinit_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xf7bbd310 (most recent call first):
<no Python frame>

[regression potential]

any regression would likely break nspawn creation or operation of containers, particularly on armhf, but possibly on other archs

[scope]

this is needed only in bionic.

this is fixed upstream by commit 6ca677106992321326427c89a40e1c9673a499b2 which was included first in v244, so this is fixed already in focal and later.

[original description]

Recent Linux kernels introduced a number of new syscalls ending in _time64 to fix Y2038 problem; it appears recent glibc, including the version in focal, test for the existence of these. systemd-nspawn in bionic (237-3ubuntu10.38) doesn't know about these so blocks them by default. It seems however glibc isn't expecting an EPERM, causing numerous programs to fail.

In particular, running do-release-upgrade to focal in an nspawn container hosted on bionic will break as soon as the new libc has been unpacked.

Solution (tested here) is to cherrypick upstream commit https://github.com/systemd/systemd/commit/6ca677106992321326427c89a40e1c9673a499b2

A newer libseccomp is also needed but this is already being worked on, see bug #1876055.

It's a pretty trivial fix one the new libseccomp lands, and there is precedent for SRU-ing for a similar issue in bug #1840640.

https://patchwork.kernel.org/patch/10756415/ is apparently the upstream kernel patch, which should give a clearer idea of which architectures are likely to be affected - I noticed it on armhf.

Revision history for this message
Balint Harmath (bharmath) wrote :

Nice write down of the problem. Also provided solution. Here it goes onto the next state.

Changed in systemd (Ubuntu):
status: New → Confirmed
Revision history for this message
Steve Dodd (anarchetic) wrote : Re: nspawn on arm blocks _time64 syscalls, breaks upgrade to focal in containers

Thinking about it, it probably only applies to arm, or at least to 32 bit archs (I think 64bit archs use 64-bit time already.) I'll try and find a reference for that ..

summary: - nspawn blocks _time64 syscalls, breaks upgrade to focal in containers
+ nspawn on arm blocks _time64 syscalls, breaks upgrade to focal in
+ containers
Revision history for this message
Steve Dodd (anarchetic) wrote :

https://patchwork.kernel.org/patch/10756415/ is the upstream kernel patch it seems.

summary: - nspawn on arm blocks _time64 syscalls, breaks upgrade to focal in
- containers
+ nspawn on some 32-bit archs blocks _time64 syscalls, breaks upgrade to
+ focal in containers
description: updated
Revision history for this message
Steve Dodd (anarchetic) wrote :

This bug also seems to generate "Assertion 'clock_gettime(map_clock_id(clock_id), &ts) == 0' failed at src/basic/time-util.c:55, function now(). Aborting" in various places if you try to boot an existing 20.04 container on bionic with systemd-nspawn.

Revision history for this message
Balint Reczey (rbalint) wrote :

This commit is already present in Focal and later releases.

Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Changed in systemd (Ubuntu Focal):
status: New → Fix Released
Revision history for this message
Dan Streetman (ddstreet) wrote :

@anarchetic could you update the description with the SRU template:
https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

and in particular having a clearly documented reproducer in the [test case] section will help.

thanks!

Dan Streetman (ddstreet)
description: updated
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Low
status: New → In Progress
Dan Streetman (ddstreet)
description: updated
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Steve, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.45 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Steve Dodd (anarchetic) wrote :

LGTM!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/237-3ubuntu10.45)

All autopkgtests for the newly accepted systemd (237-3ubuntu10.45) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

corosync/2.4.3-0ubuntu1.1 (armhf)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64)
openssh/1:7.6p1-4ubuntu0.3 (amd64, ppc64el, arm64, i386, armhf, s390x)
linux-hwe-5.0/5.0.0-65.71 (i386)
linux-hwe-5.4/5.4.0-67.75~18.04.1 (i386)
gvfs/1.36.1-0ubuntu1.3.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Dan Streetman (ddstreet) wrote :

root@test:~/f# dpkg -l|grep systemd-container
ii systemd-container 237-3ubuntu10.44 armhf systemd container/nspawn tools
root@test:~/f# systemd-nspawn
Spawning container f on /root/f.
Press ^] three times within 1s to kill container.
root@f:~# python3
Fatal Python error: pyinit_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0xf7aea310 (most recent call first):
<no Python frame>

root@test:~/f# dpkg -l|grep systemd-container
ii systemd-container 237-3ubuntu10.45 armhf systemd container/nspawn tools
root@test:~/f# systemd-nspawn
Spawning container f on /root/f.
Press ^] three times within 1s to kill container.
root@f:~# python3
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

description: updated
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.45

---------------
systemd (237-3ubuntu10.45) bionic; urgency=medium

  [ Ioanna Alifieraki ]
  * d/p/lp1911187-systemctl-do-not-shutdown-immediately-on-scheduled-shutdo.patch:
    Do not shutdown immediately when scheduled shutdown fails (LP: #1911187)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=257135a59455f4e4063e78cdd3f5cfeca2597b5b

  [ Dimitri John Ledkov ]
  * d/p/lp1878969-meson-initialize-time-epoch-to-reproducible-builds-compat.patch:
    meson: initialize time-epoch to reproducible builds compatible value
    (LP: #1878969)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6f5a0c94ff4a486ee0b72af926672b24d16ff5a8

  [ Dan Streetman ]
  * d/p/lp1913189-test-accept-that-char-device-0-0-can-now-be-created-.patch:
    - Fix failing test case under 5.8 kernel (LP: #1913189)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15143ec6cd584a18866390a042348a543e5aa22d
  * d/p/lp1913423-hashmap-make-sure-to-initialize-shared-hash-key-atom.patch:
    Thread-safe init of hashmap shared key (LP: #1913423)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=95c189adb9c3e22576b26b084c7edf001cbc8307
  * d/p/lp1890448-hwdb-Add-EliteBook-to-use-micmute-hotkey.patch:
    Add EliteBook to use micmute hotkey (LP: #1890448)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19b48bdac5129aa772fbcd2dbf8d1bb5c30c1510
  * d/p/debian/patches/lp1902553-test-disable-QEMU-based-testing-for-TEST-16-EXTEND-T.patch:
    Disable TEST-03 run under qemu (LP: #1902553)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4e37d20ec379d169cfd53088d0c3b4d7bb65d25b
  * d/p/debian/patches/lp1883447-seccomp-add-all-time64-syscalls.patch:
    Add *time64 syscalls (LP: #1883447)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a459492c67c5c5855b03daca4b44141705495376
  * d/p/lp1685754-pid1-by-default-make-user-units-inherit-their-umask-.patch:
    Inherit umask for --user processes (LP: #1685754)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=00df8d0e76975594adb765182c587ef495262fe1
  * d/p/debian/patches/lp1880258-log-nxdomain-as-debug.patch:
    Change NXDOMAIN 'errors' to log level debug (LP: #1880258)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9684abed02669bfcf696763b887518cf54cd3f69
  * d/p/lp1913763-udev-rules-add-rule-to-create-dev-ptp_hyperv.patch:
    Create symlink for hyperv-provided ptp device (LP: #1913763)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff2a9ed2ece6bbd86a3d57f42b26cb1a6ca2845a

 -- Ioanna Alifieraki <email address hidden> Tue, 23 Feb 2021 03:45:01 +0200

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.