Systemd fails to serialize tasks correctly on daemon-reload

For now assigning to myself to test/verify the fix and making sure no other spends cycles on it if there is no need to yet.

I was able to prove that this fixes my issue around libvirt-lxc loosing containers on restart.
It could have an almost random number of other effects that are broken by it and we don't know so I'd highly appreciate considering that fix - even for SRU's actually.

But OTOH I need more of a systemd Expert to look at it and consider potential side-effects.

A test version can be found in [1] and its ppa [2].

I'll also re-file my Debian bug towards systemd to let them consider it as well.
But for Ubuntu atm - please review and consider the debdiff I attach hereby.

[1]: https://bileto.ubuntu.com/#/ticket/2857
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2857/

FYI - To help Debian as well and avoid long term Delta I supplied a backport for debians systemd git under the debbug I linked above.

But since our delta seems rather long I'm not sure if a re-merge is planned soon, so please consider adding the patch on Ubuntu as well (maybe wait until after the Debian discussion acked it).

Ah in case one wants to check the issue I filed with upstream as well as the fix there have good steps documented. No need to just copy them until we need them for an SRU template.

In reply to:
[13:47] <xnox> cpaelzer, can you simply state the ids of the patches you are after; and series you want them in; and i can look into cherrypciking/backporting them and SRUing them throughout.
[13:48] <xnox> cpaelzer, as i do manage systemd SRUs holistically, and there is always a queue of things that people want/need in the next upload.

First of all thanks Dimitri, I think I backported them already (apply as-is), but if you want to do that along your usual work with the ./debian/git-cherry-pick script that is just as fine.
I see that this helps you more than a debdiff to do your SRUs holistically.

Xenial is not affected.
Very strictly Yakkety has only 4 days service left, but I checked and it is affected.
Zesty and Artful are affected for sure.

The patches upstream are:
aae7e17f9c50015789ccc18a2b8d177730f2d2be
8b108bd0ef46f05fa7938abc787e919249459874

So TL;DR: Please aae7e17f + 8b108bd0 into (Yakkety), Zesty and Artful

Hi,
I wanted to notify that the merge of systemd version 234 fixed this bug - it is only in proposed so far, but I checked against that.
So we can consider artful done (soon), but it would be great to consider SRUs.

Hi you mentioned to bulk-tackle those to save some time on upload verification.
Any news on this one go forward - might just need a few status updates and then SRU work?

Hello ChristianEhrhardt, or anyone else affected,

Accepted systemd into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/232-21ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

# Before upgrade
./test.sh
              accounts-daemon.service : Tasks: 3 (limit: 4915)
                          atd.service : Tasks: 1 (limit: 4915)
                         cron.service : Tasks: 1 (limit: 4915)
                         dbus.service : Tasks: 1 (limit: 4915)
                       iscsid.service : Tasks: 2 (limit: 4915)
                       polkit.service : Tasks: 3 (limit: 4915)
                      rsyslog.service : Tasks: 4 (limit: 4915)
                        snapd.service : Tasks: 9 (limit: 4915)
                          ssh.service : Tasks: 1 (limit: 4915)
             systemd-journald.service : Tasks: 1 (limit: 4915)
               systemd-logind.service : Tasks: 1 (limit: 4915)
             systemd-resolved.service : Tasks: 1 (limit: 4915)
              accounts-daemon.service :
                          atd.service :
                         cron.service :
                         dbus.service :
                       iscsid.service :
                       polkit.service :
                      rsyslog.service :
                        snapd.service :
                          ssh.service :
             systemd-journald.service :
               systemd-logind.service :
             systemd-resolved.service :
              accounts-daemon.service : Tasks: 3 (limit: 4915)
                          atd.service : Tasks: 1 (limit: 4915)
                         cron.service : Tasks: 1 (limit: 4915)
                         dbus.service : Tasks: 1 (limit: 4915)
                       iscsid.service : Tasks: 2 (limit: 4915)
                       polkit.service : Tasks: 3 (limit: 4915)
                      rsyslog.service : Tasks: 4 (limit: 4915)
                        snapd.service : Tasks: 9 (limit: 4915)
                          ssh.service : Tasks: 1 (limit: 4915)
             systemd-journald.service : Tasks: 1 (limit: 4915)
               systemd-logind.service : Tasks: 1 (limit: 4915)
             systemd-resolved.service : Tasks: 1 (limit: 4915)

# Confirmed still failing

# Upgrade install
root@zesty-test:~# vim /etc/apt/sources.list
root@zesty-test:~# apt update
Hit:1 http://archive.ubuntu.com/ubuntu zesty InRelease
Hit:2 http://archive.ubuntu.com/ubuntu zesty-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu zesty-backports InRelease
Get:4 http://archive.ubuntu.com/ubuntu zesty-proposed InRelease [240 kB]
Hit:5 http://security.ubuntu.com/ubuntu zesty-security InRelease
Get:6 http://archive.ubuntu.com/ubuntu zesty-proposed/universe Sources [15.7 kB]
Get:7 http://archive.ubuntu.com/ubuntu zesty-proposed/main Sources [12.4 kB]
Get:8 http://archive.ubuntu.com/ubuntu zesty-proposed/main amd64 Packages [22.2 kB]
Get:9 http://archive.ubuntu.com/ubuntu zesty-proposed/main Translation-en [13.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu zesty-proposed/universe amd64 Packages [27.4 kB]
Get:11 http://archive.ubuntu.com/ubuntu zesty-proposed/universe Translation-en [16.8 kB]
Fetched 349 kB in 0...

Hello ChristianEhrhardt, or anyone else affected,

Accepted systemd into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/232-21ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Reproducing the bug on zesty:
$ dpkg-query -W systemd
systemd 232-21ubuntu5
$ bash test.sh
              accounts-daemon.service : Tasks: 3 (limit: 4915)
                        acpid.service : Tasks: 1 (limit: 4915)
                          atd.service : Tasks: 1 (limit: 4915)
                         cron.service : Tasks: 1 (limit: 4915)
                         dbus.service : Tasks: 1 (limit: 4915)
                       iscsid.service : Tasks: 2 (limit: 4915)
                 lvm2-lvmetad.service : Tasks: 1 (limit: 4915)
                        lxcfs.service : Tasks: 3 (limit: 4915)
                       polkit.service : Tasks: 3 (limit: 4915)
                      rsyslog.service : Tasks: 4 (limit: 4915)
                        snapd.service : Tasks: 6 (limit: 4915)
                          ssh.service : Tasks: 1 (limit: 4915)
             systemd-journald.service : Tasks: 1 (limit: 4915)
               systemd-logind.service : Tasks: 1 (limit: 4915)
             systemd-networkd.service : Tasks: 1 (limit: 4915)
             systemd-resolved.service : Tasks: 1 (limit: 4915)
            systemd-timesyncd.service : Tasks: 2 (limit: 4915)
                systemd-udevd.service : Tasks: 1
              accounts-daemon.service :
                        acpid.service :
                          atd.service :
                         cron.service :
                         dbus.service :
                       iscsid.service :
                 lvm2-lvmetad.service :
                        lxcfs.service :
                       polkit.service :
                      rsyslog.service :
                        snapd.service :
                          ssh.service :
             systemd-journald.service :
               systemd-logind.service :
             systemd-networkd.service :
             systemd-resolved.service :
            systemd-timesyncd.service :
                systemd-udevd.service :
              accounts-daemon.service : Tasks: 3 (limit: 4915)
                        acpid.service : Tasks: 1 (limit: 4915)
                          atd.service : Tasks: 1 (limit: 4915)
                         cron.service : Tasks: 1 (limit: 4915)
                         dbus.service : Tasks: 1 (limit: 4915)
                       iscsid.service : Tasks: 2 (limit: 4915)
                 lvm2-lvmetad.service : Tasks: 1 (limit: 4915)
                        lxcfs.service : Tasks: 3 (limit: 4915)
                       polkit.service : Tasks: 3 (limit: 4915)
                      rsyslog.service : Tasks: 4 (limit: 4915)
                        snapd.service : Tasks: 6 (limit: 4915)
                          ssh.service : Tasks: 1 (limit: 4915)
             systemd-journald.service : Tasks: 1 (limit: 4915)
               systemd-logind.service : Tasks: 1 (limit: 4915)
             systemd-networkd.service : Tasks: 1 (limit: 4915)
             systemd-resolved.service : Tasks: 1 (limit: 4915)
            systemd-timesyncd.service : Tasks: 2 (limit: 4915)
                systemd-udevd.service :...

This bug was fixed in the package systemd - 232-21ubuntu7

---------------
systemd (232-21ubuntu7) zesty; urgency=medium

  * networkd: accept `:' in ifnames in systemd/networkd. (LP: #1714933)
  * networkd: add support for ActiveSlave and PrimarySlave netdev options.
    (LP: #1709135)
  * Cherrypick upstream fix for a race between .mount and .automount units,
    which currently may result in automounts hanging. (LP: #1709649)
  * systemd.postinst: Fix-up version number check in the previous sru.
    The version check in the postinst was too tight, thus the SRU fix failed
    validation. (LP: #1710410)

systemd (232-21ubuntu6) zesty; urgency=medium

  * link: Fix offload features initialization.
    This fixes a regression introduced in v232 which caused TCP
    segmentation offloads being disabled by default, resulting in
    significant performance issues under certain conditions. (Closes: #864073)
    (LP: #1703393)
  * loginctl: Fix loginctl ignoring user given session IDs at command-line
    (LP: #1682154)
  * Disable fallback DNS servers.
    This causes resolved to call-home to google, attempt to access network when
    none is available, and spams logs. (LP: #1449001)
  * initramfs-tools: trigger udevadm add actions with subsystems first.
    This updates the initramfs-tools init-top udev script to trigger udevadm
    actions with type specified. This mimicks the
    systemd-udev-trigger.service. Without type specified only devices are
    triggered, but triggering subsystems may also be required and should happen
    before triggering the devices. This is the case for example on s390x with zdev
    generated udev rules. (LP: #1713536)
  * Enable systemd-resolved by default. (LP: #1710410)
  * core: fix systemd failing to serialize tasks correctly on daemon-reload.
    (LP: #1702823)

 -- Dimitri John Ledkov <email address hidden> Wed, 04 Oct 2017 14:22:02 +0100

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.