Ubuntu
systemd package

systemd-resolved of systemd 239 is failing in cosmic containers

Bug #1789627 reported by Christian Ehrhardt  on 2018-08-29

This bug report is a duplicate of: Bug #1780227: locking sockets broken due to missing AppArmor socket mediation patches. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	systemd (Ubuntu)	New	Critical	Unassigned

Bug Description

Hi,
a few hours ago I realized that some of my containers have no working dns resolution anymore.
Usually I'd think I broke something in my host network, but I was suspicious s it hit me on my laptop and on a server at about the same time.

After a while I found that in those containers I have:
systemd-resolve --status
Failed to get global data: Failed to activate service 'org.freedesktop.resolve1': timed out (service_start_timeout=25000ms)

Later I found two more things leading me to some assumptions:
1. I had no resolv.conf so the service seems to have issues
root@c:~# ll /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Aug 28 22:18 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
root@c:~# ll /run/systemd/resolve/stub-resolv.conf
ls: cannot access '/run/systemd/resolve/stub-resolv.conf': No such file or directory

2. I realized this only affects cosmic container
Bionic container on the same machine is ok (so Host network should be ok I think).
I didn't realize at first as other cosmic's were ok, but those were the containers not updated yet and tonight there was a publish of https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu4

Knowing that I checked logs and found:
Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[158]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: Dependency failed for Wait for Network to be Configured.
Aug 29 10:23:25 c systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Service...
Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[1]: cloud-init.service: Failed to reset devices.list: Operation not permitted
Aug 29 10:23:25 c systemd[161]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
Aug 29 10:23:25 c systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 2.
Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[1]: Starting Network Service...
Aug 29 10:23:25 c systemd[165]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 3.
Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Service...
Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[168]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 4.
Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed to reset devices.list: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Service...
Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[171]: systemd-networkd.service: Failed at step USER spawning /lib/systemd/systemd-networkd: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Aug 29 10:23:25 c systemd[1]: Stopped Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Service.
Aug 29 10:23:25 c systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[174]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3.
Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[183]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 4.
Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Failed to set devices.allow on /system.slice/systemd-resolved.service: Operation not permitted
Aug 29 10:23:25 c systemd[1]: Starting Network Name Resolution...
Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed to update dynamic user credentials: Permission denied
Aug 29 10:23:25 c systemd[186]: systemd-resolved.service: Failed at step USER spawning /lib/systemd/systemd-resolved: Permission denied
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=217/USER
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Aug 29 10:23:25 c systemd[1]: Stopped Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Aug 29 10:23:25 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:23:25 c systemd[1]: Failed to start Network Name Resolution.
Aug 29 10:23:25 c systemd[1]: Reached target Host and Network Name Lookups.
Aug 29 10:23:25 c systemd[1]: Reached target Network.

# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-08-29 10:39:04 UTC; 10min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
  Process: 328 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
Main PID: 328 (code=exited, status=217/USER)

Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Aug 29 10:39:04 c systemd[1]: Stopped Network Name Resolution.
Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Aug 29 10:39:04 c systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:39:04 c systemd[1]: Failed to start Network Name Resolution.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

Download full text (5.1 KiB)

- Dropped my special KVM-LXD profile that I usually use
- Spawned a new Bionic container
# working
- upgraded to systemd of cosmic

Right on upgrade I get the breakage:
The following packages will be upgraded:
  libnss-systemd libpam-systemd libsystemd0 systemd
4 upgraded, 0 newly installed, 0 to remove and 307 not upgraded.
Need to get 3404 kB of archives.
After this operation, 470 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libnss-systemd amd64 239-7ubuntu4 [111 kB]
Get:2 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libpam-systemd amd64 239-7ubuntu4 [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu cosmic/main amd64 systemd amd64 239-7ubuntu4 [2967 kB]
Get:4 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libsystemd0 amd64 239-7ubuntu4 [214 kB]
Fetched 3404 kB in 1s (4342 kB/s)
(Reading database ... 28491 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_239-7ubuntu4_amd64.deb ...
Unpacking libnss-systemd:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../libpam-systemd_239-7ubuntu4_amd64.deb ...
Unpacking libpam-systemd:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../systemd_239-7ubuntu4_amd64.deb ...
Unpacking systemd (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../libsystemd0_239-7ubuntu4_amd64.deb ...
Unpacking libsystemd0:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Setting up libsystemd0:amd64 (239-7ubuntu4) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up systemd (239-7ubuntu4) ...
Installing new version of config file /etc/systemd/journald.conf ...
Installing new version of config file /etc/systemd/logind.conf ...
Installing new version of config file /etc/systemd/resolved.conf ...
Installing new version of config file /etc/systemd/system.conf ...
Warning: The unit file, source configuration file or drop-ins of systemd-networkd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: Stopping systemd-networkd.service, but it can still be activated by:
  systemd-networkd.socket
Warning: The unit file, source configuration file or drop-ins of systemd-resolved.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for systemd-resolved.service failed because the control process exited with error code.
See "systemctl status systemd-resolved.service" and "journalctl -xe" for details.
dpkg: error processing package systemd (--configure):
installed systemd package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.8.3-2) ...
Processing triggers for dbus (1.12.2-1ubuntu1) ...
dpkg: dependency problems prevent configuration of libnss-systemd:amd64:
libnss-systemd:amd64 depends on systemd (= 239-7ubuntu4); however:
  Package systemd is not configured yet.

dpkg: error processing package libnss-systemd:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpam-systemd:amd64:
libpam-systemd:amd64 depends on systemd (= 239-7...

- Dropped my special KVM-LXD profile that I usually use
- Spawned a new Bionic container
  # working
- upgraded to systemd of cosmic

Right on upgrade I get the breakage:
The following packages will be upgraded:
  libnss-systemd libpam-systemd libsystemd0 systemd
4 upgraded, 0 newly installed, 0 to remove and 307 not upgraded.
Need to get 3404 kB of archives.
After this operation, 470 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libnss-systemd amd64 239-7ubuntu4 [111 kB]
Get:2 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libpam-systemd amd64 239-7ubuntu4 [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu cosmic/main amd64 systemd amd64 239-7ubuntu4 [2967 kB]
Get:4 http://archive.ubuntu.com/ubuntu cosmic/main amd64 libsystemd0 amd64 239-7ubuntu4 [214 kB]
Fetched 3404 kB in 1s (4342 kB/s)   
(Reading database ... 28491 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_239-7ubuntu4_amd64.deb ...
Unpacking libnss-systemd:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../libpam-systemd_239-7ubuntu4_amd64.deb ...
Unpacking libpam-systemd:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../systemd_239-7ubuntu4_amd64.deb ...
Unpacking systemd (239-7ubuntu4) over (237-3ubuntu10.3) ...
Preparing to unpack .../libsystemd0_239-7ubuntu4_amd64.deb ...
Unpacking libsystemd0:amd64 (239-7ubuntu4) over (237-3ubuntu10.3) ...
Setting up libsystemd0:amd64 (239-7ubuntu4) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up systemd (239-7ubuntu4) ...
Installing new version of config file /etc/systemd/journald.conf ...
Installing new version of config file /etc/systemd/logind.conf ...
Installing new version of config file /etc/systemd/resolved.conf ...
Installing new version of config file /etc/systemd/system.conf ...
Warning: The unit file, source configuration file or drop-ins of systemd-networkd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Warning: Stopping systemd-networkd.service, but it can still be activated by:
  systemd-networkd.socket
Warning: The unit file, source configuration file or drop-ins of systemd-resolved.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for systemd-resolved.service failed because the control process exited with error code.
See "systemctl status systemd-resolved.service" and "journalctl -xe" for details.
dpkg: error processing package systemd (--configure):
 installed systemd package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.8.3-2) ...
Processing triggers for dbus (1.12.2-1ubuntu1) ...
dpkg: dependency problems prevent configuration of libnss-systemd:amd64:
 libnss-systemd:amd64 depends on systemd (= 239-7ubuntu4); however:
  Package systemd is not configured yet.

dpkg: error processing package libnss-systemd:amd64 (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpam-systemd:amd64:
 libpam-systemd:amd64 depends on systemd (= 239-7ubuntu4); however:
  Package systemd is not configured yet.

dpkg: error processing package libpam-systemd:amd64 (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          No apport report written because the error message indicates its a followup error from a previous failure.
                                          Errors were encountered while processing:
 systemd
 libnss-systemd:amd64
 libpam-systemd:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@b-systemd:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-08-29 10:55:44 UTC; 19s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
  Process: 819 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
 Main PID: 819 (code=exited, status=217/USER)

Aug 29 10:55:44 b-systemd systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 10:55:44 b-systemd systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Aug 29 10:55:44 b-systemd systemd[1]: Stopped Network Name Resolution.
Aug 29 10:55:44 b-systemd systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Aug 29 10:55:44 b-systemd systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 10:55:44 b-systemd systemd[1]: Failed to start Network Name Resolution.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

I got a report of 239-7ubuntu4 that Metal isn't affected and works being upgraded.

Knowing that I fetched a cosmic VM which as of the current image starts with 239-7ubuntu4 as well and works fine.

Free was so kind to double check if it is me and spawned a cosmic container as well on his system.

I have seen that there is a new systemd in cosmic-proposed.
I tested upgrading the working bionic right through onto 239-7ubuntu5 but it fails with the same errors.

Updating the bug title/description to break in containers only as it seems.

summary:

- systemd-resolved failing in cosmic
+ systemd-resolved of systemd 239 is failing in cosmic containers

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

The result of the Test Free did is here as well, and now I wonder what I did.
It is working for him just fine ... ?

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

But OTOH picking a random system - here a powerpc box and starting a cosmic container reproduces the same issue:

cpaelzer@diamond:~$ lxc launch ubuntu-daily:c cpaelzer-cosmic-systemd
Creating cpaelzer-cosmic-systemd
Starting cpaelzer-cosmic-systemd
cpaelzer@diamond:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.10 (Cosmic Cuttlefish)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu Cosmic Cuttlefish (development branch)"
VERSION_ID="18.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=cosmic
UBUNTU_CODENAME=cosmic
cpaelzer@diamond:~$ lxc exec cpaelzer-cosmic-systemd bash
root@cpaelzer-cosmic-systemd:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-08-29 11:10:32 UTC; 1min 15s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
  Process: 179 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
Main PID: 179 (code=exited, status=217/USER)

Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: Stopped Network Name Resolution.
Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Aug 29 11:10:32 cpaelzer-cosmic-systemd systemd[1]: Failed to start Network Name Resolution.

But OTOH picking a random system - here a powerpc box and starting a cosmic container reproduces the same issue:

cpaelzer@diamond:~$ lxc launch ubuntu-daily:c cpaelzer-cosmic-systemd
Creating cpaelzer-cosmic-systemd                                                                                                                                          
Starting cpaelzer-cosmic-systemd
cpaelzer@diamond:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.10 (Cosmic Cuttlefish)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu Cosmic Cuttlefish (development branch)"
VERSION_ID="18.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=cosmic
UBUNTU_CODENAME=cosmic
cpaelzer@diamond:~$ lxc exec cpaelzer-cosmic-systemd bash
root@cpaelzer-cosmic-systemd:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-08-29 11:10:32 UTC; 1min 15s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
  Process: 179 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=217/USER)
 Main PID: 179 (code=exited, status=217/USER)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

Then OTOH on a Fresh Cosmic VM it works to spawn a cosmic container.

ubuntu@c-systemd:~$ lxc launch ubuntu-daily:cosmic/amd64 c
Creating c
Starting c
ubuntu@c-systemd:~$ lxc exec c bash
root@c:~# systemctl status systemd-resolved
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-08-29 11:15:16 UTC; 29s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
Main PID: 152 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 1152)
   Memory: 6.5M
   CGroup: /system.slice/systemd-resolved.service
           └─152 /lib/systemd/systemd-resolved

Aug 29 11:15:16 c systemd[1]: Starting Network Name Resolution...
Aug 29 11:15:16 c systemd-resolved[152]: Positive Trust Anchors:
Aug 29 11:15:16 c systemd-resolved[152]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Aug 29 11:15:16 c systemd-resolved[152]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Aug 29 11:15:16 c systemd-resolved[152]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in
Aug 29 11:15:16 c systemd-resolved[152]: Using system hostname 'c'.
Aug 29 11:15:16 c systemd[1]: Started Network Name Resolution.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

Some more back and forth,

Interim summary:
x86-18.10->Container (liblxc1 from git): ok
x86-18.10->Container (liblxc1 from deb): ok
ppc-18.10->Container (lxc from deb) : broken
x86-18.04->Container (lxc from snap): broken
x86->KVM-18.10->Container (lxc from deb): ok
x86->KVM-18.10->Container (lxc from snap): ok

I do not recognize the pattern yet :-/

But the PPC based test is on diamond which many of you have access to, please give that a try there.

This blocks all sort of work, so I'd appreciate if we could find what the root cause is.
For the potential that more than just me are affected to have non-usable containers this is crit I think - we can rate it down once we know why it only affects some systems.

Changed in systemd (Ubuntu):
importance:	Undecided → Critical

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

Pattern?: I realized that in all good cases before I had lxd running in 18.10
So I added these runs with the same but with 18.04 KVMs
x86->KVM-18.04->Container (lxc from deb): ok
x86->KVM-18.04->Container (lxc from snap): ok

Ok, that wasn't the reason either :-/

I need to investigate the actual error around "Failed to update dynamic user credentials: Permission denied".

So far the issue reproduces on these systems:
- Diamond (ppc64 18.10 deb)
- Horsea (x86 18.04 deb)
- my Laptop (18.04 snap)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2018-08-29:

On Container restart I found a bunch of unrelated apparmor denies that look like:
[1220983.698955] audit: type=1400 audit(1535545118.043:8745): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-cpaelzer-cosmic-systemd_</var/lib/lxd>" name="/run/" pid=21102 comm="mount" flags="rw, nosuid, nodev, remount"

That is LXD on the Host being denied to do things

Further when restarting systemd-resolved I saw these:
[1221051.971026] audit: type=1400 audit(1535545186.315:8854): apparmor="DENIED" operation="file_lock" profile="lxd-cpaelzer-cosmic-systemd_</var/lib/lxd>" pid=22329 comm="(resolved)" family="unix" sock_type="dgram" protocol=0 addr=none

Knowing that I also realized that the broken systems all had no reboot for quite some time, but the repro KVMs are obviously new.
With that in mind I found bug 1780227 sounds close enough I think.

Rebooted the host to a newer kernel and e voila that is it.

That said I'll make this a dup, but this is a rather "hard" impact.
We should make known that Cosmic since today fails to work in containers prior to Kernels:
- 4.4.0-134.160
- 4.15.0-33.36

Unfortunately the Guest-Container can enforce no dependencies onto the host kernel.
I'll discuss potential extra communication in standup today.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1780227 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntusystemd package

systemd-resolved of systemd 239 is failing in cosmic containers

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
systemd package