NFSd does not start due to systemd ordering cycle with setserial

I wonder if some update arrived between the bug reporting...

After reboot:
[ 43.777385] systemd[1]: Found dependency on nfs-server.service/start
[ 43.803123] systemd[1]: Found dependency on network.target/start
[ 43.828678] systemd[1]: Breaking ordering cycle by deleting job NetworkManager.service/start

the rpcinfo shows it started OK.

There is something weird going on, since every boot behaves a bit differently:

[ 44.264445] systemd[1]: Found ordering cycle on nfs-server.service/start
[ 44.264447] systemd[1]: Found dependency on nfs-mountd.service/start
[ 44.264448] systemd[1]: Found dependency on network.target/start
[ 44.264449] systemd[1]: Found dependency on NetworkManager.service/start
[ 44.264450] systemd[1]: Found dependency on basic.target/start
[ 44.264451] systemd[1]: Found dependency on sockets.target/start
[ 44.264452] systemd[1]: Found dependency on acpid.socket/start
[ 44.264453] systemd[1]: Found dependency on sysinit.target/start
[ 44.264455] systemd[1]: Found dependency on setserial.service/start
[ 44.264457] systemd[1]: Found dependency on remote-fs.target/start
[ 44.264460] systemd[1]: Found dependency on remote-fs-pre.target/start
[ 44.264461] systemd[1]: Found dependency on nfs-server.service/start
[ 44.264462] systemd[1]: Breaking ordering cycle by deleting job nfs-mountd.service/start
[ 44.264463] systemd[1]: Job nfs-mountd.service/start deleted to break ordering cycle starting with nfs-server.service/start

This time NFS is not working.

rpcinfo
   program version netid address service owner
    100000 4 tcp6 ::.0.111 portmapper superuser
    100000 3 tcp6 ::.0.111 portmapper superuser
    100000 4 udp6 ::.0.111 portmapper superuser
    100000 3 udp6 ::.0.111 portmapper superuser
    100000 4 tcp 0.0.0.0.0.111 portmapper superuser
    100000 3 tcp 0.0.0.0.0.111 portmapper superuser
    100000 2 tcp 0.0.0.0.0.111 portmapper superuser
    100000 4 udp 0.0.0.0.0.111 portmapper superuser
    100000 3 udp 0.0.0.0.0.111 portmapper superuser
    100000 2 udp 0.0.0.0.0.111 portmapper superuser
    100000 4 local /run/rpcbind.sock portmapper superuser
    100000 3 local /run/rpcbind.sock portmapper superuser

But sudo systemctl start nfs-kernel-server
starts nfs server beautifully

The nfs-server unit has a wrong dependency on 'network.target'. The only services that should depend on this are those which implement the network management service itself.

I assumed that this would be fixed by removing the dependency on network.target (which is certainly wrong), but I'm not sure that it does, because nfs-server is Before=remote-fs-pre.target.

More investigation is needed.

I am getting the same problem here, but using the open-iscsi initiator (which - of course - depends on the network being initialized):

Found ordering cycle on sysinit.target/start
[ 3.657825] systemd[1]: Found dependency on open-iscsi.service/start
[ 3.657827] systemd[1]: Found dependency on network-online.target/start
[ 3.657828] systemd[1]: Found dependency on NetworkManager-wait-online.service/start
[ 3.657830] systemd[1]: Found dependency on basic.target/start
[ 3.657831] systemd[1]: Found dependency on sockets.target/start
[ 3.657833] systemd[1]: Found dependency on acpid.socket/start
[ 3.657834] systemd[1]: Found dependency on sysinit.target/start
[ 3.657836] systemd[1]: Breaking ordering cycle by deleting job open-iscsi.service/start
[ 3.657837] systemd[1]: Job open-iscsi.service/start deleted to break ordering cycle starting with sysinit.target/start

So, it is quite a systematic problem with the System-V init scripts and their dependency setup in systemd.

I can confirm this, by installing network-manager, nfs-kernel-server, and setserial into a clean VM. setserial's init.d script is very demanding as it both wants to start in run level "S" (sysinit.target in systemd's mapping) *and* requires $remote_fs. But we can't start remote file systems that early if we need NetworkManager for them, this only works with very simple networking like ifupdown.

I suppose /etc/init.d/setserial wants $remote_fs as it reads stuff from /var/lib/setserial/ which might potentially be on NFS. Having setserial on an NFS /var with NetworkManager is certainly a very special case, and I suppose we could just ignore it. But I think we can relax the early startup instead.

As far as I can see, the most appropriate course here is to provide a systemd unit for setserial with more fine-grained ordering, or just a drop-in to relax the overly harsh "Default-Start: S" (as it really just Before=system-getty.slice). I don't want to completely replace the init.d script as it has lots of code (which really doesn't belong there, but I leave that to the Debian maintainer, and this is too intrusive for an SRU).

Vivid and Wily fixes uploaded.

This bug was fixed in the package setserial - 2.17-48ubuntu1

---------------
setserial (2.17-48ubuntu1) wily; urgency=medium

  * Add debian/setserial.service systemd unit, mostly to relax the very
    demanding Before=sysinit.target plus After=remote-fs.target combination
    resulting from the init.d script's $remote_fs dependency. This causes an
    insatisfiable dependency loop when using NFS and NetworkManager.
    (LP: #1452644)
  * Add dh-systemd build dependency and call dh_systemd_enable in
    debian/rules.

 -- Martin Pitt <email address hidden> Wed, 27 May 2015 11:57:47 +0200

Is this going to be fixed in Vivid? It's randomly crippling my server on reboots. Can I install the Wily version on 15.04?

Oh, my vivid SRU got rejected: https://launchpad.net/ubuntu/vivid/+queue?queue_state=4&queue_text=setserial

I don't know why, I'll just reupload it.

Hello kex, or anyone else affected,

Accepted setserial into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/setserial/2.17-48ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

hello.

The fix removes setserial from the dependency loop, but there are other dependencies, so nfsd is still not starting:

kesä 10 22:29:41 xpc systemd[1]: Cannot add dependency job for unit gssproxy.service, ignoring: Unit gssproxy.service failed to load: No such file or directory.
kesä 10 22:29:41 xpc systemd[1]: Cannot add dependency job for unit fancontrol.service, ignoring: Unit fancontrol.service is masked.
kesä 10 22:29:41 xpc systemd[1]: Found ordering cycle on remote-fs-pre.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on nfs-server.service/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on network.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on NetworkManager.service/verify-active
kesä 10 22:29:41 xpc systemd[1]: Found dependency on basic.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on sockets.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on cups.socket/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on sysinit.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on later-readahead.service/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on remote-fs.target/start
kesä 10 22:29:41 xpc systemd[1]: Found dependency on remote-fs-pre.target/start
kesä 10 22:29:41 xpc systemd[1]: Breaking ordering cycle by deleting job nfs-server.service/start
kesä 10 22:29:41 xpc systemd[1]: Job nfs-server.service/start deleted to break ordering cycle starting with remote-fs-pre.target/start
kesä 10 22:29:41 xpc systemd[1]: Dependency failed for pNFS block layout mapping daemon.
kesä 10 22:29:41 xpc systemd[1]: ureadahead.service failed.

sudo systemctl status setserial
● setserial.service - controls configuration of serial ports
   Loaded: loaded (/lib/systemd/system/setserial.service; enabled; vendor preset: enabled)
   Active: active (exited) since ke 2015-06-10 22:29:46 EEST; 31s ago
     Docs: man:setserial(8)
  Process: 862 ExecStart=/etc/init.d/setserial start (code=exited, status=0/SUCCESS)
 Main PID: 862 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/setserial.service

kesä 10 22:29:46 xpc systemd[1]: Starting controls configuration of serial.....
kesä 10 22:29:46 xpc setserial[862]: Loading the saved-state of the serial.....
kesä 10 22:29:46 xpc setserial[862]: /dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A
kesä 10 22:29:46 xpc systemd[1]: Started controls configuration of serial ...s.
Hint: Some lines were ellipsized, use -l to show in full.
kex@xpc:~$ sudo systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
kex@xpc:~$ rpcinfo
   program version netid address service owner
    100000 4 tcp6 ::.0.111 portmapper superuser
    100000 3 tcp6 ::.0.111 portmapper superuser
    100000 4 udp6 ::.0.111 portmapper superuser
    100000 3 udp6 ::.0.111 portmapper superuser
    100000 4 tcp 0.0.0.0.0.111 portmapper superuser
    100000 3 tcp 0.0.0...

I wouldn't call this v-failed. The setserial fix might not be sufficient for your particular system, but it's necessary. As that cycle is gone for you, I mark this as v-done.

etc/init.d/later-readahead comes from the "readahead-fedora" package. I suggest to not install that -- it's not supported, we already have "readahead", and it causes that trouble for you. Please open a bug about that, but I'd rather just remove it from wily entirely.

I can confirm that the proposed package fixes the bug for me.

I did not have readahead-fedora package, but ureadahead. However, I had fedora-readahead with 14.04. It is possible that some files were not removed during upgrade to 15.04. apt-get remove ureadahead; apt-get purge readahead-fedora fixed this for me (in addition to using the proposed setserial). Now nfsd starts.

This bug was fixed in the package setserial - 2.17-48ubuntu0.1

---------------
setserial (2.17-48ubuntu0.1) vivid-proposed; urgency=medium

  * Add debian/setserial.service systemd unit, mostly to relax the very
    demanding Before=sysinit.target plus After=remote-fs.target combination
    resulting from the init.d script's $remote_fs dependency. This causes an
    insatisfiable dependency loop when using NFS and NetworkManager.
    (LP: #1452644)
  * Add dh-systemd build dependency and call dh_systemd_enable in
    debian/rules.

 -- Martin Pitt <email address hidden> Wed, 27 May 2015 11:57:47 +0200

The verification of the Stable Release Update for setserial has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.