nfs-kernel-server does not start because of dependency failure

Status changed to 'Confirmed' because the bug affects multiple users.

the workaround described earlier here does not seem to work.

starting with a clean install, a simple export, and no changes in /etc/default/nfs-*

the first error message in the journal was " rpc.nfsd: invalid option -- 'v' "

this worked:
> systemctl add-wants multi-user.target rpcbind.service
> systemctl restart nfs-kernel-server

then add the restart to /etc/rc.local and the system will reboot with nfs services running - otherwise it will not start, instead showing "Dependency failed for NFS server and services" in the journal

system updated and problem returned in 4.4.0-31

fixed by removing $RPCNFSDARGS from line 29 in

/lib/systemd/system/nfs-server.service

replacing with debug flag, this works through reboot without any restart required in rc.local

ExecStart=/usr/sbin/rpc.nfsd -d

RPCNFSDARGS was not defined in /etc/default/nfs-kernel-server
and it wasn't read from there when set - defining it as "-d" did not enable the option or allow the service to start.

Hi,
this seems still an issue in ubuntu 16.04

ii nfs-common 1:1.2.8-9ubuntu12 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.8-9ubuntu12 amd64 support for NFS kernel server

systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Aug 29 07:26:17 f42252s3 systemd[1]: Dependency failed for NFS server and services.
Aug 29 07:26:17 f42252s3 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

Any help to solve it?

It appears that nfs-mountd.service is missing dependencies:

Wants=rpcbind.service
After=rpcbind.service

(as hinted above in comment 3, but we need After= too).

I lost the system disk on my server and re-installed from scratch. I did a apt update/upgrade before installing nfs-kernel-server and now everything seems OK.

root@server:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2016-09-03 11:59:27 BST; 5min ago
  Process: 3324 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 3317 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 3324 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/nfs-server.service

Sep 03 11:59:27 server systemd[1]: Starting NFS server and services...
Sep 03 11:59:27 server systemd[1]: Started NFS server and services.

root@server:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: active (running) since Sat 2016-09-03 11:59:27 BST; 3min 5s ago
  Process: 3306 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
 Main PID: 3315 (rpc.mountd)
    Tasks: 1
   Memory: 3.3M
      CPU: 8ms
   CGroup: /system.slice/nfs-mountd.service
           └─3315 /usr/sbin/rpc.mountd --manage-gids

Sep 03 11:59:27 server systemd[1]: Starting NFS Mount Daemon...
Sep 03 11:59:27 server rpc.mountd[3315]: Version 1.2.8 starting
Sep 03 11:59:27 server systemd[1]: Started NFS Mount Daemon.

root@server:~# lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

and I can access the share from my laptop successfully.

It appears that 16.04.1 has fixed this bug

I've recently upgraded from Ubuntu Server 14.04 to 16.04.1 and I am experiencing this issue right now.

I tried to purge and reinstall nfs-kernel-server, but the issue came back at once. I was only able to work around this issue by adding

systemctl restart nfs-kernel-server

in /etc/rc.local.

So, from my perspective the issue might be gone for a clean install of Ubuntu 16.04.1, but not for systems having been upgraded from an older release.

I confirm that this bug also affects my Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-51-lowlatency i686) system (with all updates as of today, even those from xenial-proposed).

However, in my case the failure (rpc.mountd[<number>]: mountd: could not create listeners) occured only in about 50% of all boot processes and the system booted without this failure elsewise.

Using journalctl I could narrow the error down: It is a race condition: The error occurs when rpcbind is not yet ready when rpc.mountd is started and it does not occur when rpcbind is already ready at the time rpc.mountd starts.

So Malcolm Scott gives the correct solution (add dependency and order information to systemd) in comment #6. I created the snippet /etc/systemd/system/nfs-mountd.service.d/rpcbind-dep.conf:

[Unit]
Wants=rpcbind.service
After=rpcbind.service

and updated the initramfs (update-initramfs -u) and now the bug has gone. For a clean bug fix the above two lines should be added to /lib/systemd/system/nfs-mountd.service by the maintainers of the nfs-kernel-server package.

Addendum: Link to Radhat's "Storage Administration Guide" which states that "Because RPC-based services rely on rpcbind to make all connections with incoming client requests, rpcbind must be available BEFORE any of these services start."

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/s2-nfs-methodology-portmap.html

Why has noone updated the package by adding the following to /lib/systemd/system/nfs.mountd.service:
Wants=rpcbind.service
After=rpcbind.service

I confirmed that this solves the issue.

So why isn't the package updated yet?

Same here! Running 16.04.1 (with all packages updated) and nfs still fails to start after reboot and returns the familiar dependency error message.

I've the workaround suggest by Thomas Runge with great success. So thanks for that Thomas!

Strangely, this bug is open for more than 7 months now... when is it going to be fixed in the core?

we had some hacks running so the service starts when the system comes up, and a recent update to the latest 16.04 (4.4.0-57-generic #78) broke them.

current workarounds, placed in /etc/rc.local :

##
systemctl add-wants multi-user.target rpcbind.service
service nfs-kernel-server restart
systemctl restart rpcbind
##

---------------------------------------------------

# run the commands manually to enable the services, it may be necessary to issue 'systemctl daemon-reload' after adding the target.

# reboot to check that the service starts. some of these may not be needed every startup but are there to recover the config after an upgrade breaks the workaround.

# check
service nfs-kernel-service status

# for debugging:

# set debug flags
rpcdebug -m rpc -s all
rpcdebug -m nfsd -s all

# tail the logfile
tail -f /var/log/kern.log

# clear debug flags:
rpcdebug -m rpc -c all
rpcdebug -m nfsd -c all

Okay, sorry for the delay. It took me sometime to review all SysV, Upstart scripts and Systemd unit files. What "linuxball" said is correct, we're missing rpcbind dependency for nfs-mountd.service.

# nfs-server.service

[Unit]
...
Requires= network.target proc-fs-nfsd.mount rpcbind.target
Requires= nfs-mountd.service
Wants= nfs-idmapd.service

After= local-fs.target
After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

Wants=auth-rpcgss-module.service
After=rpc-gssd.service rpc-svcgssd.service

Before=remote-fs-pre.target

Wants=nfs-config.service
After=nfs-config.service

As it stands, "rpcbind.target is being put as a dependency and to run before "nfs-server.service" but not before "nfs-mountd.service":

[Unit]
...
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

Since the services run in parallel there is an window of opportunity for "nfs-mountd.service" to be started before "rpcbind.target".

Only problem is that rpcbind.target is for synchronization with SysV scripts:

rpcbind.target
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.

Right now we have:

system/nfs-server.service:Requires= ... rpcbind.target
system/nfs-server.service:After= ... rpcbind.target ...
system/rpc-statd.service:Requires= ... rpcbind.target
system/rpc-statd.service:After= ... rpcbind.target

So the correct fix would be:

Wants= ... rpcbind.target
After= ... rpcbind.target

For all NFS daemons depending on rpcbind to be fully operational, since the systemd generators would take also care of SysV script dependencies (allegedly if rpcbind is started through SysV generator).

The thing is that, this upstream discussion has taken place:

https://patchwork.kernel.org/patch/8976701/

I think I should move on and "cherry-pick" this change as well. Upstream "Linux NFS" changed the dependency from "target" to "service" - which makes perfectly sense based on the explanation from Neil Brown.

I'm providing a PPA soon and will ask for feedback and, at the same time, I'll attach this small fix as a debdiff proposal for sponsors team to upload it to -propose if your feedback is good. I'd like to ask you to reproduce the issue and to make sure issue is gone with the package I'll provide.

Sorry for the amount of time this bug was opened without a fix. I think it never gained much traction because the workaround as simple so people had stick with the workaround you guys provided.
Jeroen, if you want to understand how Ubuntu development works, specifically for fixes you can read this Wiki: https://wiki.ubuntu.com/StableReleaseUpdates.

Will get back to this today.

Just one thing that I'm trying to figure out:

nfs-mountd.service already has:

[Unit]
...
BindsTo=nfs-server.service

Which means, it "requires" nfs-server.service AND it is stopped as soon nfs-server.service is stopped. Judging by this upstream commit: http://paste.ubuntu.com/23940610/

"""
So while I think it is valid to use rpcbind.target for ordering
(before/after) it shouldn't be used for dependencies (Wants/Requires).
The rpcbind.target file included in systemd does not "Require" the
actual service, so requiring rpcbind.target itself is pointless.
"""

This race condition might be happening because of the ".target" choice for rpcbind dependency - on nfs-server - instead of the "lack" of dependency in nfs-mountd. My only doubt is that developer preferred to depend on "rpcbind.socket" instead of "rpcbind.service". I'll go ahead and give it a try by testing ".socket", based on upstream commits, and ask for you to test it.

Thank you
Rafael

Could anyone able to reproduce this issue check if the following PPA:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799

Solves the problem ?

Thank you
Rafael

I'm attaching the debdiffs just "in case". After I get confirmation that the packages from PPA solves the issue I'll go on and ask for sponsorship.

Thanks, Rafael -- I'll put together an environment in which to test the packages (I'd rather not reboot my live NFS servers).

However, I think nfs-mountd.service still needs tweaking. rpc.mountd (if using NFSv2 or v3) needs to communicate with rpcbind on startup to register services. Currently nfs-mountd.service has

   BindsTo=nfs-server.service

but as far as I can tell this does NOT imply an ordering (it's Requires= plus the additional behaviour of stopping when another service stops; I don't think it covers After=).

So, I suspect you may also need to add to nfs-mountd.service:

   After=nfs-server.service

Furthermore, arguably every RPC service (rpc-statd.service, nfs-mountd.service, etc.) should explicitly have

   BindsTo=rpcbind.service
   After=rpcbind.service

because they literally are bound to rpcbind. If rpcbind is stopped/restarted, every RPC service must also be stopped/restarted so as to rebind (as stated in rpcbind(8)). Depending on rpcbind.socket seems wrong when rpcbind must itself be running for any RPC service to start serving requests.

Or perhaps just put that in nfs-server.service and have everything transitively bind to rpcbind via that unit.

I had this issue when running under an LXD/LXC container. If you need I
can deploy it out and test. Just let me know.

On Mon, Feb 6, 2017 at 1:00 PM, Rafael David Tinoco <
<email address hidden>> wrote:

> ** Patch added: "yakkety_nfs-utils_1.2.8-9.2ubuntu2.debdiff"
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+attachment/4814234/+files/yakkety_nfs-utils_1.2.8-
> 9.2ubuntu2.debdiff
>
> ** Changed in: nfs-utils (Ubuntu Xenial)
> Importance: Undecided => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1590799
>
> Title:
> nfs-kernel-server does not start because of dependency failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+subscriptions
>

Malcolm,

Your first comment makes sense: I really don't know if order will be guaranteed without After= for when we're using BindsTo :\. I think it might be safer for us to use it since it will cause no harm having it (i'll have to include this upstream somehow so we don't break our future releases).

NOW, I DO agree with your statement on rpcbind. I thought it was weird creating dependency on the socket itself. I think the idea was to make services startup "faster", the developer "assumed" the socket being ready would make that happen. For rpcbind it doesn't make much sense since it will take care of all port orchestration for the rest of services and we have to have the service up and running, not only its socket.

I'll make those changes and will update PPA. For now you can keep with this version - since it will, at least, remove the ".target" logic. I'll do it tonight.

Tks Malcom, please be sure to reproduce it before checking if it works (so we can "verify" it later, when the SRU is done).

o/

The attachment "xenial_nfs-utils_1.2.8-9ubuntu13.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Summarizing

From systemd.special we have:

sockets.target:
A special target unit that sets up all socket units (see systemd.socket(5) for details) that shall be active after boot. Services that can be socket-activated shall add Wants= dependencies to this unit for their socket unit during installation. This is best configured via a WantedBy=sockets.target in the socket unit's "[Install]" section.

Lets see who is starting rpcbind.socket:

[Install]
WantedBy=sockets.target

Unit file will be linked as "wanted" by "sockets.target".

$ systemctl list-dependencies sockets.target
sockets.target
● ├─acpid.socket
● ├─apport-forward.socket
● ├─dbus.socket
● ├─dm-event.socket
● ├─rpcbind.socket
... <many others>

$ systemctl --reverse list-dependencies sockets.target
sockets.target
● └─basic.target
● └─multi-user.target
● └─graphical.target

basic.target starts the sockets.target.
sockets.target start rpcbind.socket.
rpcbind.target doesn't start rpcbind.service.

Lets who starts rpcbind.service:

[Install]
Also=rpcbind.socket

rpcbind.{socket,service} are enabled/disabled together (not started).

$ systemctl list-dependencies rpcbind.service
rpcbind.service
● ├─rpcbind.socket
● ├─system.slice
● ├─remote-fs-pre.target
● └─rpcbind.target
    └─...

rpcbind.service depends on rpcbind.target (not with my patch) AND rpcbind.socket (started by basic.target with the sockets.target). Lets see rpcbind.target (its a special target made by a generator, its unit file doesn't contain anything):

rpcbind.target:
The portmapper/rpcbind pulls in this target and orders itself before it, to indicate its availability. systemd automatically adds dependencies of type After= for this target unit to all SysV init script service units with an LSB header referring to the "$portmap" facility.

So, systemd puts an After=rpcbind.target in all SysV scripts with LSB header containing "$portmap" as facility (insserv style):

$ systemctl status rpcbind.target
● rpcbind.target - RPC Port Mapper
   Loaded: loaded (/etc/insserv.conf.d/rpcbind; static; vendor preset: enabled)
  Drop-In: /run/systemd/generator/rpcbind.target.d
           └─50-hard-dependency-rpcbind-$portmap.conf

/etc/insserv.conf.d/rpcbind contains "rpcbind".
/run/systemd/generator/rpcbind.service.d/50-rpcbind-\$portmap.conf contains:

[Unit]
Wants=rpcbind.target
Before=rpcbind.target

So rpcbind.target doesn't start "rpcbind.service" and any "rpcbind.target" usage seems broken to me. rpcbind was ever started by nfs-{kernel-server,mountd}.service (since they were relying on the rpcbind.target). rpcbind (the portmap) was being activated by a connection to the rpc socket.

But NOT the NETWORK socket, the LOCAL socket:

[Socket]
ListenStream=/run/rpcbind.sock

You can test this by stopping the "rpcbind.service" and trying:

$ rpcinfo -T tcp 127.0.0.1
rpcinfo: can't contact rpcbind: RPC: Remote system error - Connection refused

$ NETPATH=/run/rpcbind.sock rpcinfo
   program version netid address service owner
    100000 4 tcp6 ::.0.111 portmapper superuser
    100000 3 tcp6 ::.0.111 portmapper superuser
    100000 4 udp6 ::.0.11...

Spyderdyne,

Tks for letting me know about the container situation. I this it is easier to reproduce it there. With this tip I was able to reproduce here at will. I first started getting this in a Xenial machine:

$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Feb 07 01:46:49 nfsstartxen systemd[1]: Dependency failed for NFS server and services.
Feb 07 01:46:49 nfsstartxen systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

AND (after changing rpcbind.target to rpcbind.socket in the unit files):

$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/etc/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-02-07 01:55:39 UTC; 39s ago
  Process: 244 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=1/FAILURE)

Feb 07 01:55:39 nfsstartxen systemd[1]: Starting NFS Mount Daemon...
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Control process exited, code=exited status=1
Feb 07 01:55:39 nfsstartxen systemd[1]: Failed to start NFS Mount Daemon.
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Unit entered failed state.
Feb 07 01:55:39 nfsstartxen systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

This is good because I expected the error to continue after my last comment - since enabling the socket is EXACTLY what I showed we had before.

Malcolm,

You were right, for the "nfs-mountd.service" I HAD TO have:

Wants=nfs-config.service rpcbind.service
After=nfs-config.service rpcbind.service

because:

BindsTo=nfs-server.service in "nfs-mountd.service"

Wasn't enough to guarantee the order if "nfs-server.service" contains:

Wants=rpcbind.service
After= rpc-statd.service

Looks like we have to force the order and I'll have to report this upstream pointing out this bug, together with this fix.

Will get back here tomorrow.

Just submitted a new fix into:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799

It will be ready soon.
It fixes the issue for me (since I was able to reproduce it).

Based on the SRU guidelines: https://wiki.ubuntu.com/StableReleaseUpdates

I'll first check with upstream possibility to get this fixed there and backport it appropriately.

Was just hit by this bug on two production servers. Is it possible to apply a fix manually? For some reason I don't see a diff in https://launchpad.net/~inaddy/+archive/ubuntu/lp1590799 .

I have replaced rpcbind.target with rpcbind.service in Requires and After lines of /etc/systemd/system/multi-user.target.wants/nfs-server.service , will this be sufficient? I don't have any other nfs-*.service files, including nfs-mountd.service, in my /etc directory.

Marat: http://paste.ubuntu.com/23948501/ this is what you want.

Using ".socket" instead of ".service" is okay. Seems that rpcbind is accessed from local socket, controlled by systemd. This is what I'll propose upstream:

http://paste.ubuntu.com/23949146/

Proposed upstream fix:

http://marc.info/?l=linux-nfs&m=148649501611915&w=2

Waiting for upstream a bit before moving on with Xenial and Yakkety proposals.

Rafael David Tinoco (inaddy): thank you very much for the diff! Have applied it to live nfs-mountd.service, nfs-server.service, and rpc-statd.service in /lib/systemd/system, will report result of the next reboot here if it happens before official fix arrives for Xenial.

Commit got into master branch upstream:

commit 907426b00bdcd69d9a56ac1870990e8ae8c6fe9f
Author: Rafael David Tinoco <email address hidden>
Date: Wed Feb 15 10:26:55 2017 -0500

    systemd: Fix nfs-mountd dependency on rpcbind

    Following commit 91da135f - it replaced "rpcbind.target" by "rpcbind.socket" in
    some unit files - "rpcbind.socket" should also be added to "nfs-mountd.service"
    as a dependency to avoid race conditions.

    Usually "rpcbind.socket" is either started as a "sockets.target" dependency, or
    as a dependency for "nfs-server.service", when unit files include it in
    "BindsTo" or "After". Unfortunately there is a possilibility to have
    "nfs-mountd.service" started when the rpcbind socket is not yet created:

    systemd[1]: Starting NFS Mount Daemon...
    systemd[1]: nfs-mountd.service: Control process exited, code=exited status=1
    systemd[1]: Failed to start NFS Mount Daemon.
    systemd[1]: nfs-mountd.service: Unit entered failed state.
    systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

    Nowadays "nfs-mountd.service" uses "BindTo" directive to "nfs-server.service".
    That, per se, doesn't guarantee ordering for NFS server to start rpcbind and for
    nfs-mountd to depend on it.

    https://bugs.launchpad.net/bugs/1590799

    Reviewed-by: NeilBrown <email address hidden>
    Signed-off-by: Rafael David Tinoco <email address hidden>
    Signed-off-by: Steve Dickson <email address hidden>

Now I can do the SRU and provide the fix proposal.

Opened Debian Bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=856328

Waiting for sponsorship.

Can confirm that it booted successfully (once) with patch live-applied in a system that earlier experienced this bug.

This bug was fixed in the package nfs-utils - 1:1.2.8-9.2ubuntu2

---------------
nfs-utils (1:1.2.8-9.2ubuntu2) zesty; urgency=medium

  * Fixing nfs-mountd dependency on rpcbind (race condition) (LP: #1590799)
    by adding "rpcbind.socket" to "nfs-mountd.service" as a dependency
    to avoid race conditions:
    - Add systemd-Fix-nfs-mountd-dependency-on-rpcbind.patch
    - Add systemd-unit-files-fix-up-dependencies-on-rpcbind.patch

 -- Rafael David Tinoco <email address hidden> Fri, 17 Mar 2017 12:19:53 +0100

Adding a Trusty task & marking it invalid so we know that it doesn't concern Trusty (no systemd there)

Updated debdiff with version change ( .13 collided with a prevously uploaded pkg)

Hello Paul, or anyone else affected,

Accepted nfs-utils into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-9.2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Paul, or anyone else affected,

Accepted nfs-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-9ubuntu12.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Brian

I followed the links you gave for testing this (my first time)

Repeating the commands I originally did in the bug report after applying the proposed packaged shows:

 root@coulomb:~# dmesg | grep nfs
[ 3.131507] systemd[1]: nfs-blkmap.service: Job nfs-blkmap.service/start failed with result 'dependency'.
[ 3.189403] Installing knfsd (copyright (C) 1996 <email address hidden>).
[ 5.115023] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
root@coulomb:~# systemctl status nfs-kernel-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor prese
   Active: active (exited) since Sun 2017-04-02 08:58:33 BST; 4min 38s ago
  Process: 929 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/
  Process: 928 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS
 Main PID: 929 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Apr 02 08:58:33 coulomb systemd[1]: Starting NFS server and services...
Apr 02 08:58:33 coulomb systemd[1]: Started NFS server and services.
root@coulomb:~# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset
   Active: active (running) since Sun 2017-04-02 08:58:33 BST; 4min 55s ago
  Process: 921 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, statu
 Main PID: 927 (rpc.mountd)
   CGroup: /system.slice/nfs-mountd.service
           └─927 /usr/sbin/rpc.mountd --manage-gids

Apr 02 08:58:33 coulomb systemd[1]: Starting NFS Mount Daemon...
Apr 02 08:58:33 coulomb rpc.mountd[927]: Version 1.2.8 starting
Apr 02 08:58:33 coulomb systemd[1]: Started NFS Mount Daemon.
root@coulomb:~# systemctl list-dependencies nfs-kernel-server
nfs-kernel-server.service
● ├─auth-rpcgss-module.service
● ├─nfs-config.service
● ├─nfs-idmapd.service
● ├─nfs-mountd.service
● ├─proc-fs-nfsd.mount
● ├─rpc-svcgssd.service
● ├─rpcbind.socket
● ├─system.slice
● └─network.target
root@coulomb:~# systemctl list-dependencies nfs-mountd.service
nfs-mountd.service
● ├─nfs-config.service
● ├─nfs-server.service
● ├─proc-fs-nfsd.mount
● └─system.slice
root@coulomb:~# lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04
root@coulomb:~# apt-cache policy nfs-kernel-server
nfs-kernel-server:
  Installed: 1:1.2.8-9ubuntu12.1
  Candidate: 1:1.2.8-9ubuntu12.1
  Version table:
 *** 1:1.2.8-9ubuntu12.1 400
        400 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.8-9ubuntu12 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

which shows the bug as fixed.

This is for Xenial.

I just noticed that the nfs.blkmap service now hows a failed dependency.

I have set tag to verification-done. Bit worried abouth the nfs.blkmap failed dependency shown in dmesg output but I assume that this is a separate issue.

I have tested both and found no apparent race condition. Since I'm the second one, I will flag this as verification-done. Paul, the nfs.blkmap is not in even in the same systemd unit files we are working on.

nfs.blkmap is declared for me in:

nfs-blkmap.service:Requisite=nfs-blkmap.target
nfs-blkmap.service:After=nfs-blkmap.target
nfs-blkmap.service:PIDFile=/var/run/blkmapd.pid
nfs-blkmap.service:ExecStart=/usr/sbin/blkmapd $BLKMAPDARGS

nfs-blkmap.target:Description= PNFS blkmaping enablement.
nfs-blkmap.target:# If this target is enabled, then blkmapd will be started

nfs-client.target:Wants=nfs-blkmap.service
nfs-client.target:After=nfs-blkmap.service

being responsible for blkmapd daemon (for pNFS).

blkmapd can be (re)started through nfs-utils.service if blkmapd.target is enabled. It can also be started by nfs-client.target, if blkmapd.target is enabled.

[ 3.131507] systemd[1]: nfs-blkmap.service: Job nfs-blkmap.service/start failed with result 'dependency'.

This looks like a leftover from some other tasks/cmds you made. If you think its needed, this should be handled in another bug (specially because its unrelated to the race itself AND this bug is already in verification phase).

I'm marking this as verification done.

Johan why are you changing status for this bug ? It has not been released yet. The status is changed automatically when the package moves from -proposed to -updates.

I'm sorry, my mistake,clicked on the wrong page.

I see confirmation of 1:1.2.8-9ubuntu12.1 from xenial-proposed, but I don't see the version from yakkety-proposed confirmed anywhere. So I'll release Xenial but defer releasing Yakkety. Please can someone test the package from yakkety-proposed and confirm what you tested and the package version you tested in a comment?

This bug was fixed in the package nfs-utils - 1:1.2.8-9ubuntu12.1

---------------
nfs-utils (1:1.2.8-9ubuntu12.1) xenial; urgency=medium

  * Fixing nfs-mountd dependency on rpcbind (race condition) (LP: #1590799)
    by adding "rpcbind.socket" to "nfs-mountd.service" as a dependency
    to avoid race conditions:
    - Add systemd-Fix-nfs-mountd-dependency-on-rpcbind.patch
    - Add systemd-unit-files-fix-up-dependencies-on-rpcbind.patch

 -- Rafael David Tinoco <email address hidden> Wed, 29 Mar 2017 10:50:52 +0200

Hello Robie,

From comment #48: "I have tested both and found no apparent race condition". Sorry for not making myself clear, both here means Yakkety and Xenial. I'm not facing the startup race condition in Xenial (as I was) AND I never faced in Yakkety (likely because the difference of services being started and intermittent behavior for this race). So my verification was to guarantee that the services would come up appropriately on both for a reasonable amount of times (several reboots).

Thank you for publishing it.

Best,
Rafael

I'm still seeing this issue while the package is installed.

~$ apt-cache policy nfs-kernel-server
nfs-kernel-server:
  Installed: 1:1.2.8-9ubuntu12.1
  Candidate: 1:1.2.8-9ubuntu12.1
  Version table:
 *** 1:1.2.8-9ubuntu12.1 500
        500 http://nl3.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.8-9ubuntu12 500
        500 http://nl3.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

~$ cat /lib/systemd/system/nfs-mountd.service
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
After=rpcbind.socket
BindsTo=nfs-server.service

Wants=nfs-config.service
After=nfs-config.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS

after updating (a copy of) a working system to the latest release (1:1.2.8-9ubuntu12.1), nfs-kernel-service was in a broken state with the same errors.

further testing is needed.

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

Thanks!

On Tue, Jun 27, 2017 at 11:24 AM, Łukasz Zemczak <<email address hidden>
> wrote:

> As part of a recent change in the Stable Release Update verification
> policy we would like to inform that for a bug to be considered verified
> for a given release a verification-done-$RELEASE tag needs to be added
> to the bug where $RELEASE is the name of the series the package that was
> tested (e.g. verification-done-xenial). Please note that the global
> 'verification-done' tag can no longer be used for this purpose.
>
> Thank you!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1590799
>
> Title:
> nfs-kernel-server does not start because of dependency failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+subscriptions
>

Hello everyone,

Kuba,

Could you please show more details on what is happening in your case ? Could you follow what I did in my previous comments, to show more logs from the nfs service(s) failure(s) ? Could you reproduce the issue - the startup problem - and show me a "systemctl status" on every possible nfs {service,target} so I can relate them ? There is a chance that what you're facing isn't, necessarily, the same problem.

Best,
Rafael

Hi Rafael,

Below requested output, seems both services are dead and there are no log indication on what happened:

kuba@kubadesktop:~$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
kuba@kubadesktop:~$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: inactive (dead)

I've also checked dmesg for any hints, seems my problem is also related to cycle dependency, but it seems it's not related to this bug.

[ 3.460873] systemd[1]: remote-fs-pre.target: Found ordering cycle on remote-fs-pre.target/start
[ 3.460877] systemd[1]: remote-fs-pre.target: Found dependency on nfs-server.service/start
[ 3.460879] systemd[1]: remote-fs-pre.target: Found dependency on network.target/start
[ 3.460881] systemd[1]: remote-fs-pre.target: Found dependency on NetworkManager.service/start
[ 3.460883] systemd[1]: remote-fs-pre.target: Found dependency on dbus.service/start
[ 3.460885] systemd[1]: remote-fs-pre.target: Found dependency on basic.target/start
[ 3.460887] systemd[1]: remote-fs-pre.target: Found dependency on sockets.target/start
[ 3.460889] systemd[1]: remote-fs-pre.target: Found dependency on acpid.socket/start
[ 3.460891] systemd[1]: remote-fs-pre.target: Found dependency on sysinit.target/start
[ 3.460893] systemd[1]: remote-fs-pre.target: Found dependency on setkey.service/start
[ 3.460895] systemd[1]: remote-fs-pre.target: Found dependency on remote-fs.target/start
[ 3.460896] systemd[1]: remote-fs-pre.target: Found dependency on remote-fs-pre.target/start
[ 3.460899] systemd[1]: remote-fs-pre.target: Breaking ordering cycle by deleting job nfs-server.service/start
[ 3.460901] systemd[1]: nfs-server.service: Job nfs-server.service/start deleted to break ordering cycle starting with remote-fs-pre.target/start

I had been experiencing the same issue, as described in comment #5. I am using Ubuntu 16.04.2 4.4.0-83, nfs-kernel-server 1.2.8-9. The fix described in comments #6 and 9 along did not work but issuing

systemctl add-wants multi-user.target rpcbind.service
service nfs-kernel-server restart

as suggested in comment #13 solved the issue.

Similar situation as reported above. This time however dmesg shows that there is a ordering cycle on nfs-server.service (depends on nfs-mountd.service)

kuba@kubadesktop:~$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
kuba@kubadesktop:~$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: inactive (dead)

Another output from dmesg related to nfs:
[ 3.366015] systemd[1]: nfs-server.service: Found ordering cycle on nfs-server.service/start
[ 3.366018] systemd[1]: nfs-server.service: Found dependency on nfs-mountd.service/start
[ 3.366021] systemd[1]: nfs-server.service: Found dependency on network.target/start
[ 3.366023] systemd[1]: nfs-server.service: Found dependency on NetworkManager.service/start
[ 3.366025] systemd[1]: nfs-server.service: Found dependency on basic.target/start
[ 3.366027] systemd[1]: nfs-server.service: Found dependency on sockets.target/start
[ 3.366029] systemd[1]: nfs-server.service: Found dependency on cups.socket/start
[ 3.366031] systemd[1]: nfs-server.service: Found dependency on sysinit.target/start
[ 3.366033] systemd[1]: nfs-server.service: Found dependency on setkey.service/start
[ 3.366035] systemd[1]: nfs-server.service: Found dependency on remote-fs.target/start
[ 3.366037] systemd[1]: nfs-server.service: Found dependency on remote-fs-pre.target/start
[ 3.366038] systemd[1]: nfs-server.service: Found dependency on nfs-server.service/start
[ 3.366040] systemd[1]: nfs-server.service: Breaking ordering cycle by deleting job nfs-mountd.service/start
[ 3.366043] systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start deleted to break ordering cycle starting with nfs-server.service/start

Kuba,

There is some on going discussions regarding how systemd generator uses existing sysv init scripts and creates on-the-fly unit files for services (and dependencies) and how it also uses existing systemd unit files. I believe that there might be a conflict on what you're experiencing there (for the services and its dependencies). Disabling the systemd services, removing the sysv init files from /etc/init.d and re-enabling the services will guarantee that services are its dependencies are started like alleged in systemd unit file (the one that should be functional and supported for Xenial, for example).

If you, or anyone, are still facing this, could you please give it a try ?

My system has been affected since yesterday.

I don't know which update has broken my system. I did an apt-get update && apt-get dist-upgrade by remote. Thats was it.

NFS broken!
NUT broken!
SSH broken!
something else ... I will see.

:-((((

I only hit this when running in a LXD server container. What he is seeing
is new.

On Oct 27, 2017 6:35 PM, "Rafael David Tinoco" <email address hidden>
wrote:

> Kuba,
>
> There is some on going discussions regarding how systemd generator uses
> existing sysv init scripts and creates on-the-fly unit files for
> services (and dependencies) and how it also uses existing systemd unit
> files. I believe that there might be a conflict on what you're
> experiencing there (for the services and its dependencies). Disabling
> the systemd services, removing the sysv init files from /etc/init.d and
> re-enabling the services will guarantee that services are its
> dependencies are started like alleged in systemd unit file (the one that
> should be functional and supported for Xenial, for example).
>
> If you, or anyone, are still facing this, could you please give it a try
> ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1590799
>
> Title:
> nfs-kernel-server does not start because of dependency failure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/
> 1590799/+subscriptions
>

Was just impacted by this, following #62 fixed this issue for me.

Thanks Rafael!

Still impacted by this. Now #62 is not working. I see that a fix is available for xenial, but I have no upgrades available.

This is a problem, as this completely hoses our vmware network that relies on NFS exports.

Does anyone know what version of which packages broke this? Until this is fixed we need a return to a older version where this is not an issue.

Yakkety is now unsupported, and the other supported releases are fixed.

Probably due to unrelated problem nfsd did not start for me to yesterday too. I've experienced the original bug this thread is about and I've had it fixed for some time. Should we create another bug or can we track it here?

Hello people:

I am using XUbuntu 18.04 and when trying to install the nfs-kernel-server the system displays an error message similar to the title of the report issue of this bug.

root@srvdata1804x64:~# apt install nfs-kernel-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  nfs-common
Suggested packages:
  open-iscsi watchdog
The following NEW packages will be installed:
  nfs-common nfs-kernel-server
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 299 kB of archives.
After this operation, 1110 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 nfs-common amd64 1:1.3.4-2.1ubuntu5 [205 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 nfs-kernel-server amd64 1:1.3.4-2.1ubuntu5 [94.0 kB]
Fetched 299 kB in 0s (12.9 MB/s)
Selecting previously unselected package nfs-common.
(Reading database ... 47730 files and directories currently installed.)
Preparing to unpack .../nfs-common_1%3a1.3.4-2.1ubuntu5_amd64.deb ...
Unpacking nfs-common (1:1.3.4-2.1ubuntu5) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.3.4-2.1ubuntu5_amd64.deb ...
Unpacking nfs-kernel-server (1:1.3.4-2.1ubuntu5) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up nfs-common (1:1.3.4-2.1ubuntu5) ...

Creating config file /etc/idmapd.conf with new version
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target -> /lib/systemd/system/nfs-client.target.
Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target -> /lib/systemd/system/nfs-client.target.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.3.4-2.1ubuntu5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service -> /lib/systemd/system/nfs-server.service.
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript nfs-kernel-server, action "start" failed.
* nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Feb 25 13:37:23 srvdata1804x64 systemd[1]: Dependency failed for NFS server and services.
Feb 25 13:37:23 srvdata1804x64 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Feb 25 13:40:16 srvdata1804x64 systemd[1]: Dependency failed for NFS server and services.
Feb 25 13:40:16 srvdata1804x64 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Feb 25 13:53:31 srvdata1804x64 systemd[1]: Dependency failed for NFS server and services.
Feb 25 13:53:31 srvdata1804x64 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'....

Still an issue for me as well just as described above on a new server install (Minimal Ubuntu 18.04.2 LTS) If Rafael David Tinoco's post at #62 is the sanctioned work-around, can someone provide more details since I don't appear to be able to make it work.

This issue just popped up for me after upgrading from 14.04 to 16.04.6.

Per comments #20 and #25, I changed nfs-mountd.service to include rpcbind.service

This issue just popped up for me after upgrading from 14.04 to 16.04.6.

Per comments #20 and #25, I changed nfs-mountd.service to include rpcbind.service to the Wants and After statements:

Wants=nfs-config.service rpcbind.service
After=nfs-config.service rpcbind.service

I also added BindsTo and Wants statements to nfs-server.service:

BindsTo=rpcbind.service
After=rpcbind.service

On a reboot, nfs-server service ran as expected.

This still doesn't solve it for me on a fresh minimal 18.04 install in an openstack cloudimage. I'm probably not getting things into the right places and hence the problem. Is the location of these lines:

> BindsTo=rpcbind.service
> After=rpcbind.service

important inside the nfs-server.service file?
(It does work with an image from a CentOS distribution but I'd really like this to work on ubuntu.)

As it turns out, I'd not subscribed to this bug so I didn't get comments after I posted mine. I had to refer back to my own comment to figure out the solution since my NFS server apparently broke earlier this week.

An update to nfs-common (I think that's the package that installs the service files) was pushed out and overwrote the "fix" I'd put in earlier.

A more complete fix (after some trial and error) was to create the "override.conf" files for nfs-server and nfs-mountd using:

sudo systemctl edit nfs-server (same for nfs-mountd).

I copy/pasted the baseline service definitions into the override files and added my own tweaks. With that, the override files I am currently running with are as follows (output from systemctl cat nfs-server and nfs-mountd):

-------
# /etc/systemd/system/nfs-server.service.d/override.conf
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires= network.target proc-fs-nfsd.mount
Requires= nfs-mountd.service
Wants=rpcbind.socket
Wants=nfs-idmapd.service

After= local-fs.target
After= network.target proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
Before= rpc-statd-notify.service

# GSS services dependencies and ordering
Wants=auth-rpcgss-module.service
After=rpc-gssd.service rpc-svcgssd.service

# start/stop server before/after client
Before=remote-fs-pre.target

Wants=nfs-config.service
After=nfs-config.service

BindsTo=rpcbind.service
After=rpcbind.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils

ExecStart=
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/sbin/exportfs -r
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStop=/usr/sbin/rpc.nfsd 0
ExecStopPost=/usr/sbin/exportfs -au
ExecStopPost=/usr/sbin/exportfs -f

ExecReload=/usr/sbin/exportfs -r

[Install]
WantedBy=multi-user.target
------

Note the addition of:
   BindsTo=rpcbind.service
   After=rpcbind.service
   ExecStart=

------
# /etc/systemd/system/nfs-mountd.service.d/override.conf
[Unit]
Description=NFS Mount Daemon
DefaultDependencies=no
Requires=proc-fs-nfsd.mount
After=proc-fs-nfsd.mount
After=network.target local-fs.target
After=rpcbind.socket
BindsTo=nfs-server.service

Wants=nfs-config.service rpcbind.service
After=nfs-config.service rpcbind.service

[Service]
EnvironmentFile=-/run/sysconfig/nfs-utils
Type=forking
ExecStart=
ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS
------

Note the addition of "rpcbind.service" on the "Wants" and "After" lines, as well as the "ExecStart=".

It took me a bit to figure out the "ExecStart=" directives were required. As I found out, the ExecStart directive is additive, so the "ExecStart=" directive is required to "clear out" the directive set in the baseline service files. (In other words, the baseline service file is read before override.conf is read).

This appears to work for me on Ubuntu 16.04.06. I am not in a position to verify these tweaks work on later versions of Ubuntu.

Thanks Maarten for the detailed write up. I was able to follow it precisely but to no avail. I believe I must be suffering from other problems. What looks like the relevant section in the journalctl output looks like so:

<SNIP>
-- Subject: Unit proc-fs-nfsd.mount has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit proc-fs-nfsd.mount has begun starting up.
May 22 11:42:13 git systemd[1]: Starting Preprocess NFS configuration...
-- Subject: Unit nfs-config.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit nfs-config.service has begun starting up.
May 22 11:42:13 git mount[1524]: mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'.
May 22 11:42:13 git systemd[1]: proc-fs-nfsd.mount: Mount process exited, code=exited status=32
May 22 11:42:13 git systemd[1]: proc-fs-nfsd.mount: Failed with result 'exit-code'.
May 22 11:42:13 git systemd[1]: Failed to mount NFSD configuration filesystem.
-- Subject: Unit proc-fs-nfsd.mount has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit proc-fs-nfsd.mount has failed.
<SNIP>

So there is something problematic with the nfsd filetype since I also get this problem

~$ sudo mount -t nfsd nfsd /proc/fs/nfsd
mount: /proc/fs/nfsd: unknown filesystem type 'nfsd'.

Which somehow means that the nfsd filetype which requires some kernel module (Provided by: nfs-kernel-server_1.3.4-2.1ubuntu5_amd64 ???) isn't there.

OK, in the interest of full disclosure, here is a bit more about the root of my problems. I was trying to use this in a virtualized environment. It turns out that the kernel compiled didn't include the nfsd module which led to the above error of "unknown filesystem type 'nfsd'". A crafty colleague of mine was able to compile said module out of tree and I can now start the nfs kernel server. Thanks all for your help.

Hello,

Long time I don't follow this thread, sorry for that. So, as I could see Stefan had issues with nfsd because of missing kernel module, and that explains why he had the nfs-kernel-server start problems (dependency nfsd could not run, unrelated to the original race I reported upstream and backported as SRU for this).

On the other hand, Maarten, it was unclear to me if you had to apply the fix (https://pastebin.ubuntu.com/p/PwQfftj38j/) "by hand" because you could not upgrade the system, and when you finally could it overwrote what you had changed, so you came up with a workaround for not having it overwritten anymore. If that is correct, this is a good case for you but not for a SRU. If not, please let me know.

So, with those 2 cases, I'm still not considering this as "a new issue" unless someone can point me a dependency issue caused by a bad service dependency ordering (like the original issue was about).

Thank you very much for all your feedback.

Best,
Rafael

Hello Rafael,

In response to your question. I upgraded my server from Trusty to Xenial and then found out my NFS services weren't running right. After some digging I came across this this bug, and I applied the work-around manually, as I mentioned in my comment #73.

A week or two later I realized my NFS service once again stopped working, which then made me realize the changes I'd applied manually were no longer present. I assumed that was due to the files being replaced with updated ones when a package update was distributed.

At that point I figured out how to make my work-around survive, as documented in my comment #75.

I do not understand the intricacies of this issue, so the only thing I can say is that me applying these changes resolves the problem for me. I'd have to find some time to dig into this more to figure out why the baseline versions of the services don't work for me.

This has not been fixed, it still happens for me occasionally on 18.04 LTS, Bionic machines. And it keeps also dying away on load.

Just to confirm that it happens to me also on host 18.04 and raw lxd guest 18.04.

@gagarin, @rostislav,

If you can reproduce this at will, could you provide more information about it ?

Executing something like this:

$ for _service in $(systemctl list-dependencies nfs-kernel-server --plain | tail -n +2 | awk '{print $1}'); do systemctl cat --full --no-pager $_service > $_service.cat | journalctl _SYSTEMD_UNIT=$_service > $_service.log | awk '{print $1}'; done ; journalctl --no-pager > big.log

and providing me "*.cat *.log" in a .tar.gz file out be very helpful.

rafaeldtinoco -at- ubuntu.com <- if you don't want to expose your logs file in this bug.

Thanks a lot!

ubuntu 20.04 here. I have the same problem. this is the tutorial that I have followed :

https://vitux.com/install-nfs-server-and-client-on-ubuntu/

these are the commands that I have issued :

$ sudo apt-get update
$ sudo apt install nfs-kernel-server
$ sudo mkdir -p /home/ziomario/Scrivania/Share
$ sudo chown nobody:nogroup /home/ziomario/Scrivania/Share
$ sudo chmod 777 /home/ziomario/Scrivania/Share
$ sudo nano /etc/exports
/home/ziomario/Scrivania/Share 192.168.1.7(rw,sync,no_subtree_check)
$ sudo exportfs -a
$ sudo systemctl restart nfs-kernel-server
$ sudo journalctl -xe

giu 19 14:03:14 ziomario-I9 systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start failed with result 'dependency'.
giu 19 14:03:14 ziomario-I9 systemd[1]: Dependency failed for NFS Mount Daemon.
-- Subject: L'unità nfs-mountd.service failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- L'unità nfs-mountd.service failed
--
-- Il risultato è dependency.

This does NOT affect Groovy:

[rafaeldtinoco@nfstests ~]$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
     Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2020-08-20 17:28:05 UTC; 48s ago
   Main PID: 1185 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 2311)
     Memory: 0B
     CGroup: /system.slice/nfs-server.service

Aug 20 17:28:04 nfstests systemd[1]: Starting NFS server and services...
Aug 20 17:28:05 nfstests systemd[1]: Finished NFS server and services.
[rafaeldtinoco@nfstests ~]$ systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
     Loaded: loaded (/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-08-20 17:27:21 UTC; 1min 40s ago
   Triggers: ● rpcbind.service
     Listen: /run/rpcbind.sock (Stream)
             0.0.0.0:111 (Stream)
             0.0.0.0:111 (Datagram)
             [::]:111 (Stream)
             [::]:111 (Datagram)
      Tasks: 0 (limit: 2311)
     Memory: 60.0K
     CGroup: /system.slice/rpcbind.socket

Warning: journal has been rotated since unit was started, output may be incomplete.
[rafaeldtinoco@nfstests ~]$ systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
     Loaded: loaded (/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-08-20 17:27:21 UTC; 1min 43s ago
TriggeredBy: ● rpcbind.socket
       Docs: man:rpcbind(8)
   Main PID: 289 (rpcbind)
      Tasks: 1 (limit: 2311)
     Memory: 2.7M
     CGroup: /system.slice/rpcbind.service
             └─289 /sbin/rpcbind -f -w

[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-blkmap.service
disabled
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-client.target
enabled
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-common.service
masked
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-config.service
static
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-idmapd.service
static
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-kernel-server.service
alias
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-mountd.service
static
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-server.service
enabled
[rafaeldtinoco@nfstests ~]$ systemctl is-enabled nfs-utils.service
static
[rafaeldtinoco@nfstests ~]$ systemctl is-active rpcbind.
rpcbind.service rpcbind.socket rpcbind.target
[rafaeldtinoco@nfstests ~]$ systemctl is-active rpcbind.service
active
[rafaeldtinoco@nfstests ~]$ systemctl is-active rpcbind.socket
active
[rafaeldtinoco@nfstests ~]$ systemctl is-active rpcbind.target
active
[rafaeldtinoco@nfstests ~]$ systemctl is-active nfs-server.service
active
[rafaeldtinoco@nfstests ~]$ systemctl is-active nfs-client.target
active

This issue also does NOT affect Bionic:

[rafaeldtinoco@bnfstests ~]$ systemctl status nfs-kernel-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2020-08-20 17:46:54 UTC; 29s ago
  Process: 1537 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
  Process: 1536 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 1535 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS)
  Process: 1561 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 1560 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1561 (code=exited, status=0/SUCCESS)

Aug 20 17:46:54 bnfstests systemd[1]: Starting NFS server and services...
Aug 20 17:46:54 bnfstests systemd[1]: Started NFS server and services.
[rafaeldtinoco@bnfstests ~]$ systemctl status rpcbind.socket
Failed to dump process list, ignoring: No such file or directory
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-08-20 17:44:25 UTC; 3min 6s ago
   Listen: /run/rpcbind.sock (Stream)
   CGroup: /system.slice/rpcbind.socket

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[rafaeldtinoco@bnfstests ~]$ systemctl status rpcbind.service
● rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-08-20 17:44:26 UTC; 3min 14s ago
     Docs: man:rpcbind(8)
 Main PID: 382 (rpcbind)
    Tasks: 1 (limit: 2338)
   CGroup: /system.slice/rpcbind.service
           └─382 /sbin/rpcbind -f -w

Aug 20 17:44:26 bnfstests systemd[1]: Starting RPC bind portmap service...
Aug 20 17:44:26 bnfstests systemd[1]: Started RPC bind portmap service.
[rafaeldtinoco@bnfstests ~]$ systemctl status nfs-
nfs-blkmap.service nfs-config.service nfs-mountd.service
nfs-client.target nfs-idmapd.service nfs-server.service
nfs-common.service nfs-kernel-server.service nfs-utils.service
[rafaeldtinoco@bnfstests ~]$ systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
   Loaded: loaded (/lib/systemd/system/nfs-mountd.service; static; vendor preset: enabled)
   Active: active (running) since Thu 2020-08-20 17:46:54 UTC; 54s ago
  Process: 1556 ExecStart=/usr/sbin/rpc.mountd $RPCMOUNTDARGS (code=exited, status=0/SUCCESS)
 Main PID: 1559 (rpc.mountd)
    Tasks: 1 (limit: 2338)
   CGroup: /system.slice/nfs-mountd.service
           └─1559 /usr/sbin/rpc.mountd --manage-gids

Aug 20 17:46:54 bnfstests systemd[1]: Starting NFS Mount Daemon...
Aug 20 17:46:54 bnfstests rpc.mountd[1559]: Version 1.3.3 starting
Aug 20 17:46:54 bnfstests systemd[1]: Started NFS Mount Daemon.
[rafaeldtinoco@bnfstests ~]$ systemctl is-enabled nfs-blkmap.service
disabled
[rafaeldtinoco@bnfstests ~]$ systemctl is-enabled nfs-client.target
enabled
[rafaeldtinoco@bnfstests ~]$ sy...