Make NFS client/server work under systemd

Status changed to 'Confirmed' because the bug affects multiple users.

I had an attempt to bring back the init.d script in https://launchpad.net/ubuntu/+source/nfs-utils/1:1.2.8-6ubuntu2 but Steve Langasek didn't like this much. He'll soon split the init.d scripts in Debian as well, then we can bring them back into Ubuntu and fix booting under systemd.

Note there are three units needed.

invoke-rc.d: Unit statd.service is masked
invoke-rc.d: Unit gssd.service is masked
invoke-rc.d: Unit idmapd.service is masked

I'd posted this to ubuntu-devel-discuss but I was just looking at systemd-boot bugs and thought that I'd post it here too, with apologies for the duplication. I hope that it helps whoever puts together the real thing.

I installed the upstream nfs-utils units and adapted them with "/etc/" overrides; and I created rpcbind units.

# diff /etc/systemd/system/nfs-config.service
/lib/systemd/system/nfs-config.service
7c7
< ExecStart=/lib/systemd/scripts/nfs-utils_env.sh
---
> ExecStart=/usr/lib/systemd/scripts/nfs-utils_env.sh

# diff /etc/systemd/system/rpc-statd-notify.service
/lib/systemd/system/rpc-statd-notify.service
19c19
< ExecStart=-/sbin/sm-notify -d $SMNOTIFYARGS
---
> ExecStart=-/usr/sbin/sm-notify -d $SMNOTIFYARGS

# diff /etc/systemd/system/rpc-statd.service
/lib/systemd/system/rpc-statd.service
16,17c16,17
< PIDFile=/run/rpc.statd.pid
< ExecStart=/sbin/rpc.statd --no-notify $STATDARGS
---
> PIDFile=/var/run/rpc.statd.pid
> ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS

# cat /lib/systemd/scripts/nfs-utils_env.sh
#!/bin/sh

NFS_CONFIG=/run/sysconfig/nfs-utils

if [ -e $NFS_CONFIG ] ; then
    rm -f $NFS_CONFIG
fi

if [ -r /etc/default/nfs-common ] ; then
    . /etc/default/nfs-common
fi

if [ -r /etc/default/nfs-kernel-server ] ; then
    . /etc/default/nfs-kernel-server
fi

RPCNFSDARGStmp="$RPCNFSDCOUNT $RPCNFSDOPTS"

mkdir -p /run/sysconfig

echo "GSSDARGS=\"\"" >> $NFS_CONFIG
echo "RPCIDMAPDARGS=\"$RPCIDMAPDARGS\"" >> $NFS_CONFIG
echo "SMNOTIFYARGS=\"\"" >> $NFS_CONFIG
echo "STATDARGS=\"$STATDOPTS\"" >> $NFS_CONFIG

echo "RPCMOUNTDARGS=\"$RPCMOUNTDOPTS\"" >> $NFS_CONFIG
echo "RPCNFSDARGS=\"$RPCNFSDARGStmp\"" >> $NFS_CONFIG
echo "SVCGSSDARGS=\"$RPCSVCGSSDOPTS\"" >> $NFS_CONFIG

[Note on RPCIDMAPDARGS: I set it to "-p /var/lib/nfs/rpc_pipefs" in
"/etc/default/nfs-common" because nfs-idmapd.service was failing
without it. The path to rpc_pipefs was defaulting to "/run/rpc_pipefs"
even though the man page says that the default is
"/var/lib/nfs/rpc_pipefs".]

# cat /lib/systemd/system/rpcbind.service
[Unit]
Description=RPC Bind
After=network.target

[Service]
Type=forking
ExecStart=/sbin/rpcbind
Restart=always

[Install]
WantedBy=rpcbind.target

[Note on rpcbind.service: I'm not sure whether this is the right unit
or whether it should have "Before=rpcbind.target multi-user.target"
and "Requires=rpcbind.target" in the "Unit" section and
"WantedBy=multi-user.target" in the "Install" section.]

# cat /lib/systemd/system/rpcbind.socket
[Unit]
Description=RPC Bind Socket

[Socket]
ListenStream=/run/rpcbind.sock

[Install]
WantedBy=sockets.target

Can't we have a solution to this, please?

My nfs-common doesn't upgrade if I'm using systemd instead of sysvinit and upstart:

Configurando nfs-common (1:1.2.8-9ubuntu1.1) ...
Failed to issue method call: Unit statd.service failed to load: No such file or directory.
invoke-rc.d: initscript statd, action "restart" failed.
dpkg: erro ao processar o pacote nfs-common (--configure):
 sub-processo script post-installation instalado retornou estado de saída de erro 6
Erros foram encontrados durante o processamento de:
 nfs-common
E: Sub-process /usr/bin/dpkg returned an error code (1)

I've recently seen some upstream activity on the native systemd jobs (like on http://www.spinics.net/lists/linux-nfs/msg48981.html), maybe we should just use these?

Taking over from Steve as discussed on IRC, he doesn't find enough time for this.

These are the steps to set up a VM to test both server and client:

# set up exports
sudo apt-get install -y nfs-kernel-server
echo -e '/home *(rw,no_subtree_check)\n/var/log/ *(ro,subtree_check)' | sudo tee -a /etc/exports
sudo service nfs-kernel-server start

# set up client mounts
sudo mkdir /mnt/nfs_home /mnt/nfs_log
cat << EOF | sudo tee -a /etc/fstab
localhost:/home /mnt/nfs_home nfs defaults,nofail 0 0
localhost:/var/log /mnt/nfs_log nfs defaults,nofail 0 0
EOF
sudo reboot

# some simple tests to see that it worked
ls -l /mnt/nfs_log/wtmp /mnt/nfs_home/ubuntu/.bash*
echo hello > /mnt/nfs_home/ubuntu/world.txt
cat ~/world.txt # should give "hello"
sudo touch /mnt/nfs_log/pwned # should fail with "read-only fs"

This works fine under upstart, but as expected fails once you install systemd-sysv:

$ sudo systemctl status nfs-kernel-server
● nfs-kernel-server.service - LSB: Kernel NFS server support
   Loaded: loaded (/etc/init.d/nfs-kernel-server)
   Active: active (exited) since Mon 2015-03-02 14:42:41 CET; 1min 9s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 507 ExecStart=/etc/init.d/nfs-kernel-server start (code=exited, status=0/SUCCESS)

Mar 02 14:42:41 autopkgtest systemd[1]: Starting LSB: Kernel NFS server support...
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Exporting directories for NFS kernel daemon...
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: ...done.
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Starting NFS kernel daemon
Mar 02 14:42:41 autopkgtest nfs-kernel-server[507]: * Not starting: portmapper is not running
Mar 02 14:42:41 autopkgtest systemd[1]: Started LSB: Kernel NFS server support.

and the client mounts hang:

$ ps aux|grep nfs
root 608 0.0 0.0 21544 1696 ? S 14:42 0:00 /bin/mount localhost:/var/log /mnt/nfs_log -n -t nfs -o defaults
root 610 0.0 0.0 21544 1688 ? S 14:42 0:00 /bin/mount localhost:/home /mnt/nfs_home -n -t nfs -o defaults
root 611 0.0 0.1 40192 3052 ? S 14:42 0:00 /sbin/mount.nfs localhost:/var/log /mnt/nfs_log -n -o rw
root 612 0.0 0.1 40192 3180 ? S 14:42 0:00 /sbin/mount.nfs localhost:/home /mnt/nfs_home -n -o rw

As we are quite a few upstream versions behind, I went through all systemd related upstream commits. There were a ton for adding the systemd/ directory with the units, which we should just start with as a whole and adjust to the ubuntu specifics/what the upstart jobs do: http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=tree;f=systemd

Aside from those, there were just 4 commits to utils/statd/start-statd, but it seems easier to just take the whole current file: http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=utils/statd/start-statd

There is an ordering cycle in rpcbind which needs to be fixed first. This is because /etc/init.d/rpcbind depends on $network which in Ubuntu is implemented to actually wait for ifupdown/NM/etc. But that needs dbus, which in turn needs $remote_fs, which needs rpcbind.

Finally there are two dependency cycles which we need to fix:

Found ordering cycle on basic.target/start
Found dependency on sysinit.target/start
Found dependency on apparmor.service/start
Found dependency on remote-fs.target/start
Found dependency on remote-fs-pre.target/start
Found dependency on nfs-client.target/start
Found dependency on rpc-statd-notify.service/start
Found dependency on nfs-server.service/start
Found dependency on rpc-svcgssd.service/start
Found dependency on basic.target/start
Breaking ordering cycle by deleting job apparmor.service/start
Job apparmor.service/start deleted to break ordering cycle starting with basic.target/start

Found ordering cycle on basic.target/start
Found dependency on sysinit.target/start
Found dependency on console-setup.service/start
Found dependency on remote-fs.target/start
Found dependency on remote-fs-pre.target/start
Found dependency on nfs-client.target/start
Found dependency on rpc-statd-notify.service/start
Found dependency on nfs-server.service/start
Found dependency on rpc-svcgssd.service/start
Found dependency on basic.target/start
Breaking ordering cycle by deleting job console-setup.service/start
Job console-setup.service/start deleted to break ordering cycle starting with basic.target/start

This is because these two don't have native systemd units, and their init.d script depends on $remote_fs. But remote-fs.target requires quite a lot (networking, dbus, perhaps NetworkManager, etc.), and we do want to protect those with apparmor. Also, apparmor only uses files in /, so $local_fs is sufficient even for the hypothetical "/usr on NFS" case.

That's less true for console-setup: This does use files from /usr; however, as we've never officially supported /usr on NFS, and nobody ever tested that under upstart, I'm very much inclined to just fix its init.d dependency to $local_fs, too.

First test package available in https://launchpad.net/~pitti/+archive/ubuntu/systemd
I added the current upstream units as a big patch, and then added separate patches for our changes, so that it's easier to see what we changed; these also become easier to upstream that way.

TODO:
 - debian/nfs-common.gssd.upstart does some elaborate NEED_GSSD detection. The systemd unit just does ConditionPathExists=/etc/krb5.keytab which looks simpler; but I don't know enough about krb to say whether it's equivalent/sufficient, or whether we need the complicated parsing? Steve?

 - $NEED_STATD isn't taken into account. At the moment, rpc.statd is started unconditionally.

 - unmounting the NFS mounts hangs on shutdown

This bug was fixed in the package rpcbind - 0.2.1-6ubuntu2

---------------
rpcbind (0.2.1-6ubuntu2) vivid; urgency=medium

  * Add systemd unit. Patch by Simon McVittie. (Closes: #748074, LP: #1312976)
 -- Martin Pitt <email address hidden> Tue, 03 Mar 2015 08:32:38 +0100

This bug was fixed in the package apparmor - 2.9.1-0ubuntu5

---------------
apparmor (2.9.1-0ubuntu5) vivid; urgency=medium

  * debian/apparmor.init: Replace unnecessary $remote_fs dependency with
    $local_fs. This is sufficient as during boot we don't use anything from
    /usr. It's also necessary to avoid dependency cycles when using NFS (as
    its dependencies should be covered by AppArmor). (LP: #1312976)
 -- Martin Pitt <email address hidden> Tue, 03 Mar 2015 08:54:33 +0100

pitti2 uploaded to PPA now. This fixes the umount hang, and drops the manual modprobing. The modules get loaded automatically without that too. Neither upstream nor the Fedora package do any extra modprobing.

nfs-utils - 1:1.2.8-9ubuntu2pitti3 uploaded to PPA. This merely cleans up the changelog a bit and adds a proper "Forwarded:" reference to an upstreamable patch.

I now tested this on both a single machine as well as two separate VMs (one server, one client), with some /mnt/* mounts as well as /home on NFS. Works fine now.

The remaining TODO is

- $NEED_STATD isn't taken into account. At the moment, rpc.statd is started unconditionally.

But that's an optimization and thus we could do this in a followup upload.

debdiff for review: http://people.canonical.com/~pitti/tmp/nfs-utils.systemd.debdiff

I split out the remaining issues into bug 1428486 and bug 1428487. I'm also going to create an autopkgtest.

This bug was fixed in the package nfs-utils - 1:1.2.8-9ubuntu3

---------------
nfs-utils (1:1.2.8-9ubuntu3) vivid; urgency=medium

  * Add 00git-start-statd-systemd.patch: Latest start-statd script from
    1.3.2 to start rpc-statd.service under systemd.
  * Add 23-systemd-pipefs_in_run.patch: systemd: Mount rpc_pipefs in /run instead of
    /var/lib/nfs/, like in the upstart units.
  * Add 24-systemd-daemon-paths.patch: Adjust program paths in systemd units.
  * debian/nfs-kernel-server.links: Add nfs-kernel-server.service alias
    symlink, to match SysV init script.
  * Add 25-systemd-server-before-client.patch: Order NFS server before client,
    to make mounting NFS shares from localhost work reliably.
  * Add debian/nfs-utils_env.sh: Translate our /etc/default files into runtime
    configuration for nfs-config.service.
  * debian/nfs-{common,kernel-server}.install: Install systemd units.
  * debian/rules: Enable/start systemd units. (LP: #1312976)
 -- Martin Pitt <email address hidden> Thu, 05 Mar 2015 07:46:41 +0100