Reboot hangs because /etc/rc6.d/S40umountfs chokes on non-existent mounts

I modified the umountfs script to run fstab-decode under strace. I'm attaching the strace output.

I've poked around at this further, and it seems that the fstab-decode command only hangs trying to unmount *direct* autofs entries in /proc/mounts, but does not hang on *indirect* ones. An example of the direct entry in /proc/mounts:

/etc/auto.nfs_cdf /cdf autofs rw,relatime,fd=18,pgrp=865,timeout=300,minproto=5,maxproto=5,direct 0 0

Note once again that this is *not* an actual mounted file system. The file system mounted by this autofs rule (if mounted) appears further down in /proc/mounts like this (I've obfuscated the IP addresses to 1.2.3.4):

homesrv:/cdf/ /cdf nfs ro,nosuid,nodev,noexec,noatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=udp,timeo=11,retrans=3,sec=sys,mountaddr=1.2.3.4,mountvers=3,mountport=52150,mountproto=udp,local_lock=none,addr=1.2.3.4 0 0

I'm attaching a one-line modification to the /etc/init.d/umountfs script that fixes the problem for us. It causes umountfs to ingore /proc/mounts entries where FSTYPE is 'autofs'.

I believe that this is a correct solution because these 'autofs' entries are not the actual file systems mounted by autofs, but the mount points or directories monitored by the automount daemon. The actual autofs-mounted file systems (if any) will not be skipped over because they will appear as a separate entry in /proc/mounts and their FSTYPE is not going to be 'autofs', but whatever it is in reality ('nfs' in our case, for example).

I don't think this analysis is correct. At the time the umountfs script finishes, *all* filesystems are supposed to be unmounted. This includes any mount points watched by autofs. So why is autofs still running at this point in the shutdown? The autofs daemon should be shut down *prior* to /etc/rc6.d/S35networking, since at that point there's no network left and autofs should no longer be trying to mount new filesystems anyway!

And if I look at the autofs5 package, it has a wrong upstart job that does 'stop on runlevel [!2345]', which does not properly serialize the shutdown to ensure the service is stopped before we reach the unmounting phase. Thus there's a race between /etc/init/rc.conf and /etc/init/autofs.conf.

Reassigning to the autofs5 package.

The attachment "umountfs.diff" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

Steve, I think that you are both right and wrong. You are right that autofs.conf does not work properly and that the autofs-related stuff should be stopped before umountfs script runs. However, there are two other things:

1. The entries in /proc/mounts with 'autofs' FSTYPE *are not file systems that need unmounting*, hence it is both safe and valid to ignore them in the umountfs script. The real file systems that have been mounted by the automounter would show up as separate entries in /proc/mounts with the proper FSTYPE ('ext4', 'nfs' or whatever).

2. The fstab-decode command should not hang on any input. It should be more robust and handle the inconsistencies like this more gracefully. Especially it should not hang when given a mount point at which no file system is mounted, which is the case here.

In other words, I think that my patch to the umountfs script is correct in as much as it accounts for the imperfect world in which the automounter was not shut down properly and fstab-decode can be hung.

The patch is also completely safe because ignoring the 'autofs' entries from /proc/mounts is completely harmless since they are not mounted file systems.

Looks like autofs needs to do 'stop on deconfiguring-networking'. That will ensure that the main process is completely stopped before the network is shutdown. I wonder if we should also consider raising the 'kill timeout' above 5 seconds, as it may take longer than that to unmount a lot of mounts, which could also explain the phantom mounts leftover, if the daemon was SIGKILL'd before it was done. Is it conceivable that these phantom mounts will be there even after autofs is completely shutdown in a graceful way?

I'm changing this status to Confirmed. There's clearly a problem with autofs's shutdown, and it needs to be addressed.