systemd unmounts mounted filesystems when lxcfs is installed

fstab:

# / was on /dev/sda1 during installation
UUID=71eeb904-068b-41c2-9d34-59365ca4c3ad / ext4 errors=remount-ro 0 1
# /home was on /dev/sda3 during installation
UUID=bb228002-fcfc-49ba-be13-8d62ea7d940e /home ext4 defaults 0 2

UUID=91c87914-f96b-496f-b71c-11407835d58b /media/wdblack2 ext4 errors=remount-ro 0 2
UUID="d8b4062a-ef6b-40f5-a77e-77b98f999b50" /media/darkstore ext4 errors=remount-ro 0 2
UUID="8c39af66-02c7-4269-9ba1-4704369f8aeb" /media/sstore ext4 errors=remount-ro 0 2
UUID=0564f93c-ed50-48d3-abc9-89c572936ee8 /media/gstore ext4 defaults 0 2
UUID=12b1ff1b-31ba-4e77-a8ea-cfe75a61bf2d none swap sw 0 0

This seems to be the interesting excerpt from the log, for /home (which is on partition UUID=bb22..). The home.mount unit is bound to the corresponding .device unit for the partition, but this somehow goes wrong:

$ egrep 'bb22.*device|home.mount' /tmp/systemd-nomounts.log
Feb 09 17:23:59 duhast systemd[1]: home.mount changed dead -> mounted
Feb 09 17:23:59 duhast systemd[1]: Unit home.mount is bound to inactive service. Stopping, too.
Feb 09 17:23:59 duhast systemd[1]: Trying to enqueue job home.mount/stop/fail
Feb 09 17:23:59 duhast systemd[1]: Installed new job home.mount/stop as 9
Feb 09 17:23:59 duhast systemd[1]: Enqueued job home.mount/stop as 9
Feb 09 17:23:59 duhast systemd[1]: Job home.mount/stop finished, result=canceled
Feb 09 17:23:59 duhast systemd[1]: Installed new job home.mount/start as 59
Feb 09 17:23:59 duhast systemd[1]: Installed new job dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device/start as 60
Feb 09 17:23:59 duhast systemd[1]: Expecting device dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device...
Feb 09 17:24:01 duhast systemd[1]: dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device changed dead -> plugged
Feb 09 17:24:01 duhast systemd[1]: Job dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device/start finished, result=done
Feb 09 17:24:04 duhast systemd[1]: Job home.mount/start finished, result=done
Feb 09 17:24:09 duhast systemd[1]: home.mount changed mounted -> dead
Feb 09 17:24:13 duhast systemd[3556]: dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device changed dead -> plugged
Feb 09 17:25:42 duhast systemd[6834]: dev-disk-by\x2duuid-bb228002\x2dfcfc\x2d49ba\x2dbe13\x2d8d62ea7d940e.device changed dead -> plugged

Tim: Can you please try "sudo systemctl disable cgmanager"? If you have lxcfs installed, then please disable that as well. Can you then please reboot a few times and check if everything works then? This worked for me. (It seems to be cgmanager; lxcfs pulls this in as well, so that one needs to be disabled too).

Debugging notes: Lennart suggested that the "dead -> mounted" looks fishy: from systemd itself it would first go to "mounting", and it would log its "mount" calls too; indeed on a working boot I see:

Feb 11 22:55:59 donald systemd[1]: Installed new job home.mount/start as 49
Feb 11 22:55:59 donald systemd[1]: Installed new job dev-disk-by\x2duuid-f86539b0\x2d3a1b\x2d4372\x2d83b0\x2dacdd029ade68.device/start as 48
Feb 11 22:55:59 donald systemd[1]: Expecting device dev-disk-by\x2duuid-f86539b0\x2d3a1b\x2d4372\x2d83b0\x2dacdd029ade68.device...
Feb 11 22:56:00 donald systemd[1]: dev-disk-by\x2duuid-f86539b0\x2d3a1b\x2d4372\x2d83b0\x2dacdd029ade68.device changed dead -> plugged
Feb 11 22:56:00 donald systemd[1]: Job dev-disk-by\x2duuid-f86539b0\x2d3a1b\x2d4372\x2d83b0\x2dacdd029ade68.device/start finished, result=done
Feb 11 22:56:02 donald systemd[1]: home.mount changed dead -> mounting
Feb 11 22:56:02 donald systemd[1]: home.mount changed mounting -> mounting-done
Feb 11 22:56:02 donald systemd[1]: Job home.mount/start finished, result=done
Feb 11 22:56:02 donald systemd[1]: Child 530 belongs to home.mount
Feb 11 22:56:02 donald systemd[1]: home.mount mount process exited, code=exited status=0
Feb 11 22:56:02 donald systemd[1]: home.mount changed mounting-done -> mounted
Feb 11 22:56:04 donald systemd[1335]: dev-disk-by\x2duuid-f86539b0\x2d3a1b\x2d4372\x2d83b0\x2dacdd029ade68.device changed dead -> plugged
Feb 11 22:56:04 donald systemd[1335]: home.mount changed dead -> mounted

which is how it's supposed to look like; but in the failed case, something is mounting/unmounting /home (and other fstab partitions) during early boot. I replaced /bin/mount and /bin/umount with a wrapper which logs the calls to "mount"; nothing calls umount during boot, and in the successful case I see systemd doing all the mounts, in the failing case they are completely missing. That indicates that the other thing that's messing with the mounts is not calling the binary but the mount() syscall.

Martin, Yes I can confirm that disabling cgmanager and lxcfs makes things work again here.

I finally understand what's going on. This started a few weeks ago when we got lxcfs. What happens:

 - lxcfs' upstart job and systemd unit have this gem:
     sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab

   This will turn /etc/mtab into a file if it was a symlink to /proc/mounts. But these days we actually want it to be a symlink, as /etc/mtab as a static file is just a braindead and broken concept.

 - When systemd starts up, it parses existing mounts (via /etc/mtab) and creates mounts for them. This will include all the old mounts from the previous boot due to the previous issue. That's what triggers all these "dead -> mounted" transitions for all mounts right at the beginning of boot. But as the backing devices of them aren't discovered/there yet, but a mount unit is always bound to its underlying device unit, systemd cleans them up, i. e. stops them. This is usually meant for cleaning stale mounts from unplugged devices; it also makes sense here as these mounts indeed are not mounted at the moment.

 - Later on, the fstab generated mounts are actually being executed. But this races with the async mount cleanup handler from above, so depending on the particular machine, installed services, and current wind speed it may happen at a harmless (before the fstab mounts) or at a bad time (after the fstab mounts).

 - Even later, our debian-fixup.service transitions /etc/mtab from a file to a symlink. This is supposed to be an one-time migration, but lxcfs always reverts that during shutdown.

systemd uses libmount's mnt_table_parse_mtab() which in our packages seems to default to /etc/mtab instead of /proc/mounts.

systemd fix sent upstream: http://lists.freedesktop.org/archives/systemd-devel/2015-February/028264.html

This bug was fixed in the package lxcfs - 0.5-0ubuntu2

---------------
lxcfs (0.5-0ubuntu2) vivid; urgency=medium

  * debian/lxcfs.service: Drop seddery of /etc/mtab, which turns /etc/mtab
    into a file. Under systemd, /etc/mtab is always a symlink to /proc/mounts.
    (LP: #1419623)
  * debian/lxcfs.upstart: Don't sed /etc/mtab if it is a symlink, to avoid
    reverting it back to a file if it was changed to be a symlink.
 -- Martin Pitt <email address hidden> Thu, 12 Feb 2015 10:41:57 +0100

Quoting Martin Pitt (<email address hidden>):
> I finally understand what's going on. This started a few weeks ago when
> we got lxcfs. What happens:
>
> - lxcfs' upstart job and systemd unit have this gem:
> sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab
>
> This will turn /etc/mtab into a file if it was a symlink to /proc/mounts. But these days we actually want it to be a symlink, as /etc/mtab as a static file is just a braindead and broken concept.

init jobs could be updated to do

 [ ! -h /etc/mtab ] && sed -i "/^lxcfs \/var\/lib\/lxcfs fuse.lxcfs/d" /etc/mtab

Yes, that's precisely what I did :) (for the upstart job); well, I used -L, not -h, but same difference.

The same should also be done to the sysvinit script I guess. I'll pick up those changes in the packaging branch and update for sysvinit.

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=999fb5e2b0a0

Stéphane Graber [2015-02-12 15:05 -0000]:
> The same should also be done to the sysvinit script I guess. I'll pick
> up those changes in the packaging branch and update for sysvinit.

Thanks for that! I completely forgot about the sysvinit script...

This bug was fixed in the package systemd - 218-10ubuntu1

---------------
systemd (218-10ubuntu1) vivid; urgency=medium

  [ Martin Pitt ]
  * Merge with Debian unstable. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also, check if devices are actually defined
      in /etc/network/interfaces as we don't use Debian's net.agent.
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor1 dep to the Ubuntu version where it moved to /lib.
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being triggered by Jenkins, but is totally
      unreproducible when running this manually on the exact same machine.

    Upgrade fixes, keep until 16.04 LTS release:
    - systemd Conflicts/Replaces/Provides systemd-services.
    - Remove obsolete systemd-logind upstart job.
    - Clean up obsolete /etc/udev/rules.d/README.

  * ifup@.service: Fix syntax error. (LP: #1421556, #1420601)

  [ Didier Roche ]
  * Add systemd-fsckd multiplexer and feed its output to plymouth. This
    provides an aggregate progress report of running file system checks and
    also allows cancelling them with ^C.
    (LP: #1316796; Closes: #775093, #758902)

systemd (218-10) experimental; urgency=medium

  * Pull latest keymaps from upstream git. (LP: #1334968, #1409721)
  * rules: Fix by-path of mmc RPMB partitions and don't blkid them. Avoids
    kernel buffer I/O errors and timeouts. (LP: #1333140...