aufs fstab entries require ordering logic

This is by design of mountall, it mounts the contents of fstab in a necessary order rather than the specified order. Will fix the fstab manpage later.

We should handle aufs entries though, that's simply a bug

Actually, let's just re-use this bug for the aufs ordering issue

I've uploaded a new mountall package to the ubuntu-boot PPA:

https://launchpad.net/~ubuntu-boot/+archive/ppa

I would appreciate it if you could install this and try it out. *BEFORE* you reboot though, could you run "sudo mountall --debug > mountall.log 2>&1" and attach that to this bug - then after you reboot, let me know whether it worked or not.

Thanks

Hi,
I'he tried to execute mountall --debug (version 0.2.0~boot2) on an already started system and it worked ok. The result is attached.
But when I tried to boot the system it hanged with the following error:
  rm: cannot remove /focefsck: Read-only file system
  init: mountall post-stop process (329) terminated with status 1
  General error mountting filesystems
The /etc/fstab was:
  /dev/sda1 / ext4 relatime 0 1
  /dev/sda2 /.snapshot/home ext4 ro 0 0
  none /.snapshot/home_ram tmpfs defaults 0 0
  none /home aufs br=/.snapshot/home_ram:/.snapshot/home=ro 0 0
And /proc/mounts was (after executing mount -o remount,rw /):
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,relatime,mode=755 0 0
/dev/sda1 / ext4 rw,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /var/run tmpfs rw,nosuid,relatime,mode=755 0 0
none /var/lock tmpfs rw,nosuid,nodev,noexec,relatime 0 0
none /lib/init/rw tmpfs rw,nosuid,relatime,mode=755 0 0
/dev/sda2 /.snapshot/home ext4 ro,relatime,barrier=1,data=ordered 0 0
none /.snapshot/home_ram tmpfs rw,relatime 0 0
none /home aufs rw,relatime,si=142201b2 0 0

If I start the system with the root filesystem on aufs it hangs without error. Similar to bug "https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/431204" Virtual (aufs) root device never ready.

On Thu, 2009-10-08 at 15:04 +0000, vj wrote:

> I'he tried to execute mountall --debug (version 0.2.0~boot2) on an already started system and it worked ok. The result is attached.
> But when I tried to boot the system it hanged with the following error:
> rm: cannot remove /focefsck: Read-only file system
> init: mountall post-stop process (329) terminated with status 1
>
That's encouraging - I'm aware of this second bug and am fixing it this
evening.

Scott
--
Scott James Remnant
<email address hidden>

Sorry about the issues with the previous PPA versions, as usual things worked just fine when I tested it in the various rigs I have here - of course it flatly failed when installed on normal systems because I hadn't actually tested that ;)

I've uploaded a new ~boot4 version, this one feels much better (and I'm running it on my laptop now :p)

As before, after installing the package but *before* you reboot, please run with --debug and attach the log to the bug - then after rebooting, let me know how it works out.

Thanks for all your help with testing, this is a big change and it's good to know that it's now working for 95% of people and your help getting it work for the final 5% is greatly appreciated!

Hi,
there are two differences between mountall 0.2.0~boot2 and 0.2.0~boot4 debug file:
 * pid numbers
 * lines "update_mount" related to the aufs mount appear two lines later
and one difference in behavior:
 * the 0.2.0~boot4 does not return, it has to be killed with ctrl+c
When tried to boot the new version it worked fine.
So the problem seems solved.
Thanks,
  Vicenç.

On Fri, 2009-10-09 at 14:03 +0000, vj wrote:

> * the 0.2.0~boot4 does not return, it has to be killed with ctrl+c
> When tried to boot the new version it worked fine.
>
After booting, if you run "sudo status mountall" is it still running?

If so, could you run "sudo stop mountall", then "sudo mountall --debug
2>&1 | tee debug.log" - and attach the log

Especially note whether it stops.

Scott
--
Scott James Remnant
<email address hidden>

This bug was fixed in the package mountall - 0.2.0

---------------
mountall (0.2.0) karmic; urgency=low

  [ Colin Watson ]
  * Always check the root filesystem if --force-fsck is used, regardless of
    passno. LP: #435707.

  [ Johan Kiviniemi ]
  * Have each fsck instance create a lock for each underlying physical device.
    If you have a single disk or RAID, all filesystem checks will happen
    sequentially in order to avoid thrashing. On more complex configurations,
    you’ll benefit from the parallel checks mountall has been doing all along.
    LP: #434974.

  [ Scott James Remnant ]
  * Flush standard output and error before spawning processes to make
    capturing logs easier (otherwise we end up repeating things still in
    the buffer), and before calling exec().
  * Turn the code upside down so that each mount knows what it's waiting
    for, and allow multiple dependencies. This makes the code much more
    readable putting the "policy" in a single function, and will make it
    much easier in future when this is done by Upstart.
  * For kernel filesystems listed in fstab, honour the order that they
    are listed in fstab. LP: #432571, #433537, #436796
  * Always create new swap partition mounts for each fstab entry, don't
    treat them as updating the same. LP: #435027.
  * Virtual filesystems under local or remote filesystems (and local under
    remote) don't delay the virtual or local events. LP: #431040.
  * Simplify event emission, this has the advantage that we can now output
    what mount points we're waiting for and what they are waiting for as
    well.
  * Fixed issue with trailing slashes. LP: #443035.
  * Only run hooks if the filesystem was not already mounted. LP: #444252.
  * Don't clean up /tmp when run without --tmptime argument.
  * Ignore loop and ram devices until ready. LP: #441454.
  * Add options to binfmt_misc filesystem, which will probably cause it to
    be mounted on boot as well.
  * Synchronously mount local and virtual filesystems, I suspect this is
    the real cause behind the XFS races as one will modprobe and the other
    will not (and fail). LP: #432620.
  * Synchronously activate swap to avoid out of memory issues when checking
    the root filesystem.
  * Enumerate existing udev devices on startup, so we don't always have to
    see udev be coldplugged.
  * Don't break on general errors for non-essential filesystems.
    LP: #441144.
  * Don't repeat attempts to mount a filesystem without having first
    succeded to mount another.
  * Still restart mountall even if the recovery shell fails.
  * Don't queue filesystem check when device is "none", or missing, or the
    filesystem is marked nodev.
  * Generate a "mount" event before mounting a filesystem, and wait for its
    effects to complete.

 -- Scott James Remnant <email address hidden> Fri, 09 Oct 2009 16:50:46 +0100

Yes, it's still running after boot
I've stopped it and executed again with the --debug option.
The result is the same as version 0.2.0~boot2 (few lines changed their order relative to version 0.2.0~boot4).
Nothing is printed after it stops, neither after ctrl+c hit.

On Fri, 2009-10-09 at 17:30 +0000, vj wrote:

> Yes, it's still running after boot
> I've stopped it and executed again with the --debug option.
> The result is the same as version 0.2.0~boot2 (few lines changed their order relative to version 0.2.0~boot4).
> Nothing is printed after it stops, neither after ctrl+c hit.
>
Could you attach it again, please.

There are minor differences that are important.

Scott
--
Scott James Remnant
<email address hidden>

Here it is, hope it helps.

On Sat, 2009-10-10 at 12:51 +0000, vj wrote:

> Here it is, hope it helps.
>
> ** Attachment added: "mountall--debug.txt"
> http://launchpadlibrarian.net/33404847/mountall--debug.txt
>
Unfortunately this is significantly truncated :-(

How did you terminate mountall?

Scott
--
Scott James Remnant
<email address hidden>

Hi
forget what I said about "the 0.2.0~boot4 does not return, it has to be killed with ctrl+c".
I was not aware of it is working in the foreground, so I killed it.
The issue involved in this bug is solved.
Thanks,
  Vicenç.

Hi,

I've also seen this problem on my system, where I use AUFS overlays over squashfs filesystems. I'm not sure if I understand the ubuntu-boot ppa page correctly, in that there doesn't seem to be any mention of updated mountall there but I am under the impression that the fixes mentioned in this thread are all released now.

My problem was twofold - firstly, when mountall parses my aufs line in function parse_fstab() it overrides the squashfs with a call to function update_mount(), instead of adding a second entry with new_mount().

I overcame this by adding a special condition for AUFS mounts, so that it calls new_mount() instead.

The second problem was in function mount_policy(). Now that I was getting a mount point for my squashfs, and one for my AUFS,
mount_policy() was making them dependents of each other, and this cyclic dependency was causing an endless loop.

I solved for this on my system by adding a condition in mount_policy() whereby an AUFS mount will nominate it's matching mountpoint as a dependency, and then adding a condition in is_parent() that returns FALSE if the length of the root is EQUAL TO OR GREATER THAN the length of the path.

Compiled and tested happily on my own system, a patch to mountall.c has been attached (to clarify what I am talking about here, as opposed to promoting it as the proper solution)

I'm still experiencing the problem that aufs is mounted before the underlying mount is ready:

Snippet from mountall debug log (see full log in attachment)

update_mount: /var/lock: /var/lock none tmpfs nodev,noexec,nosuid,showthrough
update_mount: /media/aufs: /media/aufs none aufs br:/media/sdb1=rw,create=pmfs,cpup=bup check
update_mount: /media/sdb1: /media/sdb1 /dev/sdb1 ext4 errors=remount-ro check

/etc/fstab:

proc /proc proc nodev,noexec,nosuid 0 0
UUID=5ba3227b-b091-4d55-915f-852bb93fd8fe / ext4 errors=remount-ro 0 1
UUID=ccc2ea9f-7228-4665-95d5-3240f7e5b373 none swap sw 0 0
UUID="1b0761fa-6c7b-4e3c-b26a-73613850d029" /media/sdb1 ext4 errors=remount-ro 0 2
none /media/aufs aufs br:/media/sdb1=rw,create=pmfs,cpup=bup 0 3