/forcefsck doesn't add '-f' argument to fsck (for ext4, ext3, ext2)

Could you add "set -x" after the "script" line in /etc/init/mountall.conf and capture the output somehow?

As you can see, this job does attempt to honour the /forcefsck file

I modified /etc/init/mountall.conf as follows:

$ diff -u mountall.conf.old mountall.conf
--- mountall.conf.old 2009-09-25 05:40:33.331304981 -0400
+++ mountall.conf 2009-09-25 05:39:48.551305237 -0400
@@ -22,6 +22,10 @@
 console output

 script
+ mount -t tmpfs none /tmp
+ exec > /tmp/mountall.log 2>&1
+ set -x
+ cat /proc/mounts
     . /etc/default/rcS
     [ -f /forcefsck ] && force_fsck="--force-fsck"
     [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"

When the /forcefsck file is not present, I get the following logged to /tmp/mountall.log:

+ cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,relatime,mode=755 0 0
/dev/mapper/grok-root / ext4 ro,relatime,barrier=1,data=ordered 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /tmp tmpfs rw,relatime 0 0
+ . /etc/default/rcS
+ TMPTIME=0
+ SULOGIN=no
+ DELAYLOGIN=no
+ UTC=yes
+ VERBOSE=no
+ FSCKFIX=no
+ RAMRUN=yes
+ RAMLOCK=yes
+ [ -f /forcefsck ]
+ [ no = yes ]
+ [ -n 0 ]
+ tmptime=--tmptime=0
+ exec mountall --daemon --tmptime=0
fsck from util-linux-ng 2.16
fsck from util-linux-ng 2.16
root: clean, 436712/7192576 files, 5188293/28753920 blocks
boot: clean, 167/62248 files, 31424/248976 blocks

When /forcefsck is present, I get the following logged to /tmp/mountall.log:

+ cat /proc/mounts
rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev tmpfs rw,relatime,mode=755 0 0
/dev/mapper/grok-root / ext4 ro,relatime,barrier=1,data=ordered 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /tmp tmpfs rw,relatime 0 0
+ . /etc/default/rcS
+ TMPTIME=0
+ SULOGIN=no
+ DELAYLOGIN=no
+ UTC=yes
+ VERBOSE=no
+ FSCKFIX=no
+ RAMRUN=yes
+ RAMLOCK=yes
+ [ -f /forcefsck ]
+ force_fsck=--force-fsck
+ [ no = yes ]
+ [ -n 0 ]
+ tmptime=--tmptime=0
+ exec mountall --daemon --force-fsck --tmptime=0
fsck from util-linux-ng 2.16
fsck from util-linux-ng 2.16
boot: 167/62248 files (0.0% non-contiguous), 31424/248976 blocks
root: 436712/7192576 files (0.3% non-contiguous), 5188229/28753920 blocks

So the --force-fsck argument is being passed to mountall. Mountall appears to be running fsck in both cases, but there must be different command-line arguments. However, mountall isn't running fsck.ext4 with the -f option.

On Sun, 2009-09-27 at 22:19 +0000, a7x wrote:

 status incomplete

extending your debugging, could you add "--debug" to the mountall
invocation?

Scott
--
Scott James Remnant
<email address hidden>

I should also note that after running "sudo touch /forcefsck" and rebooting, "/forcefsck" is still there after I log in. I figured it would be deleted after fsck is run.

I'm attaching two files: one from when /forcefsck did not exist and one from when /forcefsck was present.

You have this line in /etc/fstab:

  /dev/mapper/grok-root / ext4 ro,relatime,barrier=1,data=ordered 0 0

The last field probably ought to be 1, not 0.

Jaunty mostly ignored /forcefsck with a zero passno field, but the root filesystem was a special case; if you used /forcefsck, then it would be checked regardless of passno. The attached branch changes mountall to match this behaviour.

The last field is 1. From my /etc/fstab:

# / was on /dev/mapper/grok-root during installation
UUID=1f5e4adc-cd7b-4b38-8952-58dc160368c0 / ext4 relatime,errors=remount-ro 0 1

I'm going to reopen this because I don't think you saw comment #7 (passno=1 in my /etc/fstab).

Mm, I think my fix was correct, but I no longer believe that it has anything directly to do with this bug. fsck does seem to be getting run ...

I think the issue I'm seeing is that /forcefsck doesn't cause the '-f' argument to be added to fsck for ext{2,3,4} (unlike Jaunty).

The code is definitely there to do that:

        if (force_fsck)
                NIH_MUST (nih_str_array_add (&args, NULL, &args_len, "-f"));

I've uploaded a new mountall package to the ubuntu-boot PPA:

https://launchpad.net/~ubuntu-boot/+archive/ppa

I would appreciate it if you could install this and try it out. *BEFORE* you reboot though, could you run "sudo mountall --debug > mountall.log 2>&1" and attach that to this bug - then after you reboot, let me know whether it worked or not.

Thanks

Obviously add --force-fsck to that command ;)

b.t.w. the title made me think it was ext4 related so I made a new bug that got marked as duplicate: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/444798 Sorry about that :p

But just for clarity this also happens for ext3 file systems.

Upgrading to mountall-0.2.0~boot3 left my system unbootable. I booted into another installation, chrooted into the broken install, and downgraded mountall to 0.1.8. Unfortunately, it's still unbootable. Usplash comes up, I enter my cryptsetup password, and after a while of inactivity the usplash logo goes away and I only see the following text:

key slot 0 unlocked.
Command successful.
File descriptor 3 left open
  2 logical volume(s) in volume group "grok" now active

I think it was trying to get me to manually run fsck but it never dropped me to a shell (as far as I could tell). When I took out 'quiet splash' from my kernel command-line arguments it told me to manually run fsck and it dropped me to a shell. I manually ran fsck and fixed some "Superblock modified in the future" errors and now I can boot again.

I'm attaching two mountall log files, one from running 'sudo mountall --force-fsck --debug 2>&1' before upgrading to mountall-0.2.0~boot3 and one after upgrading.

mountall-0.1.8 never returned (I had to end it with ctrl-c) and the output is truncated, probably because stdout wasn't flushed when I killed it.

I tried upgrading to mountall-0.2.0~boot3 again and the same thing happened -- I couldn't boot. The boot process just stopped at the point where the filesystems would be checked and mounted. It didn't ask to manually run fsck, so I'm not sure what's wrong.

I modified the script section of /etc/init/mountall.conf as follows so that I could collect a log of what mountall-0.2.0~boot3 was doing:

script
    # in case I need to manually run fsck before mountall runs
    /bin/bash -li

    # create a writeable place to put a log file
    mount -t tmpfs none /tmp

    # redirect stderr/stdout of all of this to a log file
    (
        set -x
        cat /proc/mounts
        . /etc/default/rcS
        [ -f /forcefsck ] && force_fsck="--force-fsck"
        [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"
        [ -n "$TMPTIME" ] && tmptime="--tmptime=$TMPTIME"
        mountall --debug --daemon $force_fsck $fsck_fix $tmptime
    ) >/tmp/mountall.log 2>&1

    # so that i can copy the log file to a more permanent location
    /bin/bash -li
end script

Attached is the log file.

Sorry about the issues with the previous PPA versions, as usual things worked just fine when I tested it in the various rigs I have here - of course it flatly failed when installed on normal systems because I hadn't actually tested that ;)

I've uploaded a new ~boot4 version, this one feels much better (and I'm running it on my laptop now :p)

As before, after installing the package but *before* you reboot, please run with --debug and attach the log to the bug - then after rebooting, let me know how it works out.

Thanks for all your help with testing, this is a big change and it's good to know that it's now working for 95% of people and your help getting it work for the final 5% is greatly appreciated!

This bug was fixed in the package mountall - 0.2.0

---------------
mountall (0.2.0) karmic; urgency=low

  [ Colin Watson ]
  * Always check the root filesystem if --force-fsck is used, regardless of
    passno. LP: #435707.

  [ Johan Kiviniemi ]
  * Have each fsck instance create a lock for each underlying physical device.
    If you have a single disk or RAID, all filesystem checks will happen
    sequentially in order to avoid thrashing. On more complex configurations,
    you’ll benefit from the parallel checks mountall has been doing all along.
    LP: #434974.

  [ Scott James Remnant ]
  * Flush standard output and error before spawning processes to make
    capturing logs easier (otherwise we end up repeating things still in
    the buffer), and before calling exec().
  * Turn the code upside down so that each mount knows what it's waiting
    for, and allow multiple dependencies. This makes the code much more
    readable putting the "policy" in a single function, and will make it
    much easier in future when this is done by Upstart.
  * For kernel filesystems listed in fstab, honour the order that they
    are listed in fstab. LP: #432571, #433537, #436796
  * Always create new swap partition mounts for each fstab entry, don't
    treat them as updating the same. LP: #435027.
  * Virtual filesystems under local or remote filesystems (and local under
    remote) don't delay the virtual or local events. LP: #431040.
  * Simplify event emission, this has the advantage that we can now output
    what mount points we're waiting for and what they are waiting for as
    well.
  * Fixed issue with trailing slashes. LP: #443035.
  * Only run hooks if the filesystem was not already mounted. LP: #444252.
  * Don't clean up /tmp when run without --tmptime argument.
  * Ignore loop and ram devices until ready. LP: #441454.
  * Add options to binfmt_misc filesystem, which will probably cause it to
    be mounted on boot as well.
  * Synchronously mount local and virtual filesystems, I suspect this is
    the real cause behind the XFS races as one will modprobe and the other
    will not (and fail). LP: #432620.
  * Synchronously activate swap to avoid out of memory issues when checking
    the root filesystem.
  * Enumerate existing udev devices on startup, so we don't always have to
    see udev be coldplugged.
  * Don't break on general errors for non-essential filesystems.
    LP: #441144.
  * Don't repeat attempts to mount a filesystem without having first
    succeded to mount another.
  * Still restart mountall even if the recovery shell fails.
  * Don't queue filesystem check when device is "none", or missing, or the
    filesystem is marked nodev.
  * Generate a "mount" event before mounting a filesystem, and wait for its
    effects to complete.

 -- Scott James Remnant <email address hidden> Fri, 09 Oct 2009 16:50:46 +0100

The status of this bug was automatically set to 'fix released' when mountall-0.2.0 was released even though the change mentioned in the changelog didn't actually apply to this bug. I'm reopening this bug.

I was able to boot with 0.2.0~boot4, so I was able to capture new logs (one without /forcefsck and one with). I'm attaching them.

Looking at the logs, I'm now convinced the '-f' argument is added to fsck. It looks like the real issue is that the user isn't notified that an fsck is happening and the user isn't given the option to cancel the fsck (regression from Jaunty). I'll mark this bug as invalid and open a new bug (if one doesn't already exist).

See bug #446596.