LVM - /var failed to mount during boot

Same here for /home on LVM, here is the mountall --debug log

Could you please describe what you mean by "failed to boot".

You say that you see a message about the filesystem, what is that message? (Please copy it exactly)

If the message offers you to wait, what happens if you wait?

What happens if you press each of the keys?

I am experiencing exactly the same problem since I upgraded to Lucid two days ago.

Seems to be related to the following issue in Lucid:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/527666

The boot process hangs with the following message:
The disk drive for /xxx is not ready yet or not present.
Continue to wait; or Press S to skip mount or M for manual recover

/xxx is any partition on LVM.

I have to execute the following procedure to get to the GDM screen:
1. Enter M (for Manual Recovery)
2. Execute "mount -a" which mounts all filesystems on LVM without problems.
3. CTRL-D to close the shell and continue with the reboot.

Check the bug report mentioned above for detailed information on the issue.

Same here, but sometimes everything works (about half the time):

I have /home under LVM:
/dev/cassini/cassini-home /home ext4 errors=remount-ro 0 1

The boot process (sometimes) hangs with the following message:
The disk drive for /home is not ready yet or not present.
Continue to wait; or Press S to skip mounting or M for manual recovery

I press M
# mount /home
# exit

and then the boot proceeds. See my mountall logs at comment 3

Hey folks,

I have a random hunch on this one. In my PPA you'll find a new mountall
package, could you try it out and see whether it makes things better or
worse?

  sudo add-apt-repository ppa:scott/ppa
  sudo apt-get update
  sudo apt-get upgrade

Check you have mountall 2.14~ppa1

  dpkg-query -W mountall

Then reboot

Thanks,

Scott
--
Scott James Remnant
<email address hidden>

I was also affected by this bug for about a week. Continuing to wait for /home to be mounted did not seem to work (for about 5 minutes, at least) so i did the same thing than Thierry Carrez.

My /home partition is under LVM on top of a software RAID 1.

I tried it three time with this mountall. It worked the first one, failed the second, and worked the third, but with a message i did not have time to write down concerning my /dev/md0 array not yet ready.

Something else worth mentioning is that /dev/shm seems to not have been set with the right mode. It is easy to notice because chromium browser is then unable to display its panels.

Tried the PPA over in bug #567910, still get problems....

I am getting tempted to have ago at this myself.

May I suggest a fix for mountall to get this thing from being a show stopper?

This whole thing could be avoided if a mount for the file system read from fstab is tried. If this fails, then the 'press 'M' for shell, or 's' to skip' should be shown. The file system is in /etc/fstab for a reason, and having a blind go and checking the return code of the mount operation is justified given that udev is creating all the device nodes etc needed.

Missing an add/change event from udev in early boot can be a hard thing to debug.,

unless you set a a 20G partition as LVM, add 10 !G LVs to /etc/fstab, and run gdb on mountall in a terminal window... You can activate and deactivate the the test volume group with '# vgchange -a y /dev/<volume_group>' wile logged in to the desktop.

The heat needs to be taken off this one, and it can be fixed properly after the Lucid release with an update.

The bug 527666 is still not fixed, and the two bugs seem to related. I upgraded to mountall from the PPA, but I still have problems randomly, getting some filesystems on LVs mounted. The issue does not seem to occur consistently, because I have upgraded two laptops to Lucid and only one exhibits the problem. The laptop that still exhibits the issue has / on LVM, but the / filesystem is mounted without problems, while randomly some other filesystems on LVM fail to get mounted.

scott: just tried 2.14~ppa1 but it doesn't fix the issue (which we believe to be also a dupe of bug #527666 and bug #540155). We keep getting random LVM mount issues at boot.

From what I see, the ppa build fixes the buffer to listen to udev. However, the udev logs don't even list the missing devices (/var/log/udev) which makes us think that udev itself doesn't know about the device.

lvscan, however, sees it fine.

If you are still having problems, please:

 * make sure you have mountall 2.14~ppa1 installed (from my PPA)
 * edit /etc/init/mountall.conf and add --debug to the exec line
 * reboot
 * attach /var/log/boot.log to this bug, along with /var/log/udev

2.13 didn't work for me, but after trying out 2.14 from the PPA it booted up cleanly. I'm attaching the boot.log with --debug and /var/log/udev.

What Happened:
System booted and stuck at mounting /var and /local/mnt, continued booting after pressing S to skip

What I did:
Added PPA and upgraded to mountall 2.14, which then the system booted fully and mounted all LVM partitions without needing to press S.

lsb_release -rd
Description: Ubuntu 10.04 LTS
Release: 10.04

apt-cache policy mountall
mountall:
  Installed: 2.14~ppa1
  Candidate: 2.14~ppa1
  Version table:
 *** 2.14~ppa1 0
        500 http://ppa.launchpad.net/scott/ppa/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status
     2.13 0
        500 http://apt-dev/ubuntu/ lucid/main Packages

* I have upgraded to mountall 2.14 from PPA, but the boot still fails in more the attempts. The bug is kind of tricky, because after the upgrade, it seemed as if the issue was fixed, but in subsequent boots the problem reappeared.

* The bug does not apply only to /var, but it can be any other filesystem that is on LVM.

* The issue is somehow correlated with another issue with the permissions of /dev/shm. Each time the boot fails and I have to enter the recovery shell and execute "mount -a" and CTRL-D to continue with the boot, the permissions of /dev/shm are messed up and Google Chrome Browser fails to start. It is a quite strange issue, because I have checked the permissions in the recovery shell and the permissions are OK (rwxrwxrwt), but after graphical login the permissions are suddenly rwx-r-x-r-t. The issue with the permissions does not occur, when once in a while the boot works correctly.

I am sending the boot.log, but I there is no /var/log/udev on this box. How can I activate logging for udev?

Oops, sorry /var/log/udev was sitting there, but I had not seen it... I am attaching the log for udev for the same boot attempt...

I've done some more testing trying to isolate the bug. I generated copies of the boot.log and udev log files in the recovery shell, to leave out log entries from the boot process after leaving the recovery shell.

The /etc/fstab contains the following file systems:
/dev/mapper/vg0-lvRoot / ext3 relatime,errors=remount-ro 0 1
/dev/sda1 /boot ext3 relatime 0 2
/dev/mapper/vg0-lvHome /home ext3 relatime 0 2
/dev/mapper/vg0-lvOpt /opt ext3 relatime 0 2
/dev/mapper/vg0-lvUsr /usr ext3 relatime 0 2
/dev/mapper/vg0-lvVar /var ext3 relatime 0 2
/dev/mapper/vg0-lvPhoto /home/media/photo ext3 relatime 0 2
/dev/mapper/vg0-lvMusic /home/media/music ext3 relatime 0 2
/dev/mapper/vg0-lvDownload /home/download ext3 relatime 0 2
/dev/mapper/vg0-lvVM /vm ext3 relatime 0 2

The /opt and /vm filesystem did not get mounted on this boot attempt:/dev/mapper/vg0-lvRoot on / type ext3 (rw,relatime,errors=remount-ro)
/dev/sda1 on /boot type ext3 (rw,relatime)
/dev/mapper/vg0-lvHome on /home type ext3 (rw,relatime)
/dev/mapper/vg0-lvUsr on /usr type ext3 (rw,relatime)
/dev/mapper/vg0-lvVar on /var type ext3 (rw,relatime)
/dev/mapper/vg0-lvPhoto on /home/media/photo type ext3 (rw,relatime)
/dev/mapper/vg0-lvMusic on /home/media/music type ext3 (rw,relatime)
/dev/mapper/vg0-lvDownload on /home/download type ext3 (rw,relatime)

I am attaching the boot.log.recshell and udev.recshell file, that reflect the state of these log files after dropping to the recovery shell prompt.

Given the fact that the mounting of partitions and the boot process still fails half the time can the status of the issue be updated?

On Sat, 2010-04-24 at 16:36 +0000, Ali Onur Uyar wrote:

> I am sending the boot.log, but I there is no /var/log/udev on this box.
> How can I activate logging for udev?
>
If boot fails, and you have /var on a separate partition, this can often
be found as /dev/.udev.log -- please attach that

Scott
--
Scott James Remnant
<email address hidden>

On Sat, 2010-04-24 at 16:36 +0000, Ali Onur Uyar wrote:

> ** Attachment added: "boot.log"
> http://launchpadlibrarian.net/45203739/boot.log
>
Would I be correct that, on this boot, it was /opt that was not mounted?

Scott
--
Scott James Remnant
<email address hidden>

Ali: thanks for your debugging work so far.

Could you boot up to the recovery shell, and then run:

  udevadm info --export-db

Capture the output, and attach it to this bug (along with the boot.log
and udev log of the same boot)

 status incomplete

Scott
--
Scott James Remnant
<email address hidden>

Hi Scott, I will be glad to help as much as I can.

I am sending you the following information in the attached zip file:
udevadm_info.recshell - Output of 'udevadm info --export-db' in recovery shell.
boot.log.recshell - /var/log/boot.log from recovery shell.
udev.recshell - /var/log/udev from recovery shell.
dev_mapper.recshell - Permissions of LV devices 'ls -l /dev/mapper' in recovery shell.
fs_ext3.recshell - List of mounted filesystems in recovery shell.
dev_shm.afterboot - Permissions of /dev/shm after mounting filesystems manually and continuing with boot with CTRL-D.

If you compare the filesystem list with the report I has sent before, you will see that more filesystems failed to mount on this occasion. On rare occasions all filesystems are mounted without problems and the boot works smoothly. On other occasions only two filesystems do not get mounted and on other cases like this one 5 filesystems might fail to get mounted.

The filesystems that fail to get mounted are always the ones on LVM with the device files that have root:disk ownership.

The /dev/shm permission problem is correlated with the bug. Whenever there is a problem mounting the filesystems on LVM, the permissions of /dev/shm also get messed up after /boot.

Ali: I don't suppose you can grab "dmesg" output as well?

Scott
--
Scott James Remnant
<email address hidden>

On Mon, 2010-04-26 at 00:36 +0000, Ali Onur Uyar wrote:

> The filesystems that fail to get mounted are always the ones on LVM with
> the device files that have root:disk ownership.
>
Do the filesystems that *do not* get mounted have different ownership?

> The /dev/shm permission problem is correlated with the bug. Whenever
> there is a problem mounting the filesystems on LVM, the permissions of
> /dev/shm also get messed up after /boot.
>
While this may be true, right now I simply have no explanation for how
the permissions of this filesystem could get "messed up" - it's a useful
indicator for whether we fix it I guess.

Scott
--
Scott James Remnant
<email address hidden>

On Mon, 2010-04-26 at 00:36 +0000, Ali Onur Uyar wrote:

> boot.log.recshell - /var/log/boot.log from recovery shell.
>
Unfortunately it looks like this has the contents from the *previous
boot* (since it shows the boot completing); could you finish booting,
and then attach the "new" /var/log/boot.log

Thanks

Scott
--
Scott James Remnant
<email address hidden>

Thanks to the contributed debugging information, I think I've tracked down this bug. Since mountall had a bug where it was overflowing events, I think that may have diluted the debugging information at first, and that has now been fixed.

The remaining bug is in lvm itself.

In my PPA you'll find a new dmsetup package, could you try it out and see whether it makes things better or worse?

  sudo add-apt-repository ppa:scott/ppa
  sudo apt-get update
  sudo apt-get upgrade

Check you have dmsetup 2.02.54-1ubuntu4~ppa1

  dpkg-query -W dmsetup

Then reboot.

Hi Scott,

I upgraded the device-mapper packages from your PPA repo and I've done some testing.

I've done five reboots and one shutdown-restart cycle for testing without any boot problems so far; seems like things have definitely gotten much better; congratulations. :-)

All LV device nodes have root:disk ownership and the permissions of /dev/shm are correct.

If you still need the dmesg output, please let me know.

On Mon, 2010-04-26 at 03:29 +0000, Ali Onur Uyar wrote:

> I upgraded the device-mapper packages from your PPA repo and I've done
> some testing.
>
> I've done five reboots and one shutdown-restart cycle for testing
> without any boot problems so far; seems like things have definitely
> gotten much better; congratulations. :-)
>
> All LV device nodes have root:disk ownership and the permissions of
> /dev/shm are correct.
>
> If you still need the dmesg output, please let me know.
>
Oh good.

Thanks for all your help. I'll upload this to main archive and make the
Release Manager's day (they hate respins on release week)

Scott
--
Scott James Remnant
<email address hidden>

This bug was fixed in the package lvm2 - 2.02.54-1ubuntu4

---------------
lvm2 (2.02.54-1ubuntu4) lucid; urgency=low

  * Some idiot thought it'd be a good idea if device mapper didn't respond
    to "add" events, like those during boot. Take their change out back
    and shoot it in the head. LP: #561390.
 -- Scott James Remnant <email address hidden> Sun, 25 Apr 2010 21:36:25 -0700

Just tried uploaded packages, problem seems to be gone ! Thanks

was having the same problem here with /data refusing to mount at boot

added your ppa and updated.

dpkg-query -W dmsetup
shows:
dmsetup 2:1.02.39-1ubuntu4

dpkg-query -W mountall
shows:
mountall 2.15~ppa1

however when i boot i still get message to skip or go to maintenance shell but if i skip it goes to maintenance shell and gives message that there's a general error mounting filesystems and i'm unable to boot at all.

have edited fstab to prevent loading the problematic drive but need help actually identifying why this is happening please.

this is what my fstab currently contains:

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda6 during installation
UUID=c83c45b9-ae13-406a-bcf5-09509651fd84 / ext4 errors=remount-ro 0 1
# /data was on /dev/sda8 during installation
#UUID=0aae64be-4f6d-4a12-b1f1-606c45030d76 /data ext4 defaults 0 2
# /home was on /dev/sda7 during installation
UUID=2be3d35d-3107-4319-ac0a-4ecf9e4fdc97 /home ext4 defaults 0 2
# swap was on /dev/sda5 during installation
UUID=6d7cb062-c9f0-4cd9-a936-be440226c5a4 none swap sw 0 0

/data is the problem and when boot goes to the maintenance shell it appears to be telling me that UUID=0aae64be-4f6d-4a12-b1f1-606c45030d76 doesn't exist...

well for some reason the UUID for /data was wrong in my fstab (no idea how this could have been as i had done nothing to it before this and i thought the whole point of a uuid was that it was unique to that device) have changed the uuid listed and now works fine.

Hmm, today, I rebooted a server we have with Lucid and got this exact problem, even with latest package that is supposed to fix it.

I rebooted it again and it booted without problems, so It smells like this bug is not completely fixed... (root partition is on a LVM2 partition).

On Tue, 2010-04-27 at 09:10 +0000, Carlos Perelló Marín wrote:

> Hmm, today, I rebooted a server we have with Lucid and got this exact
> problem, even with latest package that is supposed to fix it.
>
Please run "dpkg-query -W dmsetup" and report the version returned.

Scott
--
Scott James Remnant
<email address hidden>

Hi.

I came from bug #557909.
It seems to be solved after I perform a simple upgrade.
Thank you Scott.

Regards

I seem to be having a similar issue, although with a mdadm raid 5 array. I get the screen stating 'the disk for/mnt/md0 is not ready yet or not present'. If i press M and try 'mount -a' it states that 'special device /dev/md0 does not exist'. If I try 'mdadm --assemble --scan' it states 'no devices found for /dev/md0'. If I skip the mounting of the array, and start the array then mount it through palimpset everything mounts as should be (devices in array and mount location).

I have added Scott's repositories and upgraded mountall to no avail. The output of 'dpkg-query -W dmsetup' is 'dmsetup 2:1.02.39-1ubuntu4'. This seems different from the version I should have. I have included my /var/log/boot.log. I will attach the udev file from the same directory in my next post (I cant attach two files to this post)

On Sat, 2010-05-01 at 10:25 +0000, teeedubb wrote:

> I seem to be having a similar issue, although with a mdadm raid 5 array.
> I get the screen stating 'the disk for/mnt/md0 is not ready yet or not
> present'. If i press M and try 'mount -a' it states that 'special device
> /dev/md0 does not exist'. If I try 'mdadm --assemble --scan' it states
> 'no devices found for /dev/md0'. If I skip the mounting of the array,
> and start the array then mount it through palimpset everything mounts as
> should be (devices in array and mount location).
>
This means your RAID array has not been activated, so is a different
issue to this bug -- you should file a bug on mdadm

Scott
--
Scott James Remnant
<email address hidden>

Could this bug be affecting mounts on a Coraid SR1521 AoE etherdisk too? See bug report: https://bugs.launchpad.net/bugs/575053

Thanks,
/mysteron

Description of bug fix:

"Some idiot thought it'd be a good idea if device mapper didn't respond to "add" events, like those during boot. Take their change out back and shoot it in the head. LP: #561390."

You are publicly calling a fellow developer an idiot... I think this is extremely unprofessional behaviour! (Not to mention that it is borderline libel.)
What you think privately is your own concern, but you don't go around publicly calling someone else an "idiot"!

This is the relevant file portion:

# What idiot thought this was a good idea?
#ACTION=="add", ENV{STARTUP}!="1", NAME="", GOTO="dm_end"

I suppose you had no idea. Please do some research before jumping the gun.

On Thu, Oct 07, 2010 at 07:05:36AM -0000, Kelsey Thornton wrote:
> "Some idiot thought it'd be a good idea if device mapper didn't respond
> to "add" events, like those during boot. Take their change out back and
> shoot it in the head. LP: #561390."

Unless the kernel has been customised, udev add events issued by dm tell
userspace nothing useful and must be ignored. Change events are the ones
to act upon as they indicate that there is a device available for use.
If you're having problems in this area, take a look at how the latest
versions of other distros have packaged the upstream releases and made it all
work together successfully e.g. Fedora/RHEL or SuSE (but not Debian).

Alasdair

I have to agree with the sentiment of Kelsey. Regardless of how bad the original code was, it does not warrant the sort of written statement that appeared in the changelogs. Please keep in mind the Code of Conduct that you have signed and treat everybody with respect.

In particular, these changelog messages appear during the system updates and a statement like "take their change out back and shoot it in the head" should never appear to an end-user.

The "idiot" message showed up in the Update Manager yesterday in Lucid. Whatever the reasons are (code or humour) this is embarrassing for the Ubuntu community and for a professional product. The way it shows up it looks like a clear breach of the Ubuntu Code of Conduct. I would suggest editing the changelog to get rid of it.

"Idiot" word was the only reason why I opened this page. This is very unprofessional...

Hi,

This is a bugreport not a discussion forum.

If you have a complain about the behavior of any ubuntu developer I'm pretty sure there are better ways than this one.

http://bit.ly/aro1lo

I hope that i'm not the only one that realized that Scott was saying "idiot" to himself, am i?

The comment on the commit log was supposed to make fun of himself, there's no CoC issue here ;)

I have an ubuntu 10.04 server with text mode and find out the following output:
cat /var/log/syslog | grep device-mapper
Apr 7 11:57:58 server01 kernel: [ 30.590688] device-mapper: table: 251:2: crypt: Device lookup failed
Apr 7 11:57:58 server01 kernel: [ 30.590869] device-mapper: ioctl: error adding target to table
Apr 7 11:57:58 server01 kernel: [ 30.599411] device-mapper: ioctl: asked to rename a non existent device cryptswap1_unformatted -> cryptswap1

I have read all posts in these bug report
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/527666/comments/112

and then directed to this bug report
https://bugs.launchpad.net/bugs/561390

I have found out by chance this morning the above massages disappeared totally after I plus "#" in front of the line "cryptswap1 /dev/md2 /dev/urandom swap, cipher=aes-cbc-essiv:sha256."
This line was written in the configuring file /etc/crypttab which I have been reported it as a bug solution in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/206129/comments/51

Now the booting process is OK and the swap mounted in /etc/fstab on Raid1 by uuid work well.

Thank all of you !

I have similar symptoms on a natty installation (upgraded), see #796755.

I have this issue of "open: Permission denied Error locking counter" when trying to mount my home folder from a live sesion. I used the guide on https://help.ubuntu.com/community/EncryptedPrivateDirectory.