cryptsetup devices not mounted on boot

I have targeted this for alpha-6 just for consideration, as this regression was caused by recently added mountall. Feel free to drop from milestone if you feel it does not impact enough users (due to specific config).

I had this same problem but with my NFS based /home. This problem appeared today after I upgraded some packages. Now I mount /home manually and everything works fine except that mounting home manually sucks donkey balls.

This also seems to affect a /home on a regular old LVM volume (no encryption, nothing special). During the boot sequence, udev, complains about some bits of its files which it doesn't understand. "%k" warnings and "SYMLINK" errors. I'll see if I can get a photo of the boot screen if that'll help.

Isn't this just a case of mountall waiting for devices to appear when they actually need extra help to appear? (vgchange -a y, comes to mind for LVM.)

On Wed, 2009-09-16 at 11:19 +0000, Henrik Heino wrote:

> I had this same problem but with my NFS based /home. This problem
> appeared today after I upgraded some packages. Now I mount /home
> manually and everything works fine except that mounting home manually
> sucks donkey balls.
>
Henrik: if you are having issues with NFS, that is a different bug to
this one which specifically concerns LUKS. Please file a new bug.

Scott
--
Scott James Remnant
<email address hidden>

On Wed, 2009-09-16 at 17:43 +0000, Bardur Arantsson wrote:

> This also seems to affect a /home on a regular old LVM volume (no
> encryption, nothing special).
>
Bardur, if you are having problems with LVM that is a different bug to
this (which only affects LUKS). Please file a new bug.

Scott
--
Scott James Remnant
<email address hidden>

The cryptsetup issue is that these are enabled by an initscript, and initscripts aren't run until filesystems are mounted.

Chicken.
Egg.

Ideally we would convert cryptsetup to be run from a udev rule (as it always should have been).

A short term solution would be to run it as-is after the udevtrigger job completes

I'm adding lvm to this bug as LVM+LUKS is a configurable option when installing ubuntu using the alternative CD. IMHO this is even a common setup for notebooks, as I do have it on my notebook. So this use case is broken completely currently.

This bug was fixed in the package cryptsetup - 2:1.0.6+20090405.svn49-1ubuntu3

---------------
cryptsetup (2:1.0.6+20090405.svn49-1ubuntu3) karmic; urgency=low

  * debian/cryptdisks-enable.upstart: add upstart job to enable encrypted
    disks once we've finished probing for udev devices, so that mountall
    can use them. LP: #430496.

 -- Scott James Remnant <email address hidden> Thu, 17 Sep 2009 00:04:00 +0100

fixed in cryptsetup, mountall was just waiting for cryptsetup to get around to decrypting them

nothing to do with LVM

Scott, I have to reopen this bug, as this does not solve the problem with /home on dm-crypt+LVM.
Main problem seems to be that now I get asked for my password several times in the row, even though that I enter the correct password (as I can see it in recovery mode as it is printed back to me). Also, /home is still not mounted after that, looking like mountall still hangs.

If I comment out /home in /etc/fstab, the boot continues (after being asked for crypt password several times), but this asking for password does seem to confuse console #1 completely, as for example in recovery mode, I cannot even navigate menu to select going to root shell to manually mount the file system.
Currently my workround is that I boot with commented out /home with regular boot, then in KDM, I switch to console #2 (as in console #1 there are still cryptsetup asking for password and keyboard does not work for login) and login, mount manually my /home and the switch back to KDM and login.
If there is anything I can do to help debug, just ask.

Since this upgrade, my machine now hangs eternally on "starting sad crypto disks...". I have cryptsetup installed, but I don't actually have internal encrypted disks. Purging cryptsetup just brought me back to "blank/powered-off screen for more than a minute".. I'll just reinstall this machine from scratch.

Damn, the cryptsetup kludged worked for me in the setup I created.

I guess we need to figure it out properly; I would appreciate help from people who actually know and use this stuff.

Basically how it *should* work is that there should be a udev rule or upstart job (maybe the latter since it has to block for input) that detects an encrypted block device, asks for the passphrase, and then creates the unencrypted block device.

(udev will then see the unencrypted device, and mountall will be able to mount it)

Changing to critical as it keeps a service from starting

This bug was fixed in the package cryptsetup - 2:1.0.6+20090405.svn49-1ubuntu4

---------------
cryptsetup (2:1.0.6+20090405.svn49-1ubuntu4) karmic; urgency=low

  * debian/cryptdisks-enable.upstart: Things that often help include
    not setting stdin/out to /dev/null, so you can actually type the
    passphrase. I am an idiot. LP: #430496.

 -- Scott James Remnant <email address hidden> Thu, 17 Sep 2009 17:58:01 +0100

> debian/cryptdisks-enable.upstart: Things that often help include
> not setting stdin/out to /dev/null, so you can actually type the
> passphrase. I am an idiot. LP: #430496.

Scott, no: you rock! Because you dare to change this old kludge of sysvinit and moving us in new world with upstart.

Everything works now with my /home on dm-crypt+LVM and this is for the first time in Karmic cycle. Thanks!
I will also close my older (pre upstart job).

On Thu, 2009-09-17 at 19:02 +0000, Luka Renko wrote:

> > debian/cryptdisks-enable.upstart: Things that often help include
> > not setting stdin/out to /dev/null, so you can actually type the
> > passphrase. I am an idiot. LP: #430496.
>
> Scott, no: you rock! Because you dare to change this old kludge of
> sysvinit and moving us in new world with upstart.
>
> Everything works now with my /home on dm-crypt+LVM and this is for the first time in Karmic cycle. Thanks!
> I will also close my older (pre upstart job).
>
This may not fix it for everybody, but I would appreciate it if those
people could open new bugs individually rather than commenting here.

With so many different fstab and crypttab combinations, while they often
sound like the same bug, the fixes are often quite different.

Scott
--
Scott James Remnant
<email address hidden>

scott: see bug #431040, please. I'd like to help (especially since it's made my laptop unusable!).

It seems to me that it's not completely fixed. I have usplash disabled ("splash" option removed from GRUB configuration) and still experience this problem. When I to re-enabled splash mounting encrypted /home went fine. Scott: Could you check it?

Swistak [2009-10-23 20:51 -0000]:
> It seems to me that it's not completely fixed. I have usplash disabled
> ("splash" option removed from GRUB configuration) and still experience
> this problem. When I to re-enabled splash mounting encrypted /home went
> fine.

usplash is not at all related to this original blkid fix. Can you
please give us the output of "sudo blkid -p /dev/affected_device"?
(Find it out with "mount" in a working system).

Here's the output:
/dev/sda6: UUID="9d5d03d5-85ba-4a4e-9fbe-e87c650cd800" VERSION="256" TYPE="crypto_LUKS" USAGE="crypto"
It's my /home partition.

Basically when usplash is on I'm asked to type the password and after doing that system boots correctly, even though I get something like "Could not mount all entries in /etc/fstab".
With usplash off I'm asked for password several times and it has no effect.

So the blkid output is correct, thanks. Seems that the cryptsetup mount scripts or mountall break things then.

This is not quite fixed, yet.
I encounter the following issues :
* the startup scripts do not always succed in correctly unlocking the keyslot. Looks like sometimes the keyboard
input does not get to the password reader but to something else.
* And if you can switch to console 8 unfortunately one can even see the password typed in the first console where the luks password was asked; This is not ok first because the password does not get where it should get, and second because it creates a security risk as anyone could press (ctrl)-alt-f8 and see the password for the encrypted disk(s).
* if cryptdisks is enabled, all other services should depend on cryptdisks, nothing should continue until the user either correctly unlocks all the encrypted volumes OR he cancels the operation or he fails the password for the numtries parameter in crypttab.
NOW, for example gdm starts anyway. Needless to say, this create an issue when you are trying to enter the cryptovolumes password and then gdm takes over the screen and the keyboard.
* The tty's on the other side seem to wait for the cryptdisks. This creates the following issue : If you did not manage to properly enter the password, after gdm started you could try to switch back to console 1 and retry to enter the password. However, because of how the dependencies are made, you are presented with a blank screen. If you press enter a few times, eventually the startup sequence completes (i think it actually terminates cryptdisks init script) and so the other services get to be started (getty for ex).

Just reproduced this on a fresh install in a VM:
- made a clean install;
- cryptsetup luksFormat /dev/somepartition
- added esomepartition /dev/somepartition none luks
in /etc/crypttab
-added /dev/mapper/esomepartition in /etc/fstab
rebooted.
After reboot, the startup scripts get to the point where a password for the encrypted partition is asked.
However, after just a few keystrokes gdm overtakes the screen and the keyboard.
This is a quite simple and straightforward scenario, has anyone tested these things ?!

PS :
As gdm overtook the keyboard and screen, it seems that the init scripts somehow hang, and other things like for example the tty's do not start at all. You'll have to get back to tty1 and send a few enters to unhang the sequence.

I have home and swap as cryptdisks and after the upgrade to karmic today I don't get the passphrase prompt at all, the boot just hangs at:

One or more of the mounts listed /etc/fstab cannot yet be mounted:
(ESC for recovery shell)
/home: waiting for /dev/mapper/home
swap: waiting for /dev/mapper/swap
/var/tmp: waiting for tmpfs

If I enter the recovery shell and run "/etc/init.d/cryptdisks start" I get the prompt and can then type "mount /home" and "swapon /dev/mapper/swap" to successfully mount those partitions. However, CTRL-D returns to output the following:

mountall start/starting
One of more of the mounts listed in /etc/fstab cannot yet be mounted:
(ESC recovery shell)
swap: waiting for /dev/mapper/swap
/var/tmp: waiting for tmpfs

(home isn't listed that time)

Does anyone know of a way to at least get this system to boot for the time being?

OK, commenting out the swap partition and the two tmpfs mountpoints from /etc/fstab allows the boot sequence to continue after interrupting mountall to run "/etc/init.d/cryptdisk start; mount /home". At least I can use my system while we sort this out.

> swap: waiting for /dev/mapper/swap
> /var/tmp: waiting for tmpfs

Please show /etc/fstab and /etc/crypttab.

Marc

Marc A. Donges wrote:
>> swap: waiting for /dev/mapper/swap
>> /var/tmp: waiting for tmpfs
>>
>
> Please show /etc/fstab and /etc/crypttab.
>
> Marc
>
>
It is my fault, sorry. The bug is not the above.
I am user. Yesterday, i tried to upgrade the ubuntu 9.04
to 9.10. All things were going right, but when finished
the installation, the ethernet card stoped. The informations
of my ethernet card is :

09:00.0 Ethernet controller: Attansic Technology Corp. Atheros
AR8121/AR8113/AR8114 PCI-E Ethernet Controller (rev b0)
        Subsystem: Acer Incorporated [ALI] Device 015e
        Flags: fast devsel, IRQ 17
        Memory at be100000 (64-bit, non-prefetchable) [size=256K]
        I/O ports at 3000 [size=128]
        Capabilities: [40] Power Management version 2
        Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-
        Capabilities: [58] Express Endpoint, MSI 00
        Capabilities: [6c] Vital Product Data <?>
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [180] Device Serial Number ff-8b-23-00-b6-2b-16-ff
        Kernel modules: atl1e

The last kernel, wich the above card were working, is 2.6.28-16 for
64bit version.

The upgrade install the kernel 2.6.31-14. That kernel has not drivers
for my ethernet controller.
I was looking for that problem and find the next :

http://ubuntuforums.org/showthread.php?t=770173&page=1

but when download the appropriate source files and try to compile
receive error
for 'struct' of data.

I think it is useful to describe the problem and it will fixed at the
feauter, because
i think the ubuntu is very nice OS and it must be at the top...

Thanks.

On Sat, Oct 31, 2009 at 10:12:35PM -0000, Blue wrote:
> This is a quite simple and straightforward scenario

Well, no, not really. To have run into the problem you did, you must have
devices configured in your /etc/crypttab that are being enabled at boot
time, but that are not used as the underlying devices for any core
filesystems, including /, /usr, /var, or /home (or even swap). I think
that's quite an unusual setup.

To work around this issue, you can mark the filesystem that sits on the
encrypted volume as "bootwait" in your /etc/fstab, as documented in the
release notes:
<http://www.ubuntu.com/getubuntu/releasenotes/910#Login%20screen%20presented%20before%20optional%20filesystems%20are%20mounted>

We could do with an explicit comment there about the need for this with
cryptsetup. Opening a task on ubuntu-release-notes for this.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Steve :
I will try with bootwait and i'll get back to you.
It is actually used for a core fs, but indirectly (mounts somewhere, and then bind mounts from somewhere are created for some user homes)

PS
 Shouldn't this at least be fully consistent ? I mean ok, you do not wait for non core filesystems, but you either do not wait with anything unless bootwait is specified, or you wait with everything ? Now tty's will not start while gdm will start.
One would rather expect tty's to start so that you can log on another console and resolve things, not the gdm to start.
Even more than that, i think that if such a filesystem (non core filesystem) will fail for some reason (for example it had a severe corruption ), then weird things will happen :
- first you WILL need at least a tty to log in and solve the problem
- you must not have gdm steal your display and keyboard
I think tty's should be one of the first things to load, and either way load before gdm (at least if the core file systems are ok).
Now, tty's will wait for any filesystem but gdm won't.

On Mon, Nov 02, 2009 at 08:18:39AM -0000, Blue wrote:

> Shouldn't this at least be fully consistent ? I mean ok, you do not wait
> for non core filesystems, but you either do not wait with anything unless
> bootwait is specified, or you wait with everything ?

No, the default is to wait only for those filesystems that are known to be
required for successfully starting the other services. cryptsetup is a
special case in that it interferes with gdm startup directly, regardless of
where the related filesystems are mounted (and even if they're not mounted
at all by default).

We should try to find a better solution for Lucid on this issue that doesn't
require use of bootwait.

> Now, tty's will wait for any filesystem but gdm won't.

Not true; both gdm and ttys wait for the same set of filesystems, gdm is
just started first whereas ttys are started only after the rest of the init
scripts run.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Sorry, but I think you are wrong. While cryptdisks-early is running (and waiting for a passphrase) tty's will NOT start, however gdm will.

I attached two process lists that prove my previous post. The first listing is the process list as the system boots (and i log on using gdm). It can be seen that the cryptsetup scripts are running, and waiting for a passphrase. It can be seen that no tty nor other daemons like ssh are running.
The second listing is obtained after I manually killed the crypto scripts. It can be seen that both ttys and other daemons started.
I also added a pstree before and after.

Sorry for the previous duplicate file.

...with bootwait added to fstab the thing waits indefinitely for the password, which is good. (it still produces "some filesystem could not be mounted, pres esc for recovery shell,etc" (which is not appropriate in this context), However, at least it waits with every thing untill passphrase is entered.
Without bootwait, the above mentioned scenario happens (gdm starts, ttys do not).
Also, shouldn't the bootwait documented somewhere in the manpages? When I first ran into this I looked up the mount manpage to find such an option, quite like _netdev is mentioned in the same manpage.

"cryptdisks-early" - well, that's not supposed to be responsible for decrypting the disks anyway, we have a native upstart job that's supposed to take care of it. The init scripts were left in the package in Ubuntu 9.10 by mistake; I just don't know if it's safe to try to remove them in SRU.

Yes - you're right that cryptdisks-early will block, and that this will prevent ttys from starting but not prevent gdm from starting. I'll have a look at fixing this as an SRU to karmic.

I think it's not exactly the same issue because I use it with pam_mount (Ubuntu 9.10 Karmic).
I get an error like this:
pam_mount(mount.c:67): Errors from underlying mount program:
As far as I can find out the underlying mount program is cryptsetup. Is this the same issue? Is it related?

See bug #473615 for discussion of removing the init scripts.

Documented at <https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#Optional%20encrypted%20partitions%20must%20be%20marked%20bootwait%20in%20/etc/fstab>:

In addition to the above, users who have configured any encrypted partitions in {{{/etc/crypttab}}} to start at boot time (i.e., not using the {{{noauto}}} option) should make sure that the filesystems on these volumes are listed in {{{/etc/fstab}}} if they are not mounted at a standard system mountpoint. Failure to do this on a desktop system will lead to problems from the X server and {{{cryptsetup}}} trying to control the console at the same time. At best, this will prevent the user from seeing the passphrase prompt; at worst it will also cause the X server to spin and consume 100% CPU. (Bug:430496)

I am also experiencing this with Karmic. I find that I have to enter my passphrase multiple times. Sometimes it doesn't work at all, and I have to Ctrl/Alt/Del, reboot, and then try again. Eventually the system gets my passphrase, but this is clearly a mission critical problem. I haven't seen the password prompt not appear - rather something appears to be eating my keystrokes.

My crypttab is:

  lap1:[~] % cat /etc/crypttab
  # <target name> <source device> <key file> <options>
  crypt-home /dev/sda3 none luks

fstab:

  dev/mapper/crypt-home /home ext3 defaults,relatime 0 2

Will adding 'bootwait' to the fstab options help ?

Pete

Pete,

The 'bootwait' option should have no effect for the /home filesystem.

Could you please file a new bug report against the cryptsetup package for the issue you're describing?

Hi Steve

Just to confirm - you want me to file a bug report on this even though
it sounds like a duplicate of the issues that others are seeing ?

(it doesn't sound like a cryptsetup problem to me - it sounds as if it
is related to the issue of other programs eating the stdin - but i'm no
expert.)

Happy to do this if you think this is a distinct bug.

Pete

Yes, I think we want a new bug report because we know we've already fixed *a* bug here, and we know you're seeing a problem in a case where bootwait makes no difference. That's a different bug than the one we've been tracking, despite having broadly similar symptoms.

Using Kubuntu 9.10 alternate CD.

First time, I encrypted using manual partition during install, which creates /etc/crypttab and /etc/fstab entries for /home and swap.
Same issues on boot, kdm refuses to accept the password, no matter how many times entered.

Second time, I installed only / and swap (using random key so no password required), it gives waiting for swap partition to be initialized, then proceeds with /etc/fstab. Adding a manual encrypted /home (requiring a password) gives the original problem.

This has worked well in previous releases since 2007 June, only Karmic has made some changes which have caused this problem.

Did another bug get open for this (as per #49/#50)? I haven't been able to find one.

I am seeing the same behaviour as comment #25, with a brand new install of 9.10, which I then updated meaning I have cryptsetup 2:1.0.6+20090405.svn49-1ubuntu7.2 and mountall 1.0. If I don't have 'bootwait', gdm starts, but my encrypted partition isn't mounted (and switching back to tty1 there is no sign of a prompt for the passphrase). If I do set 'bootwait' for the partition, the boot stops when mounting fails. A few hits of return sometimes gives a passphrase prompt, which doesn't work, saying the passphrase is wrong, as if it's only seeing the 'return', and not any other keystrokes. After a mixture of repeated attempts (so cryptsetup fails?) or waiting for mountall to timeout (I'm guessing), I eventually managed to get a maintenance prompt, so that I could remove the 'bootwait', and at least get back in to a state where the machine booted. (trying to boot in to single user mode to remove the 'bootwait' didn't work, because it still hit the failing mount).

/etc/crypttab:
# <target name> <source device> <key file> <options>
data /dev/sda6 none luks
cswap /dev/sda5 /dev/urandom swap

/etc/fstab:
/dev/mapper/data /mnt/data ext3 defaults,relatime 1 2
/dev/mapper/cswap none swap sw 0 0

cswap appears to work *sometimes*, but usually only after a huge delay. dmesg for this boot (no 'bootwaits' in fstab) shows me that it came up after 2137 seconds.

BTW, 'data' isn't one of the 'essential' system mount points, but does contain, among other things, /home.

If there is another open bug that this should be posted to instead, let me know.

Cheers,
--Dave

@Dave Check and see if this bug matches your symptoms. I have been having the issues that I mention in the folowing ticket:

https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/473615

If you think this fits what you are experiencing, maybe make a comment there if it is not fixed for you either.

@throughnothing thanks - that bug's related to init scripts, which i didn't *think* was an issue for me, since they're not in /etc/rc2.d/, but then I'm now quite foggy about how the whole boot process works these days. The symptoms certainly match, so I'll post my observations there. Cheers.

The cause (for me) of these problems in Lucid was due to cryptsetup not removing NL/LF characters in the password file. Temporary fix is obviously entering the password in a file without such characters.

echo -n secretpasswordstring > /etc/cryptkey