[lucid] plymouth in initramfs doesn't know to chroot() when init does, can't load files from disk

To answer Steve's question on #504052, here's the version of cryptsetup, mountall and plymouth I'm running:

stgraber@castiana:~$ dpkg -l | grep -E "cryptsetup|mountall|plymouth"
ii cryptsetup 2:1.1.0~rc2-1ubuntu9 configures encrypted block devices
ii libplymouth2 0.8.0~-7 graphical boot animation and logger - shared
ii mountall 2.4 filesystem mounting tool
ii plymouth 0.8.0~-7 graphical boot animation and logger - main p

> Pressing ESC, I get out of the splash and can see that cryptsetup is trying to unlock
> /dev/sda2 but never prompts me for the passphrase. As a result, the laptop simply can't
> boot.

Oh, interesting. The code in cryptsetup to select whether to prompt via plymouth is here in /lib/cryptsetup/cryptdisks.functions:

                if [ -x /bin/plymouth ] && plymouth --ping; then
                        KEYSCRIPT="plymouth ask-for-password --prompt"
                        keyscriptarg=$(echo -e "$keyscriptarg")
                else
                        KEYSCRIPT="/lib/udev/watershed /lib/cryptsetup/askpass"
                fi

Is /dev/sda2 a luks volume, or a non-luks volume? (There are separate code paths for each) Could you try adding some debug statements to this code, to find out what's preventing cryptsetup from talking to plymouth?

BTW, I had tested that plymouth prompting for non-root devices works here; but I only tested with a LUKS device and it was on a system that also had root on LUKS, so perhaps one of these variables masked the problem.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

/dev/sda2 is a LUKS volume containing a LVM with one VG and one LV, my
/home being that LV.

I'll attach some debug information from the cryptsetup scripts, probably
in a few hours.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAktWMOoACgkQjxyfqkjBhuwY3wCeOr2SRdVYHILmvUmrIv6bUNVb
x+IAnjZPG4aweeDqv22JaiVx5oEE8oVS
=NS/w
-----END PGP SIGNATURE-----

Ok, I did some tests, I modified both upstart jobs /etc/init/cryptdisks-* to record a full shell trace to a log file, then rebooted.
As usual I didn't get any kind of prompt from plymouth but assumed that it was waiting for the key anyway and so entered it.

Looking at the -udev trace, we can see that the passphrase was actually accepted (although I didn't get any kind of feedback from plymouth).

I'm also attaching my fstab, crypttab and /etc/default/cryptdisks.

Feel free to ask for any more information.

+ KEYSCRIPT=plymouth ask-for-password --prompt
+ echo -e Unlocking the disk /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 (sda2_crypt)\nEnter passphrase:
+ keyscriptarg=-e Unlocking the disk /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 (sda2_crypt)
Enter passphrase:
+ PARAMS= --key-file=-
+ [ 0 -lt 3 ]
+ [ -n plymouth ask-for-password --prompt ]
+ plymouth ask-for-password --prompt -e Unlocking the disk /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 (sda2_crypt)
Enter passphrase:

Hmm, well, this part is a bashism I introduced, oops :/ I'll get that fixed ASAP.

Do you want to test whether replacing "echo -e" with "printf" here in /lib/cryptsetup/cryptdisks.functions, to see if that solves your problem?

+ cryptsetup --key-file=- luksOpen /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 sda2_crypt_unformatted
error: unexpectedly disconnected from boot status daemon

Hmm, this is a concern, showing up in log.enable - but it could be down to the known lack of locking between these jobs; if the -udev job gets there first (as expected), and successfully unlocks the disk, the -enable job will still be waiting for input when the boot finishes and plymouth shuts down. So I guess that doesn't explain the problem, either.

Please check whether s/echo -e/printf/ fixes this. If it doesn't, then we'll have to dig some more.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I replaced both "echo -e" by "printf" and rebooted, I still get the
exact same behavior. Then I tried moving cryptdisks-enable.conf away for
a test boot but it didn't change anything either. I still don't get
prompted by plymouth and entering a valid key unlocks sda2 but gets
stuck just afterward and won't boot.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAktYQcIACgkQjxyfqkjBhuxHjACdHx9GXBI5gLNCeykNZNV+jU/Y
aEsAn3bI1p9aGPi80sdjAyPE4v3zHTc9
=F2oN
-----END PGP SIGNATURE-----

Just uploaded a new version of cryptsetup that fixes the echo -e bug, and also stops cryptsetup-enable from double-prompting (invisibly) for the same disk. That won't fix the root problem here, but may make debugging easier.

After upgrading, can you edit /etc/init/cryptdisks-udev.conf to attach to the running plymouth process with strace, to see what it's doing when it should be writing prompts? I wonder if it's trying to load plugins from disk, and this is failing because it's in some sort of limbo between the initramfs and the root system.

Something like the following should work (N.B.: untested, may hang your system, and definitely will break if you have more than one disk in /etc/crypttab):

pre-start script
    (strace -ff -e file -s 256 -o /var/run/plymouth-not-prompting -p $(pidof plymouthd) &)
end script
post-start script
    killall strace
end script

Here you go, looking at it, I guess your next question would be this one:
root@castiana:/root# ls -l /lib/plymouth/label.so
-rw-r--r-- 1 root root 10464 2010-01-11 08:52 /lib/plymouth/label.so
root@castiana:/root# ls -l /lib/plymouth/themes/ubuntu-logo/
total 56
-rw-r--r-- 1 root root 870 2010-01-11 08:52 box.png
-rw-r--r-- 1 root root 296 2010-01-11 08:52 bullet.png
-rw-r--r-- 1 root root 350 2010-01-11 08:52 entry.png
-rw-r--r-- 1 root root 1896 2010-01-11 08:52 lock.png
-rw-r--r-- 1 root root 199 2010-01-11 08:52 progress_bar.png
-rw-r--r-- 1 root root 215 2010-01-11 08:52 progress_box.png
-rw-r--r-- 1 root root 233 2010-01-11 08:52 ubuntu-logo.plymouth
-rw-r--r-- 1 root root 28008 2010-01-11 08:52 ubuntu-logo.script

Also, I noticed something weird that I'm not sure is related or not.
When booting with no mountpoint requiring a luks volume (basically how I boot at the moment by commenting the right line in my fast), cryptsetup seems to start anyway in the background and waits for a key. Only issue is that plymouth never asks for that key and boots as if no key was needed at all (it's actually right as no mountpoint require it but then cryptsetup shouldn't start either).

A quick "ps aux" after a boot with all encrypted partitions commented out in my fstab (but the luks volume present in crypttab):
root 1719 0.0 0.0 3980 568 ? S 01:16 0:00 /lib/udev/watershed /lib/cryptsetup/askpass Unlocking the disk /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 (sda2_crypt)\nEnter passphrase:
root 1720 0.0 0.0 19204 2844 ? S<L 01:16 0:00 cryptsetup --key-file=- luksOpen /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 sda2_crypt_unformatted
root 1722 0.0 0.0 3848 444 ? S 01:16 0:00 /lib/cryptsetup/askpass Unlocking the disk /dev/disk/by-uuid/851db151-043d-46fd-9632-626d6f57bb86 (sda2_crypt)\nEnter passphrase:

As I mentioned, plymouth simply completely ignores that and boot to gdm as if nothing happened :)

"by commenting the right line in my fast" should be "by commenting the right line in my fstab"
Just saw a few typos in that comment too, guess it's getting too late for me to comment on Launchpad ;)

Yep, there we are - it's trying to load files from disk, but apparently the process is still using the initramfs as its root, so fails to load any files off the disk past the point of exec()ing upstart.

Since we don't want to force plymouth to waste more cycles in the initramfs reading all the files into memory before run-init unlinks them, we should instead fix this by giving plymouth a way to be notified of the change in root device and chroot at the same time. (In your particular case, it would also work to just ensure cryptsetup doesn't force plymouth into the initramfs - but there are still other reasons to want plymouth to not be fragile in this respect.)

> Also, I noticed something weird that I'm not sure is related or not.
> When booting with no mountpoint requiring a luks volume (basically how
> I boot at the moment by commenting the right line in my fstab),
> cryptsetup seems to start anyway in the background and waits for a
> key. Only issue is that plymouth never asks for that key and boots as
> if no key was needed at all (it's actually right as no mountpoint
> require it but then cryptsetup shouldn't start either).

Commenting it out of the fstab will not prevent cryptsetup from triggering when the device becomes available. For that, you would need to set 'noauto' in the options in /etc/crypttab as well.

opening a task on cryptsetup for the issue of putting plymouth in the initramfs unnecessarily.

It turns out that there *is* chroot handling in plymouth, and /usr/share/initramfs-tools/scripts/init-bottom/plymouth has code to run this. But for some reason, my /scripts/init-bottom/ORDER is missing any references to plymouth:

  /scripts/init-bottom/udev
  [ -e /conf/param.conf ] && . /conf/param.conf

So we need to figure out why that's missing.

This bug was fixed in the package plymouth - 0.8.0~-8

---------------
plymouth (0.8.0~-8) lucid; urgency=low

  [ Steve Langasek ]
  * Include label.so in the initramfs, as well as the font support files it
    depends on, needed for any text prompts or messages. This will bloat the
    initramfs substantially (about 3MB) and impact boot performance as a
    result, but the solution to that is to not include plymouth in the
    initramfs unless we know it's needed for prompting (e.g., cryptsetup).
    LP: #496765.
  * Move x11 renderer into a separate package, to avoid pulling GTK into the
    server install by default. LP: #509579.
  * debian/rules: fix up the dh_makeshlibs exclude path to match where we're
    currently installing.
  * src/plugins/splash/script/plugin.c: the script plugin has no support at
    all for text consoles, so if there are no pixel displays available,
    return an error so that plymouth can fall back to the text plugin.
    LP: #506717.
  * Always include the 'details' and 'text' themes in the initramfs, since
    plymouth uses these as built-in fallbacks and they're tiny.
  * src/main.c: when a splash plugin fails to load, make sure to unregister
    any related keyboard handlers.
  * don't let the password walk off the end of the dialog box. LP: #496782.
  * split the plymouth upstart job into plymouth and plymouth-splash, so that
    plymouth starts up early and doesn't have to race gdm; and stop
    unconditionally adding plymouth to the initramfs now that we can start it
    this way.
  * debian/plymouth.postinst: also set the theme on first install, or on
    upgrade from versions prior to 0.8.0~-7, because update-initramfs called
    before plymouth has been configured for the first time will accidentally
    cause a call to --reset to use the text theme instead of the intended
    default.
  * debian/plymouth.plymouth-splash.upstart: don't wait on tty-device-added,
    these are now guaranteed to be available; copied from gdm upstart job.
  * Depend on mountall (>= 2.0) to ensure we have the above guarantee.
  * Mark /usr/share/initramfs-tools/scripts/init-bottom/plymouth executable,
    otherwise initramfs-tools skips it. LP: #509487.
  * Drop the Debian revision (-1) from the version in the symbols file, to
    fix a lintian warning.
 -- Steve Langasek <email address hidden> Tue, 02 Feb 2010 08:11:47 +0000

I guess that plymouth upload was supposed to close the plymouth part of the issue and not the crypsetup one ?
I did a quick test boot with that plymouth, result is still the same.

Well, fixing the plymouth side of this should have been sufficient to resolve the symptoms. I can confirm here that plymouth now correctly chroots on startup.

Can you grab a fresh strace with plymouth 0.8.0~-8?

My bad, I wasn't running the right version of plymouth.
I just re-tested with 0.8.0~-8 and the result is slightly different.

I now see the passphrase prompt (padlock + label), I can type in my passphrase just fine and when validating it seems it's opening the luks volume but nothing happens and I get back to being stuck on a plymouth splash screen (only the ubuntu logo).

When pressing escape, I see that the luks volume was opened correctly, so that part seems to work now but something still block the boot process.

I'll get you a new trace.

Ok, marking this task fixed again. The hang is probably unrelated and should be filed separately.

So, here's all the calls made by plymouth using the debug code you posted before:
open("/lib/plymouth/themes/ubuntu-logo/bullet.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/box.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/entry.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/lock.png", O_RDONLY) = 12
open("/var/lib/plymouth//boot-duration", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
open("/lib/plymouth/themes/ubuntu-logo/bullet.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/box.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/entry.png", O_RDONLY) = 12
open("/lib/plymouth/themes/ubuntu-logo/lock.png", O_RDONLY) = 12
open("/lib/plymouth/themes/details/details.plymouth", O_RDONLY) = 11
open("/lib/plymouth/details.so", O_RDONLY) = 11
open("/lib/plymouth/themes/ubuntu-logo/bullet.png", O_RDONLY) = 11
open("/lib/plymouth/themes/ubuntu-logo/box.png", O_RDONLY) = 11
open("/lib/plymouth/themes/ubuntu-logo/entry.png", O_RDONLY) = 11
open("/lib/plymouth/themes/ubuntu-logo/lock.png", O_RDONLY) = 11
open("/var/lib/plymouth//boot-duration", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12

Current behavior is:
 - I get prompted for the key (padlock + entry field show up just fine)
 - I enter the key
  - If valid, the prompt disappears
  - If not, I get prompted again as expected
 - It gets stuck there and never does anything

Exitting plymouth (pressing ESC), I can see that crypsetup intializaed correctly and seemed to have unlock the LUKS volume correctly, though it seems it never actually mounted the partition inside it or loaded the LVM making it hang indefinitely there.

Could we track the hang as a separate bug report, as suggested?

What does '/sbin/blkid /dev/mapper/sda2_crypt' show for this device? Once unlocked with cryptsetup, the rest of the mounting should be handled via udev rules and mountall.

I have the same problem. My config:
/ - LVM (PV /dev/sda1)
/home - dm-crypt(/dev/sda2) + LVM (PV sda2_crypt)

When should I run blkid command?
I have currently commented out crypted partition from /etc/crypttab and /etc/fstab and I perform manual cryptsetup + vgchange -ay + mount

Here you go with the blkid for the partition, the LUKS volume and the LVM partition:

stgraber@castiana:~$ sudo blkid /dev/sda2
/dev/sda2: UUID="851db151-043d-46fd-9632-626d6f57bb86" TYPE="crypto_LUKS"

stgraber@castiana:~$ sudo blkid /dev/mapper/sda2_crypt
/dev/mapper/sda2_crypt: UUID="m73M8R-p6Hn-okx2-xFFQ-eEKl-lKbZ-dq0qgK" TYPE="LVM2_member"

stgraber@castiana:~$ sudo blkid /dev/mapper/castiana-home
/dev/mapper/castiana-home: UUID="3378fec9-7db2-4a40-90cf-168df7e93deb" TYPE="ext4"

> /dev/mapper/sda2_crypt: UUID="m73M8R-p6Hn-okx2-xFFQ-eEKl-lKbZ-dq0qgK" TYPE="LVM2_member"

This should be sufficient to trigger /lib/udev/rules.d/85-lvm2.rules, which I assume you have on your system (since it's from the lvm2 package!). If you can run the commands from that rule's RUN variable by hand, it may be instructive; or to otherwise trace the set of processes running at this point. A good way to do this may be to edit /etc/init/tty2.conf to force a getty to start earlier in the boot sequence.

I have a similar issue without LVM.
I installed from netinstall today.

i have
/boot > /dev/sda1 ext2
/ > /dev/sda2 ext4
/home > /dev/sda3 crypt setup ext4

/home is ext4 partition over LUKS encrypted partition

Please note that Marc Gariepy works with me and lives something like 2 minutes away from me :)
I've been seeing the exact same behavior on his laptop as on mine though his is a perfectly clean Lucid install and doesn't use LVM (instead the ext4 /home partition is directly in the LUKS volume).

At boot time, we get the passphrase prompt correctly, after entering the key the prompt disappears and we get stuck there (as on my laptop).
Commenting the /home line in /etc/fstab and rebooting will make the laptop boot successfully though he'll then have to manually do the "cryptsetup luksOpen" and "mount".

similar issue reported in bug 497480

this is fixed with mountall 2.10. on lucid.