Comment 10 for bug 1506139

Revision history for this message
Hadmut Danisch (hadmut) wrote :

OK,

I have debugged this down and got big steps further in identifying the problem.

An important step for debugging was to learn how to debug systemd.

    http://freedesktop.org/wiki/Software/systemd/Debugging/

was quite helpful, that

   systemctl enable debug-shell.service

helps a lot. After that, one can get a root shell when the systemd boot process is hanging.

I have identified *two* problems, both in
/lib/systemd/systemd-cryptsetup

First problem:

The system boot procedure hangs because the process

    /lib/systemd/systemd-cryptsetup attach sda2_crypt /dev/disk/by-id/ata-SanDisk_SDSSDHII480G_**********-part2 none luks,swap,discard

hangs. It waits for password input, but for some reason it's prompt and input don't make it their way to the boot console or boot splash prompt. There's a problem with the procedure for requesting a password.

Killing that process from the debug console makes the boot process continue immediately (of course without working swap).

Once knowing that this is the process causing trouble, debugging get's much easier, since it is not required anymore to try this within a boot process. You can use a running machine with any test partition for easy debugging.

BTW: systemd does not use /etc/crypttab directly, but converts the contents of /etc/crypttab to dynamically created units first, which can be found under /run/systemd. It shows

ExecStart=/lib/systemd/systemd-cryptsetup attach 'sda2_crypt' '/dev/disk/by-id/ata-SanDisk_SDSSDHII480G_**********-part2' 'none' 'luks,swap,discard'
ExecStop=/lib/systemd/systemd-cryptsetup detach 'sda2_crypt'
ExecStartPost=/sbin/mkswap '/dev/mapper/sda2_crypt'

So one knows what happens right here.

You can easily call the given command from anywhere as root with any partition, without the need to edit /etc/crypttab, because it's all command line parameters here. Makes testing pretty easy now.

Second problem:

That damned systemd-cryptsetup ignores luks (or is unable to cope with modern luks settings).

That's what the dmsetup looks like for my root partition setup in the initramfs:

0 903712768 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 8:3 4096 1 allow_discards

This looks good, because it's the same crypt-parameters (aes-xts) as I used when creating the luks partition, and it uses an offset of 4096, allowing the luks header to remain untouched.

But after running that systemd-cryptsetup for the sda2 partition (even after freshly partitioning it with cryptsetup), dmtable shows that:

0 32616448 crypt aes-cbc-essiv:sha256 0000000000000000000000000000000000000000000000000000000000000000 0 8:2 0 1 allow_discards

which contains *two* wrong settings:

- it's the wrong cipher

- it's an offset of 0, which overwrites the luks header. That's why I am seeing garbage again and again.

So it turns out that systemd-cryptsetup is tripple-buggy:

- Password dialog not working in boot process, neither in splash or non-splash mode (that's why boot process hangs)

- wrong cipher

- no offset, thus overwriting the luks header.