timeout on restart or shutdown with LUKS root

Bug #1554795 reported by Mason Loring Bliss
54
This bug affects 10 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
Won't Fix
High
Unassigned
systemd (Ubuntu)
Won't Fix
High
Unassigned

Bug Description

Using the server install ISO, it's possible to specify root on LUKS and variations thereof - for instance, root on LUKS on MD-RAID, root on LVM on LUKS on MD-RAID, and so forth. The installer does the right thing and initramfs-tools does everything necessary to support booting this sort of thing.

However, systemd gives a 90-second timeout on restart or shutdown, presumably because it cannot dispose of the things beneath root.

It's wholly unclear to me where the 90-second timeout is specified, should I wish to shorten it to reboot without the futile delay, but more to the point, there seems to be infrastructure for handling this kind of situation that doesn't exist in Ubuntu at present.

I was pointed at this:

    https://www.freedesktop.org/wiki/Software/systemd/InitrdInterface/

However, Ubuntu seems not to have anything in its initramfs-tools to facilitate "shutdown-initrd" functionality.

I haven't tested this, but I suspect this problem will exist for folks running root on MD-RAID without the LUKS as well. Either way, a relatively common vanilla install will force 90-second timeouts on users, which is unfortunate.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-2ubuntu1 [modified: usr/share/dbus-1/system-services/org.freedesktop.systemd1.service]
ProcVersionSignature: Ubuntu 4.4.0-11.26-generic 4.4.4
Uname: Linux 4.4.0-11-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl nvidia_uvm nvidia_modeset nvidia
ApportVersion: 2.20-0ubuntu3
Architecture: amd64
Date: Tue Mar 8 18:06:45 2016
InstallationDate: Installed on 2016-02-24 (13 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160219)
Lsusb:
 Bus 002 Device 002: ID 1058:0820 Western Digital Technologies, Inc. My Passport Ultra (WDBMWV, WDBZFP)
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 046d:c408 Logitech, Inc. Marble Mouse (4-button)
 Bus 001 Device 002: ID 046d:c318 Logitech, Inc. Illuminated Keyboard
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
ProcEnviron:
 TERM=xterm-color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-11-generic.efi.signed root=/dev/mapper/hostname-root ro net.ifnames=0 biosdevname=0
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/04/2015
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F1
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: X150M-PRO ECC-CF
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF1:bd12/04/2015:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnX150M-PROECC-CF:rvrx.x:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

Revision history for this message
Mason Loring Bliss (y-mason) wrote :
Revision history for this message
Mason Loring Bliss (y-mason) wrote :

I just managed to time it right such that I caught the error on my screen, and in doing to noticed a glaring typo that's likely indicative of the overall code quality of the related software.

I'd be grateful for debugging tips. I hope Canonical's not going to ship software that punishes LUKS users with these 90-second delays each reboot or shutdown. I'd settle for having a way to shorten the timeout, but systemd seems hopelessly opaque and I haven't found where this is set as yet.

Anyway, thanks in advance for fixing this annoyance.

Revision history for this message
Mason Loring Bliss (y-mason) wrote :

Assuming this isn't fixed any time soon, I'm running with this:

mason@ogre /home/mason$ cat /etc/systemd/system.conf.d/expletive.conf
# required singleton - high ceremony
[Manager]
#DefaultTimeoutStartSec=15s
DefaultTimeoutStopSec=15s

Note that while "Manager" is evidently the only possible section, declaring it is required, even for what would
otherwise be presumed to be config snippets in a system.conf.d/ directory.

To clarify previous notes:

1. I don't blame Canonical for systemd.
2. md1_crypt is swap and EXT4 / → LVM → LUKS → MD-RAID1
3. luksroot1 and luksroot2 are /home and /usr/src → ZFS mirror on two LUKS

Revision history for this message
Mason Loring Bliss (y-mason) wrote :

I just confirmed that I see this on a laptop as well. Conveniently, the laptop install is not using MD-RAID or ZFS - it's using LVM on LUKS on a single disk, so it's a simpler case.

Revision history for this message
Seth Arnold (seth-arnold) wrote : Re: [Bug 1554795] Re: timeout on restart or shutdown with LUKS root

On Sun, Mar 20, 2016 at 08:09:56AM -0000, Mason Loring Bliss wrote:
> I just managed to time it right such that I caught the error on my
> screen, and in doing to noticed a glaring typo that's likely indicative
> of the overall code quality of the related software.

The typo looks like an easy fix:

https://github.com/systemd/systemd/blob/da9a4daa08f84db68f7620d2e926690542c31689/src/core/job.c#L648

Thanks

Revision history for this message
Mason Loring Bliss (y-mason) wrote :

In addition to the typo, there may be an issue where the new setting I poked in is used, but the hardcoded default is still displayed on-screen. I've not been able to capture this on camera as yet, and I have yet to set things up such that all the generated messages are stored for later perusal, if that's in fact possible.

The lack of a shutdown-initrd seems to be the more critical issue, in any event. The typos and possible erroneously displayed data on-screen are cosmetic.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
Changed in systemd (Ubuntu):
status: New → Confirmed
Revision history for this message
Mason Loring Bliss (y-mason) wrote :

I'm going to open a seperate ticket for the typo/cosmetic issues noted, so that this ticket can focus on the core. I'll note the ticket number here once I've created it, which I'll do on the other side of my commute.

Changed in systemd (Ubuntu):
importance: Undecided → High
Changed in initramfs-tools (Ubuntu):
importance: Undecided → High
Revision history for this message
Ian (vallamost) wrote :

Are there any workarounds for this for already installed Ubuntu installs?

I have a laptop user that is having 3+ minute shutdowns for this LUKS timeout issue.

Revision history for this message
Ian (vallamost) wrote :

Reboots now hang indefinitely with the laptop user. Any update on this?

amir (amirziler)
Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Mason Loring Bliss (y-mason) wrote :

I'm curious about these two status changes:

Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Fix Committed

Would it be possible to have pointers to the commit(s) that fixed this?

Thanks!

Revision history for this message
Seth Arnold (seth-arnold) wrote :

I'm resetting these to Confirmed since it looks to me like Amir may have accidentally set them to incorrect statuses.

Thanks

Changed in initramfs-tools (Ubuntu):
status: Fix Committed → Confirmed
Changed in systemd (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Joshua Kugler (jkugler) wrote :

I have this problem on a system, and it *never* finishes stopping. It prints out a bunch of "timeout" messages about stopping various disks, then hangs forever stopping the crypt disk. I'll try to get an image of it.

Revision history for this message
Ivan Kozik (ludios) wrote :

In my case the LUKS stuff was unrelated and I was experiencing shutdown hangs because I had daemonizing processes started by /etc/rc.local: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1656237

I managed to discover that by pre-killing a bunch of processes before shutting down, which eventually narrowed down the cause.

Revision history for this message
Jan Vollendorf (u-jan) wrote :

I had the same issue, too...
Lots of messages saying "Timed out stoppping ...".
The system never rebooted.

Here is what fixed the problem for me:

Firstly, I activated systemd's debug shell using 'systemctl enable debug-shell.service'.
Then I initiated a reboot and when the problem occured, I switched to debug shell (alt+F9 [/F10 ?] ).
Using 'systemctl list-jobs', I figured out what jobs were running while the system hung.

One of the jobs was an "unattended-upgrdes" job.
This job held one ore more files on my root filesystem open.
When I killed that process, the system finished its shutdown and rebooted.

After having removed the package by 'apt-get --purge remove unattended-upgrades' the problem didn't arise again.

I am using cron-apt on my servers - therefore I do not necessarily need unattended-upgrades.

Hopefully this helps as workaround and to find the real cause of the problem.

Revision history for this message
Dan Streetman (ddstreet) wrote :

please reopen if this is still an issue

Changed in systemd (Ubuntu):
status: Confirmed → Won't Fix
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.