[Xenial] shutdown/reboot hangs at "Reached target Shutdown"

Bug #1619844 reported by Andy Shulman
60
This bug affects 10 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Won't Fix
Critical
Unassigned

Bug Description

I tried getting more information in the debug-shell but I was unsuccessful in doing so. I was able to switch to the debug shell via ctrl+alt+f9 but I couldn't actually type anything in. I could switch back to the stuck shutdown screen via ctrl+alt+f1, and hitting NumLock would turn the light on my keyboard on and off, but I couldn't do anything else. I captured a journal of the stuck reboot. Interestingly, the journal shows a few more lines after "Reached target Shutdown", which is as far as I get on my screen.

This system was installed with Xenial 16.04.1 (never been upgraded) and has had this problem since the very first boot.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-4ubuntu7
ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16
Uname: Linux 4.4.0-36-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Sat Sep 3 00:33:19 2016
InstallationDate: Installed on 2016-08-17 (16 days ago)
InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ProcEnviron:
 SHELL=/bin/bash
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-36-generic.efi.signed root=UUID=5f709fed-48c9-4830-87be-acd13f263eb1 ro
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /lib/systemd/system/mariadb.service → /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf

 3 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/03/2016
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: KYSKLi70.86A.0037.2016.0603.1032
dmi.board.name: NUC6i7KYB
dmi.board.vendor: Intel Corporation
dmi.board.version: H90766-404
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrKYSKLi70.86A.0037.2016.0603.1032:bd06/03/2016:svn:pn:pvr:rvnIntelCorporation:rnNUC6i7KYB:rvrH90766-404:cvnIntelCorporation:ct3:cvr1.0:

Revision history for this message
Andy Shulman (andy-shulman) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Revision history for this message
Alberto Salvia Novella (es20490446e) wrote :

Does the following work-around work for you?
(http://michalorman.com/2013/10/fix-ubuntu-freeze-during-restart)

After answering, please set status back to "confirmed". Thank you.

Changed in systemd (Ubuntu):
importance: Undecided → Critical
status: Confirmed → Incomplete
Revision history for this message
Andy Shulman (andy-shulman) wrote :

I can't quite figure out why, but the answer seems to be "occasionally". I was able to reboot a few times (2 out of maybe 30?) using the complete list of reboot= options. However, I suspect making the reboot= changes is a red herring -- I rebooted the machine more in the past few hours than I have in the rest of the time I've owned it. It's entirely possible that I would have gotten similarly lucky had I kept trying to reboot over and over again without making the reboot= changes.

Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Ilario (ilario) wrote :

Same problem here. Fresh install of 16.04.1 Ubuntu server. CPU intel J1900. Kernel 4.4.0-59. I tried what suggested in the link above and any other method I found around (force ahci in grub options, deactivate usb 3 legacy, using different commands for restarting..) but with no success. Still hangs at "Reached target Shutdown ". As I plan to install the PC circa 1500 Km away from where I live, pressing the physical button could be problematic...

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Journal ends with:
Sep 02 23:59:22 nuc systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Sep 02 23:59:22 nuc systemd-journald[1455]: Journal stopped

which means that something is not dieing and we are not getting any more logs recorded.....

and it indicates that all/most attempts to reboot have been exhausted and possibly it is the firmware/bios itself that are failing to reboot/shutdown the machine.

What are exact model details of the affected machines? maybe we can find same/similar in the validation lab and check if they pass Ubuntu certification.

Revision history for this message
Andy Shulman (andy-shulman) wrote :

Mine is an Intel NUC6i7KYK. As the model number suggests, it's an Intel NUC with a Skull Canyon (6th generation) i7 processor. Intel ARK link: http://ark.intel.com/products/89187/Intel-NUC-Kit-NUC6i7KYK.

Revision history for this message
Tobias Alke (skyfly1987) wrote :

I think i have a specification to this issue.

First, my system with the same problem at shutdown/reboot, but only in a very special case:

Fresh install of Ubuntu Server 16.04.2 LTS
CPU: Intel Xeon E3-1245v6 (Kabylake)
Mainboard: MSI C236A Workstation/Server Mainboard (model: 7998-012R; latest BIOS: v2.7)
RAM: 4x TRANSCEND 8GB DDR4 2133 ECC-DIMM 2Rx8 1Gx72 288P 512Mx8/CL15 (model: TS1GLH72V1H)
SSD: 2x Samsung Basic MZ-7KE256BW 850 Pro interne SSD 256GB
HDD: 2x Western Digital Red 4TB NAS (model: WDBMMA0040HNC-ERSN)

Description of the special case with the shutdown/reboot issue:

The problem only reveals when RAID-Mode auf the MSI Mainboard is enabled. After resetting to AHCI-Mode, the problem is over an the server works fine. It seems, that's a problem with the "Intel® C236 Chipset" or more specific a driver problem with intel based components on the different systems.

Revision history for this message
Andrzej Blukis (blukisandrzej) wrote : Re: [Bug 1619844] Re: [Xenial] shutdown/reboot hangs at "Reached target Shutdown"
Download full text (4.0 KiB)

Yes. I confirm that. After change in bios from raid to ahci computer is rebooting and swithing off normally.

Wysłano z mojego smartfona Samsung Galaxy.
-------- Oryginalna wiadomość --------Od: Tobias Alke <email address hidden> Data: 22.04.2017 08:53 (GMT+01:00) Do: <email address hidden> Temat: [Bug 1619844] Re: [Xenial] shutdown/reboot hangs at "Reached target
  Shutdown"
I think i have a specification to this issue.

First, my system with the same problem at shutdown/reboot, but only in a
very special case:

Fresh install of Ubuntu Server 16.04.2 LTS
CPU: Intel Xeon E3-1245v6 (Kabylake)
Mainboard: MSI C236A Workstation/Server Mainboard (model: 7998-012R; latest BIOS: v2.7)
RAM: 4x TRANSCEND 8GB DDR4 2133 ECC-DIMM 2Rx8 1Gx72 288P 512Mx8/CL15 (model: TS1GLH72V1H)
SSD: 2x Samsung Basic MZ-7KE256BW 850 Pro interne SSD 256GB
HDD: 2x Western Digital Red 4TB NAS (model: WDBMMA0040HNC-ERSN)

Description of the special case with the shutdown/reboot issue:

The problem only reveals when RAID-Mode auf the MSI Mainboard is
enabled. After resetting to AHCI-Mode, the problem is over an the server
works fine. It seems, that's a problem with the "Intel® C236 Chipset" or
more specific a driver problem with intel based components on the
different systems.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1619844

Title:
  [Xenial] shutdown/reboot hangs at "Reached target Shutdown"

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  I tried getting more information in the debug-shell but I was
  unsuccessful in doing so. I was able to switch to the debug shell via
  ctrl+alt+f9 but I couldn't actually type anything in. I could switch
  back to the stuck shutdown screen via ctrl+alt+f1, and hitting NumLock
  would turn the light on my keyboard on and off, but I couldn't do
  anything else. I captured a journal of the stuck reboot.
  Interestingly, the journal shows a few more lines after "Reached
  target Shutdown", which is as far as I get on my screen.

  This system was installed with Xenial 16.04.1 (never been upgraded)
  and has had this problem since the very first boot.

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: systemd 229-4ubuntu7
  ProcVersionSignature: Ubuntu 4.4.0-36.55-generic 4.4.16
  Uname: Linux 4.4.0-36-generic x86_64
  ApportVersion: 2.20.1-0ubuntu2.1
  Architecture: amd64
  Date: Sat Sep  3 00:33:19 2016
  InstallationDate: Installed on 2016-08-17 (16 days ago)
  InstallationMedia: Ubuntu-Server 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  ProcEnviron:
   SHELL=/bin/bash
   TERM=xterm
   PATH=(custom, no user)
   LANG=en_US.UTF-8
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-36-generic.efi.signed root=UUID=5f709fed-48c9-4830-87be-acd13f263eb1 ro
  SourcePackage: systemd
  SystemdDelta:
   [EXTENDED]   /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
   [EXTENDED]   /lib/systemd/system/rc-local....

Read more...

Revision history for this message
Darrell Enns (darrellenns) wrote :

Same issue here, on a Supermicro 6018R-WTR server. I've attached a screenshot of the hung state. Note that the watchdog message is because I've set up IPMI watchdog on this machine (as a workaround for this issue). The hang on reboot is not happening every time for me - something like 10%-20% of reboots result in a hang. Kernel version is 4.4.0-87. I have tried setting reboot=pci on the kernel command line and blacklisting mei and mei_me modules, but neither of those resolved the problem.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

It depends on what sort of raid modes are in place. If you are not using raid (e.g. RAID1) as provided by your motherboard firmware, and only use individual devices, it is best to disable raid settings in the bios and use regular acpi setting. I have to do this for example on latest generation of Dell XPS laptops that use "raid" mode on a single nvme drive with OS support drivers that are not available for linux.

Otherwise if you do use RAID controllers with mdadm, e.g. Intel Matrix RAID, DDF, etc. as part of https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1722491 there is currently mdadm update available from xenial-proposed and zesty-proposed that might resolve this issue.

To test that solution please perform the following:

1) Install mdadm from xenial-proposed/zesty-proposed
   - See https://wiki.ubuntu.com/Testing/EnableProposed
   - Or download & install packages from
xenial
https://launchpad.net/ubuntu/+source/mdadm/3.4-4ubuntu0.1/+build/13596415
zesty
https://launchpad.net/ubuntu/+source/mdadm/3.3-2ubuntu7.5/+build/13596431

2) $ sudo apt install dracut-core

3) $ sudo systemctl enable mdadm-shutdown.service

4) $ sudo systemctl start mdadm-shutdown.service

After this the expectation is for shutdown/reboots to perform clean a shutdown, maintaining the raid array in a synced state, such that it comes up clean.

Please let me know if above resolves shutdown/reboot issues for you.

Regards,

Dimitri.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Just a quick clarification, first boot/shutdown will still not be clean, but subsequent ones (those that are booted with the updated mdadm package) should be clean.

Revision history for this message
Silas S. Brown (ssb22) wrote :

This bug disappeared for me when I did

apt-get install linux-generic-hwe-16.04

to upgrade to kernel version 4.10.0 instead of 4.4.0. (I also had proposed-updates in /etc/apt/sources.list, but it was the linux-generic-hwe-16.04 package that really fixed it.)

I suspected it was a kernel bug after I installed upstart-sysv instead of systemd-sysv and the bug still persisted, and I also tried editing the upstart scripts to do "echo o > /proc/sysrq-trigger" instead of normal poweroff, whereupon the kernel reported the SysRq trigger event but didn't act on it. Then I tried the newer kernel and everything worked fine (with my other changes reverted) regardless of whether I was using upstart-sysv or systemd-sysv.

Revision history for this message
Andy Shulman (andy-shulman) wrote :

I no longer have this issue. I don't know whether it was updating mdadm or another package, but I recently issued a shutdown command and walked over to my box to hold the power button for a few seconds and finish shutting it off, only to find it was already off. I've since confirmed that a reboot command turns the box off and back on again. Thanks!

Dan Streetman (ddstreet)
Changed in systemd (Ubuntu):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.