domain shutdown fails for libvirt/lxc

Bug #1536280 reported by Thomas Maddox
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Confirmed
Undecided
Unassigned
linux (Ubuntu)
Confirmed
Medium
Unassigned

Bug Description

There appears to be a regression from kernel 3.13.0-58-generic where virDomainShutdown no longer brings the domain to shutoff state in newer Ubuntu kernels.

Details:
========

OS:
---
$ lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04

Kernel:
-------
$ uname -a
Linux thomas-devstack01 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Package:
--------
$ dpkg -l | grep libvirt-bin
ii libvirt-bin 1.2.2-0ubuntu13.1.16 amd64 programs for the libvirt library

Libvirt virt driver: LXC

Issue seen from console:
------------------------

$ sudo virsh -c lxc:/// shutdown instance-00000001
error: Failed to shutdown domain instance-00000001
error: internal error: Child process (1304) unexpected exit status 1

libvirtd logs:
--------------

2016-01-20 16:42:38.267+0000: 14089: debug : virDomainShutdown:3094 : dom=0x7f89d0004a70, (VM: name=instance-00000001, uuid=e342beb8-8666-4a47-8b40-5844e866e1f7)
2016-01-20 16:42:38.269+0000: 14089: error : virProcessWait:191 : internal error: Child process (1304) unexpected exit status 1

nova-compute logs
-----------------
*Note*: From when attempting to do a nova stop <instance> in devstack

2016-01-20 16:37:35.244 ERROR oslo_messaging.rpc.dispatcher [req-dd0fe69c-9bd2-4de5-b59b-77619699d69b demo demo] Exception during message handling: internal error: Child process
(631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher payload)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 357, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 407, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 385, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2533, in stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher do_stop_instance()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2525, in do_stop_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._power_off_instance(context, instance, clean_shutdown)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 2260, in _power_off_instance
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self.driver.power_off(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2376, in power_off
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._clean_shutdown(instance, timeout, retry_interval)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2331, in _clean_shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher guest.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 451, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher self._domain.shutdown()
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher six.reraise(c, e, tb)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 2172, in shutdown
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher libvirtError: internal error: Child process (631) unexpected exit status 1
2016-01-20 16:37:35.244 TRACE oslo_messaging.rpc.dispatcher

This all worked gracefully on 3.13.0-58-generic, as well as the newer 3.13.x kernels, including upstream 3.13.0 installed from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-trusty/linux-image-3.13.0-031300-generic_3.13.0-031300.201401192235_amd64.deb.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

marking as affecting kernel given the description.

description: updated
description: updated
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1536280

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: trusty
description: updated
Revision history for this message
Thomas Maddox (thomas-maddox) wrote :
Download full text (510.2 KiB)

FYI, got the following when running the requested command:

$ sudo apport-collect 1536280

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.dpkg-query: no packages found matching libvirt
...

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.dpkg-query: no packages found matching linux
...

Probably expected due to the packages not being named as such. The issue here is with libvirt-bin and the Ubuntu kernel.

Report is as follows:

AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 20 14:55 seq
 crw-rw---- 1 root audio 116, 33 Jan 20 14:55 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
BootDmesg:
 [ 0.000000] Initializing cgroup subsys cpuset
 [ 0.000000] Initializing cgroup subsys cpu
 [ 0.000000] Initializing cgroup subsys cpuacct
 [ 0.000000] Linux version 3.13.0-76-generic (buildd@lgw01-21) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 (Ubuntu 3.13.0-76.120-generic 3.13.11-ckt30)
 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.13.0-76-generic root=/dev/xvda1 ro cgroup_enable=memory swapaccount=1 splash quiet vt.handoff=7
 [ 0.000000] KERNEL supported cpus:
 [ 0.000000] Intel GenuineIntel
 [ 0.000000] AMD AuthenticAMD
 [ 0.000000] Centaur CentaurHauls
 [ 0.000000] e820: BIOS-provided physical RAM map:
 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
 [ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
 [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
 [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000efffffff] usable
 [ 0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
 [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000020fbfffff] usable
 [ 0.000000] NX (Execute Disable) protection: active
 [ 0.000000] SMBIOS 2.4 present.
 [ 0.000000] DMI: Xen HVM domU, BIOS 4.1.3 08/10/2015
 [ 0.000000] Hypervisor detected: Xen HVM
 [ 0.000000] Xen version 4.1.
 [ 0.000000] Xen Platform PCI: I/O protocol version 1
 [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
 [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
 [ 0.000000] You might have to change the root device
 [ 0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
 [ 0.000000] in your root= kernel command line option
 [ 0.000000] HVMOP_pagetable_dying not supported
 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
 [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
 [ 0.000000] No AGP bridge found
 [ 0.000000] e820: last_pfn = 0x20fc00 max_arch_pf...

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu):
status: New → Confirmed
penalvch (penalvch)
Changed in linux (Ubuntu):
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.