destroy an instance give a permission deny on /dev/pts

Bug #890066 reported by Patrick Hetu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

ii nova-ajax-console-proxy 2011.3-0ubuntu6.2 OpenStack Compute - AJAX console proxy
ii nova-api 2011.3-0ubuntu6.2 OpenStack Compute - API frontend
ii nova-common 2011.3-0ubuntu6.2 OpenStack Compute - common files
ii nova-compute 2011.3-0ubuntu6.2 OpenStack Compute - compute node
ii nova-compute-lxc 2011.3-0ubuntu6.2 OpenStack Compute - compute node (LXC)
ii nova-network 2011.3-0ubuntu6.2 OpenStack Compute - Network manager
ii nova-objectstore 2011.3-0ubuntu6.2 OpenStack Compute - object store
ii nova-scheduler 2011.3-0ubuntu6.2 OpenStack Compute - virtual machine scheduler
ii nova-vncproxy 2011.3-0ubuntu6.2 OpenStack Compute - VNC proxy
rc nova-volume 2011.3-0ubuntu6.2 OpenStack Compute - storage
ii python-nova 2011.3-0ubuntu6.2 OpenStack Compute Python libraries
ii python-novaclient 2.6.4~bzr112-0ubuntu1 client library for OpenStack Compute API
ii lxc 0.7.5-0ubuntu8.1 Linux containers userspace tools
ii nova-compute-lxc 2011.3-0ubuntu6.2 OpenStack Compute - compute node (LXC)
ii libvirt-bin 0.9.2-4ubuntu16 the programs for the libvirt library

2011-11-13 22:43:57,702 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.exception): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 493, in terminate_instance
(nova.exception): TRACE: self._shutdown_instance(context, instance_id, 'Terminating')
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 484, in _shutdown_instance
(nova.exception): TRACE: self.driver.destroy(instance, network_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 401, in destroy
(nova.exception): TRACE: self._cleanup(instance)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 414, in _cleanup
(nova.exception): TRACE: shutil.rmtree(target)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 244, in rmtree
(nova.exception): TRACE: rmtree(fullname, ignore_errors, onerror)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 244, in rmtree
(nova.exception): TRACE: rmtree(fullname, ignore_errors, onerror)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 244, in rmtree
(nova.exception): TRACE: rmtree(fullname, ignore_errors, onerror)
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 253, in rmtree
(nova.exception): TRACE: onerror(os.rmdir, path, sys.exc_info())
(nova.exception): TRACE: File "/usr/lib/python2.7/shutil.py", line 251, in rmtree
(nova.exception): TRACE: os.rmdir(path)
(nova.exception): TRACE: OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/instance-0000004a/rootfs/dev/pts'
(nova.exception): TRACE:

/etc/nova/nova.conf

--connection_type=libvirt
--default_image=uec-oneiric-qcow2
--dhcpbridge=/usr/bin/nova-dhcpbridge
--dhcpbridge_flagfile=/etc/nova/nova.conf
--fixed_range=192.168.100.0/22
--network_size=32
--vlan_interface=eth1
--force_dhcp_release=True
--glance_host=192.168.100.1
--use_s3=false
--image_service=nova.image.glance.GlanceImageService
--iscsi_helper=tgtadm
--libvirt_type=lxc
--lock_path=/var/lock/nova
--logdir=/var/log/nova
--network_manager=nova.network.manager.VlanManager
--num_networks=1
--osapi_extensions_path=/var/lib/nova/extensions/
--public_interface=eth0
--sql_connection=postgresql://nova:*******@localhost:5432/nova
--state_path=/var/lib/nova
--use_deprecated_auth=false
--verbose
--vnc_console_proxy_url=http://192.168.100.1:6080
--vncproxy_port=8060
--start_guests_on_host_boot=true
--dhcp_domain=cloud.koumbit.net

Tags: lxc
Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

Sometime, I also have this error, not sure if I need a open a other bug:

2011-11-13 22:59:56,503 WARNING nova.virt.libvirt_conn [-] Error from libvirt during destroy of instance-00000046. Code=1 Error=internal error Some container PIDs refused to die
2011-11-13 22:59:57,752 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 565, in reboot
(nova.exception): TRACE: self.destroy(instance, network_info, cleanup=False)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 348, in destroy
(nova.exception): TRACE: virt_dom.destroy()
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/libvirt.py", line 381, in destroy
(nova.exception): TRACE: if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
(nova.exception): TRACE: libvirtError: internal error Some container PIDs refused to die

Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

By "Sometime" I mean when I restart nova-compute.

Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

To temporarly solved the problem by modifing the file nova/virt/libvirt/connection.py and changed the line 414 from:

  shutil.rmtree(target)

to:

  utils.execute('rm', '-rf', target, run_as_root=True)

in the _cleanup method. I also added the permissions for "rm" in the sudoers file.

Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

False positive, I got the error "Device or resource busy" on the "rm -fr" BUT restarting libvirt-bin makes the directory removable again.

Revision history for this message
Thierry Carrez (ttx) wrote :

So should this bug be closed as an invalid issue ?

Changed in nova:
status: New → Incomplete
Revision history for this message
Patrick Hetu (patrick-hetu) wrote :

Yes Thierry, you can close the bug, I think it's a libvirt-bin issue fixed in newer version.

Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.