[Nova-volume][tgtd] Failed to detach volume / delete instance when target is already disconnected

Bug #1011166 reported by David Naori
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Chuck Short

Bug Description

[root@camel-nova nova]# nova volume-attach toattach 17 /dev/vdd

[root@camel-vdsa ~]# iscsiadm -m session
tcp: [21] 10.35.97.20:3260,1 iqn.2010-10.org.openstack:volume-00000011
[root@camel-vdsa ~]# iscsiadm -m session -u
Logging out of session [sid: 21, target: iqn.2010-10.org.openstack:volume-00000011, portal: 10.35.97.20,3260]
Logout of [sid: 21, target: iqn.2010-10.org.openstack:volume-00000011, portal: 10.35.97.20,3260] successful.

[root@camel-nova nova]# nova volume-detach toattach 17

Compute.log:
2012-06-10 16:52:09 ERROR nova.rpc.amqp [req-691b5982-1882-4c5e-b8fc-3b3da26a0699 2f4edfa99cab42de92eacda360043116 7eed65dd55474b9e94cd412d9f66b406] Exception during message handling
2012-06-10 16:52:09 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-06-10 16:52:09 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-06-10 16:52:09 TRACE nova.rpc.amqp return f(*args, **kw)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 153, in decorated_function
2012-06-10 16:52:09 TRACE nova.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
2012-06-10 16:52:09 TRACE nova.rpc.amqp sys.exc_info())
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-06-10 16:52:09 TRACE nova.rpc.amqp self.gen.next()
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 171, in decorated_function
2012-06-10 16:52:09 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1770, in detach_volume
2012-06-10 16:52:09 TRACE nova.rpc.amqp self._detach_volume(context, instance_ref, bdm)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1760, in _detach_volume
2012-06-10 16:52:09 TRACE nova.rpc.amqp mp)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-06-10 16:52:09 TRACE nova.rpc.amqp return f(*args, **kw)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 582, in detach_volume
2012-06-10 16:52:09 TRACE nova.rpc.amqp mount_device)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line 524, in volume_driver_method
2012-06-10 16:52:09 TRACE nova.rpc.amqp return method(connection_info, *args, **kwargs)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 946, in inner
2012-06-10 16:52:09 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 206, in disconnect_volume
2012-06-10 16:52:09 TRACE nova.rpc.amqp check_exit_code=[0, 255])
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 110, in _run_iscsiadm
2012-06-10 16:52:09 TRACE nova.rpc.amqp check_exit_code=check_exit_code)
2012-06-10 16:52:09 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
2012-06-10 16:52:09 TRACE nova.rpc.amqp cmd=' '.join(cmd))
2012-06-10 16:52:09 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-06-10 16:52:09 TRACE nova.rpc.amqp Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000011 -p 10.35.97.20:3260 --logout
2012-06-10 16:52:09 TRACE nova.rpc.amqp Exit code: 21
2012-06-10 16:52:09 TRACE nova.rpc.amqp Stdout: ''
2012-06-10 16:52:09 TRACE nova.rpc.amqp Stderr: 'iscsiadm: No matching sessions found\n'
2012-06-10 16:52:09 TRACE nova.rpc.amqp

[root@camel-nova nova]# nova volume-list
+----+-----------+-------------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
| 17 | in-use | a | 1 | None | 704340ab-4c9a-42e1-b676-fb33ae624adf |

[root@camel-nova nova]# nova delete toattach
[same error in compute log]
2012-06-10 16:54:52 TRACE nova.rpc.amqp Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-00000011 -p 10.35.97.20:3260 --logout
2012-06-10 16:54:52 TRACE nova.rpc.amqp Exit code: 21
2012-06-10 16:54:52 TRACE nova.rpc.amqp Stdout: ''
2012-06-10 16:54:52 TRACE nova.rpc.amqp Stderr: 'iscsiadm: No matching sessions found\n'

[root@camel-nova nova]# nova list
+--------------------------------------+----------+---------+----------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------+---------+----------------------+
| 704340ab-4c9a-42e1-b676-fb33ae624adf | toattach | ACTIVE | davidnet=192.168.1.4 |
+--------------------------------------+----------+---------+----------------------+

Changed in nova:
assignee: nobody → Chuck Short (zulcss)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/11930
Committed: http://github.com/openstack/nova/commit/9135f836151df4bc32e3c3fea208e3cee6f7d6dc
Submitter: Jenkins
Branch: master

commit 9135f836151df4bc32e3c3fea208e3cee6f7d6dc
Author: Chuck Short <email address hidden>
Date: Fri Aug 24 10:48:09 2012 -0500

    Trap iscsiadm error

    If the iscsitarget was previously deleted and the user tries
    to delete the iscsi target again, then you get a nasty traceback
    in the logfiles.

    According to the iscsiadm man page, iscsiadm returns an error 21,
    ISCSI_ERR_NO_OBJS_FOUND if there is no records/targets/sessions/portals
    found to execute the operation on.

    LP: #1011166

    Change-Id: I3087c1a114fe91bd7361d92688d4f926bc3f8951
    Signed-off-by: Chuck Short <email address hidden>

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.