tgt connection issues can make it impossible to delete targets

Bug #1305343 reported by Vish Ishaya
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tgt (Ubuntu)
Fix Released
High
Chuck Short
Trusty
Fix Released
High
Chuck Short

Bug Description

Under heavy load, openiscsi will sometimes report connection errors when using tgt:

Apr 8 22:02:44 node003-cont001 iscsid: Connection2:0 to [target: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9, portal: 172.21.1.1,3260] through [iface: default] is operational now

Apr 8 22:10:53 node003-cont001 iscsid: Kernel reported iSCSI connection 2:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)
Apr 8 22:10:56 node003-cont001 iscsid: connection2:0 is operational after recovery (1 attempts)
Apr 8 22:18:47 node003-cont001 iscsid: Kernel reported iSCSI connection 2:0 error (1021 - ISCSI_ERR_SCSI_EH_SESSION_RST: Session was dropped as a result of SCSI error recovery) state (3)
Apr 8 22:18:50 node003-cont001 iscsid: connection2:0 is operational after recovery (1 attempts)

Even though openiscsi has disconnected:

Apr 8 22:34:04 node003-cont001 iscsid: Connection2:0 to [target: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9, portal: 172.21.1.1,3260] through [iface: default] is shutdown.

tgt still shows the old connections:

$ sudo tgtadm --op show --mode target
Target 5: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3
        I_T nexus: 6
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3

When these duplicate connections are there, the volume cannot be deleted:

$ sudo tgt-admin -v --force --delete iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
# Removing target: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
tgtadm -C 0 --op delete --mode conn --tid 5 --sid 6 --cid 0
tgtadm -C 0 --op delete --mode conn --tid 5 --sid 5 --cid 0
tgtadm -C 0 --mode target --op delete --tid=5
tgtadm: this target is still active
Command:
 tgtadm -C 0 --mode target --op delete --tid=5
exited with code: 22.

The following errors show up in syslog illustrating the force delete failing:

Apr 9 22:31:50 node001-cont001 tgtd: conn_close_admin(237) close 6 0
Apr 9 22:31:50 node001-cont001 tgtd: tgt_event_modify(241) Cannot find event 21
Apr 9 22:31:50 node001-cont001 tgtd: iscsi_event_modify(558) tgt_event_modify failed
Apr 9 22:31:50 node001-cont001 tgtd: conn_close_admin(237) close 5 0
Apr 9 22:31:50 node001-cont001 tgtd: tgt_event_modify(241) Cannot find event 19
Apr 9 22:31:50 node001-cont001 tgtd: iscsi_event_modify(558) tgt_event_modify failed
Apr 9 22:31:50 node001-cont001 tgtd: tgt_target_destroy(2211) target 5 still has it nexus

And the connections are still there:

$ sudo tgt-admin -v --show
tgtadm -C 0 --op show --mode target
Target 5: iqn.2010-10.org.openstack:volume-b32d317e-4063-4e4f-963e-b0fdc29198f9
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 5
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3
        I_T nexus: 6
            Initiator: iqn.1993-08.org.debian:01:3844c047b3a9 alias: node003-cont001
            Connection: 0
                IP Address: 172.21.1.3

Revision history for this message
Vish Ishaya (vishvananda) wrote :

I believe this may be fixed by:

https://github.com/fujita/tgt/commit/e6e8d0fd3b83208278fb7ed765c3665ae7c5ab49

but also may require:

https://github.com/fujita/tgt/commit/05fc2cb424863fbf2f3c60a8b617308ef1e370c5

I'm doing a test of build 1.0.46 tgt to see if it resolves the issue.

summary: - tgt connection issues can make it impossible to delete volumes
+ tgt connection issues can make it impossible to delete targets
Chuck Short (zulcss)
Changed in tgt (Ubuntu Trusty):
assignee: nobody → Chuck Short (zulcss)
James Page (james-page)
Changed in tgt (Ubuntu Trusty):
importance: Undecided → High
Revision history for this message
Vish Ishaya (vishvananda) wrote :

I can confirm that 1.0.46 fixes the issue. The force delete command works properly without errors in syslog

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tgt - 1:1.0.43-0ubuntu4

---------------
tgt (1:1.0.43-0ubuntu4) trusty; urgency=medium

  * debian/patches/fix_delayed_management_request.patch: Adjust patch to fix
    a FTBFS.
  * debian/patches/fix_leak_of_task_in_scsi_flag.patch: Adjust patch to fix
    a FTBFS.
 -- Chuck Short <email address hidden> Fri, 11 Apr 2014 09:14:03 -0400

Changed in tgt (Ubuntu Trusty):
status: New → Fix Released
Juan Pavlik (jjpavlik)
no longer affects: tgt-project
Revision history for this message
Juan Pavlik (jjpavlik) wrote :

Hi guys I've hit this bug un Precise, is there going to be a backport?

Logs:
Oct 14 16:20:45 heladera-b tgtd: conn_close_admin(235) close e 0
Oct 14 16:20:45 heladera-b tgtd: tgt_event_modify(213) Cannot find event 11
Oct 14 16:20:45 heladera-b tgtd: iscsi_event_modify(413) tgt_event_modify failed
Oct 14 16:20:45 heladera-b tgtd: tgt_target_destroy(1936) target 10 still has it nexus

TGT version:
root@heladera-b:/var/lib/cinder/volumes# apt-cache show tgt
Package: tgt
Priority: optional
Section: net
Installed-Size: 595
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian Kernel Team <email address hidden>
Architecture: amd64
Version: 1:1.0.17-1ubuntu2
...

Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.