TgtAdm is broken

Bug #1038062 reported by Akira Yoshiyama
36
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Critical
John Griffith
OpenStack Compute (nova)
Fix Released
Critical
Chuck Short

Bug Description

Summary
=======

A patch for Bug #1011159 (https://review.openstack.org/10417) breaks
ISCSIDriver with TgtAdm.

a) New volumes won't be exported.
b) Deleting volumes will be failed.

Reason
======

ISCSIDriver expects tid of a volume is specified properly when it was created and the driver use tid to delete it. The driver checks that tid exists before deleting it and doesn't free it if not found.
But scsi-target-utils 1.0.17 doesn't have an option to specify tid in <target xxx> tag. Tid is numbered serially from 1. So b) happens.

Currently, I can't find the reason of a). It looks exported but Nova can't
mount it.

I tried to revert this patch and Nova and Cinder worked perfectly.

Solutions
=========

There are 2 solutions for b).

1. Use newer scsi-target-utils. It has an option "controller_tid" to
   specify tid for a target.
2. Add an argument for show_target() to specify iqn and use it in TgtAdm.
   There is no way for tgtadm/tgt-admin to check a target with the iqn but
   we can do "tgt-admin -s | grep iqn...".

Tags: volume
Akira Yoshiyama (yosshy)
description: updated
description: updated
Revision history for this message
John Griffith (john-griffith) wrote :

I suspect that problem 'a' will be addressed by https://review.openstack.org/#/c/11899/

Revision history for this message
Akira Yoshiyama (yosshy) wrote : Re: [Bug 1038062] Re: TgtAdm is broken

I checked that commit. I think it's not one to fix the problem.

Try to restart tgt and check TID assignment.
2012/08/24 4:56 "John Griffith" <email address hidden>:

> I suspect that problem 'a' will be addressed by
> https://review.openstack.org/#/c/11899/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1038062
>
> Title:
> TgtAdm is broken
>
> Status in Cinder:
> New
>
> Bug description:
> Summary
> =======
>
> A patch for Bug #1011159 (https://review.openstack.org/10417) breaks
> ISCSIDriver with TgtAdm.
>
> a) New volumes won't be exported.
> b) Deleting volumes will be failed.
>
> Reason
> ======
>
> ISCSIDriver expects tid of a volume is specified properly when it was
> created and the driver use tid to delete it. The driver checks that tid
> exists before deleting it and doesn't free it if not found.
> But scsi-target-utils 1.0.17 doesn't have an option to specify tid in
> <target xxx> tag. Tid is numbered serially from 1. So b) happens.
>
> Currently, I can't find the reason of a). It looks exported but Nova
> can't
> mount it.
>
> I tried to revert this patch and Nova and Cinder worked perfectly.
>
> Solutions
> =========
>
> There are 2 solutions for b).
>
> 1. Use newer scsi-target-utils. It has an option "controller_tid" to
> specify tid for a target.
> 2. Add an argument for show_target() to specify iqn and use it in TgtAdm.
> There is no way for tgtadm/tgt-admin to check a target with the iqn
> but
> we can do "tgt-admin -s | grep iqn...".
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1038062/+subscriptions
>

Revision history for this message
John Griffith (john-griffith) wrote :

Akira, hmm... the problem I'm running in to is that I can't seem to reproduce this. I'm running a fresh 12.04 install with a fresh devstack install. The submitted patch addressed an issue with attach not working, which I'd hoped might be related to the first failure condition you reported.

Any details regarding your setup that might help me reproduce this?

Revision history for this message
Akira Yoshiyama (yosshy) wrote :

Hi John,
I built my Folsom-3 env from github on Ubuntu 12.04, doing git clone,
python setup.py build / install.

I created & deleted volumes repeatly, so there are 2 volumes with tid=4 and
5. Then I rebooted tgtd and tids are changed as 1 and 2. Deleting volumes
failed because ISCSIDriver checked that original tids didn't exist and
skipped to free their targets.
2012/08/24 9:01 "John Griffith" <email address hidden>:

> Akira, hmm... the problem I'm running in to is that I can't seem to
> reproduce this. I'm running a fresh 12.04 install with a fresh devstack
> install. The submitted patch addressed an issue with attach not
> working, which I'd hoped might be related to the first failure condition
> you reported.
>
> Any details regarding your setup that might help me reproduce this?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1038062
>
> Title:
> TgtAdm is broken
>
> Status in Cinder:
> New
>
> Bug description:
> Summary
> =======
>
> A patch for Bug #1011159 (https://review.openstack.org/10417) breaks
> ISCSIDriver with TgtAdm.
>
> a) New volumes won't be exported.
> b) Deleting volumes will be failed.
>
> Reason
> ======
>
> ISCSIDriver expects tid of a volume is specified properly when it was
> created and the driver use tid to delete it. The driver checks that tid
> exists before deleting it and doesn't free it if not found.
> But scsi-target-utils 1.0.17 doesn't have an option to specify tid in
> <target xxx> tag. Tid is numbered serially from 1. So b) happens.
>
> Currently, I can't find the reason of a). It looks exported but Nova
> can't
> mount it.
>
> I tried to revert this patch and Nova and Cinder worked perfectly.
>
> Solutions
> =========
>
> There are 2 solutions for b).
>
> 1. Use newer scsi-target-utils. It has an option "controller_tid" to
> specify tid for a target.
> 2. Add an argument for show_target() to specify iqn and use it in TgtAdm.
> There is no way for tgtadm/tgt-admin to check a target with the iqn
> but
> we can do "tgt-admin -s | grep iqn...".
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1038062/+subscriptions
>

Changed in cinder:
assignee: nobody → Chuck Short (zulcss)
status: New → In Progress
Changed in cinder:
milestone: none → folsom-rc1
status: In Progress → Confirmed
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12082

Changed in nova:
assignee: nobody → Chuck Short (zulcss)
status: New → In Progress
Revision history for this message
John Griffith (john-griffith) wrote :

Fix is on the way

Mark McLoughlin (markmc)
Changed in nova:
importance: Undecided → Critical
milestone: none → folsom-rc1
Changed in cinder:
assignee: Chuck Short (zulcss) → John Griffith (john-griffith)
status: Confirmed → In Progress
Mark McLoughlin (markmc)
tags: added: volume
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/12072
Committed: http://github.com/openstack/cinder/commit/9785963c841aff5ea1c159da81f93c2bf62c70c6
Submitter: Jenkins
Branch: master

commit 9785963c841aff5ea1c159da81f93c2bf62c70c6
Author: Chuck Short <email address hidden>
Date: Tue Aug 28 15:06:25 2012 -0500

    Fix creation of iscsi targets

    Previously when creating iscsi volumes, we were using

    tgt-admin -e -c <config file> --update volume-id

    Unfortunately the side affect of this is that tgt-admin
    removed other volumes that weren't connected to an iscsi
    connector. Which is obvlously not what we want.

    In order to fix this we create the targets.conf for the
    volume but we call tgt-admin --update icssi qualified name.

    We're dropping the use of iscsi_targets table when using TgtAdm.
    Compatability for other target admin types is maintained.

    Fixes LP: #1038062

    Change-Id: I9060a43208df5b79e9b17dadcab8bc0a8eeef55e
    Signed-off-by: Chuck Short <email address hidden>

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12082
Committed: http://github.com/openstack/nova/commit/66f6a9edce3ccd624aba5d2a6bf3362901ed57f7
Submitter: Jenkins
Branch: master

commit 66f6a9edce3ccd624aba5d2a6bf3362901ed57f7
Author: Chuck Short <email address hidden>
Date: Tue Aug 28 19:00:49 2012 -0500

    Fix creation of iscsi targets

    Previously when creating iscsi volumes, we were using

    tgt-admin -e -c <config file> --update volume-id

    Unfortunately the side affect of this is that tgt-admin
    removed other volumes that wasnt connected to an iscsi
    connector. Which is obvlously not what we want.

    In order to fix this we create the targets.conf for the
    volume but we call tgt-admin --update icssi qualified name.
    We also set the tid in the configuration file now as well.

    Fixes LP: #1038062

    Change-Id: I23719390fbfaea5b55389a5c8ebaa8966cc283a8
    Signed-off-by: Chuck Short <email address hidden>

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: folsom-rc1 → 2012.2
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.