[SRU] multipath iscsi does not logout of sessions on xenial

Reviewed: https://review.openstack.org/374421
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=e591bc78cc01c1171060dc15399a46ff800b49c3
Submitter: Jenkins
Branch: master

commit e591bc78cc01c1171060dc15399a46ff800b49c3
Author: Patrick East <email address hidden>
Date: Wed Sep 21 15:05:48 2016 -0700

    Stop calling multipath -r when attaching/detaching iSCSI volumes

    Looking into this more there isn't any documented reason why we do this,
    and on Ubuntu 16.04 there are issues with timing and devices/symlinks
    getting messed up when we do the reload of device maps. We shouldn't
    need to be forcing multipathd to do this, it loads devices on its own.

    We'll leave in the one in 'wait_for_rw(..)' for now because there is
    some evidence that you may need to call it to update the rw state of
    the multipath devices, see:
    https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise
    _Linux/6/html/Storage_Administration_Guide/ch37s04s02.html

    Change-Id: Iec58284abdc9bcbf99df5d07289bb9d60a3554d7
    Closes-Bug: #1623700

This issue was fixed in the openstack/os-brick 1.8.0 release.

Hi, testing this fix, we found that in certain scenarios, if we don't call _rescan_multipath when attaching/detaching iSCSI volumes, the iSCSI sessions are not logged out. Our current workaround is doing a double _rescan_multipath in those places.

Tested with Netapp e-series 5524 / xenial /mitaka

Perhatps the rescans are there for a reason...

WITHOUT RESCAN:

Jan 25 09:24:40 calling os-brick to detach iSCSI Volume
Jan 25 09:24:40 Lock "connect_volume" acquired by "os_brick.initiator.connector.disconnect_volume" :: waited 0.000s
Jan 25 09:24:40 multipath -ll /dev/sdr
Jan 25 09:24:40 multipath -ll /dev/sdr" returned: 0 in 0.218s
Jan 25 09:24:40 multipath ['-ll', u'/dev/sdr']: stdout=360080e5000297ea40000050658885f45 dm-6 NETAPP,INF-01-00#012size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw#012|-+- policy='
service-time 0' prio=14 status=active#012| |- 11:0:0:135 sdr 65:16 active ready running#012| |- 3:0:0:135 sds 65:32 active ready running#012| |- 8:0:0:135 sdq 65:0 active ready running#012| `- 9:0:0:135 sdt 65:48 active ready runni
ng#012`-+- policy='service-time 0' prio=9 status=enabled#012 |- 10:0:0:135 sdo 8:224 active ready running#012 |- 10:0:0:162 sdi 8:128 active faulty running#012 |- 11:0:0:162 sdj 8:144 active faulty running#012 |- 12:0:0:135 sdn 8:208
active ready running#012 |- 12:0:0:162 sdk 8:160 active faulty running#012 |- 13:0:0:135 sdp 8:240 active ready running#012 |- 13:0:0:162 sdl 8:176 active faulty running#012 |- 3:0:0:162 sdb 8:16 active faulty running#012 |- 4:0:0
:135 sdm 8:192 active ready running#012 |- 4:0:0:162 sdc 8:32 active faulty running#012 |- 8:0:0:162 sdg 8:96 active faulty running#012 `- 9:0:0:162 sdh 8:112 active faulty running#012 stderr=
Jan 25 09:24:40 remove multipath device /dev/sdr
Jan 25 09:24:40 multipath -l /dev/sdr
Jan 25 09:24:40 multipath -l /dev/sdr" returned: 0 in 0.176s
Jan 25 09:24:40 Couldn't find multipath device /dev/mapper/360080e5000297ea40000050658885f45
Jan 25 09:24:40 Disconnect multipath device /dev/mapper/360080e5000297ea40000050658885f45
Jan 25 09:24:40 multipath -ll
Jan 25 09:24:41 multipath -ll" returned: 0 in 0.248s
Jan 25 09:24:41 multipath ['-ll']: stdout=3624a9370dbcfee6048974fe300011180 dm-1 ##,###012size=1.0G features='0' hwhandler='0' wp=rw#012360080e5000297ea40000050658885f45 dm-6 NETAPP,INF-01-00#012size=10G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 rdac' wp=rw#012|-+- policy='service-time 0' prio=14 status=active#012| |- 11:0:0:135 sdr 65:16 active ready running#012| |- 3:0:0:135 sds 65:32 active ready running#012| |- 8:0:0:135 sdq 65:0 active ready running#012| `- 9:0:0:135 sdt 65:48 active ready running#012`-+- policy='service-time 0' prio=9 status=enabled#012 |- 10:0:0:135 sdo 8:224 active ready running#012 |- 10:0:0:162 sdi 8:128 active faulty running#012 |- 11:0:0:162 sdj 8:144 active faulty running#012 |- 12:0:0:135 sdn 8:208 active ready running#012 |- 12:0:0:162 sdk 8:160 active faulty running#012 |- 13:0:0:135 sdp 8:240 active ready running#012 |- 13:0:0:162 sdl 8:176 active faulty running#012 |- 3:0:0...

anyone can help sponsor this sru ? thank a lot

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/447018

Hua, Can you update the regression potential for the Ubuntu SRU details please? 'None' isn't really a good answer, especially with Gustavo's comment above. Can we check if any other patches have landed upstream wrt his comment?

This is going to need to land in yakkety and newton before it gets backported to xenial. I've initiated the upstream backport to newton:
https://review.openstack.org/#/c/447018/

Can't seem to target this to yakkety atm, launchpad is erroring for me.

@Gustavo,

I can't reproduce your problem, this patch works well for me. I did two experiments.

One experiment is WITHOUT multipath-tools patch [1] + WITHOUT this os-brick patch, the test result can refer the link [2], we can see that multipath device can be deleted by 'multipath -r'.

Mar 22 10:33:52 juju-zhhuabj-machine-9 nova-compute[22305]: 2017-03-22 10:33:52.233 22305 WARNING os_brick.initiator.linuxscsi [req-0536068d-110c-43a4-82e4-941cdb715042 3f685b7c349c4734afc1a9a87968aca5 52bc463e2bbd46c792ca0d17effd2a86 - - -] Couldn't find multipath device /dev/mapper/360000000000000000e00000000010001

Another experiment is WITHOUT multipath-tools patch [2] + WITH this os-brick patch, the test result can refer the link [3], we can see that multipath device doesn't be delete because this os-brick patch has removed 'multipath -r'.

Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: 2017-03-22 11:06:20.520 17329 DEBUG oslo_concurrency.lockutils [req-bbcf6687-4e5d-4748-9887-726ff8489008 3f685b7c349c4734afc1a9a87968aca5 52bc463e2bbd46c792ca0d17effd2a86 - - -] Lock "connect_volume" acquired by "os_brick.initiator.connector.disconnect_volume" :: waited 0.001s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:273
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: 2017-03-22 11:06:20.523 17329 DEBUG oslo_concurrency.processutils [req-bbcf6687-4e5d-4748-9887-726ff8489008 3f685b7c349c4734afc1a9a87968aca5 52bc463e2bbd46c792ca0d17effd2a86 - - -] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sda execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:344
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: 2017-03-22 11:06:20.705 17329 DEBUG oslo_concurrency.processutils [req-bbcf6687-4e5d-4748-9887-726ff8489008 3f685b7c349c4734afc1a9a87968aca5 52bc463e2bbd46c792ca0d17effd2a86 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sda" returned: 0 in 0.183s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:374
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: 2017-03-22 11:06:20.707 17329 DEBUG os_brick.initiator.connector [req-bbcf6687-4e5d-4748-9887-726ff8489008 3f685b7c349c4734afc1a9a87968aca5 52bc463e2bbd46c792ca0d17effd2a86 - - -] multipath ['-ll', u'/dev/sda']: stdout=360000000000000000e00000000010001 dm-0 IET,VIRTUAL-DISK
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: size=1.0G features='0' hwhandler='0' wp=rw
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: |-+- policy='round-robin 0' prio=1 status=active
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: | `- 4:0:0:1 sda 8:0 active ready running
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: `-+- policy='round-robin 0' prio=1 status=enabled
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: `- 5:0:0:1 sdb 8:16 active ready running
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: stderr= _run_multipath /usr/lib/python2.7/dist-packages/os_brick/initiator/connector.py:1286
Mar 22 11:06:20 juju-zhhuabj-machine-9 nova-compute[17329]: 2017-03-22 11:06:20.709 17329 DEBUG os_brick.i...

@Gustavo,

Liang's comment in patch set 3 [1] can explain your problem, he said:

The dev can disappear momentarily right after 'multipath -r "dev"'. So it doesn't happen for every single path. If it did, it would cause a lot more issues. The multipath dev removal path reloads the dev near the beginning of the operation (_rescan_multipath). Thus the "stat" here can fail if it is executed before the dev node being re-created.

1, 'multipath -r' in _rescan_multipath() [10] can make the multipath dev disappear momentarily [9] due to the bug [9].

2, _get_multipath_device_name() [2] uses 'multipath -ll' command to find multipath device name [3], so we saw:

Jan 25 09:24:40 Lock "connect_volume" acquired by "os_brick.initiator.connector.disconnect_volume" :: waited 0.000s
...
Jan 25 09:24:40 multipath ['-ll', u'/dev/sdr']: stdout=360080e5000297ea40000050658885f45 dm-6 NETAPP,INF-01-00#012

3, _linuxscsi.remove_multipath_device() [4] will invoke remove_multipath_device() [4], so we saw:

Jan 25 09:24:40 remove multipath device /dev/sdr'

4, then find_multipath_device() will be invoked [5], then 'multipath -l' will be invoked [6]

5, the "stat" right after 'multipath -r' here [7] can fail if it is executed before the dev node being re-created. so we saw:

Jan 25 09:24:40 Couldn't find multipath device /dev/mapper/360080e5000297ea40000050658885f45

So the fix [1] was trying to fix this problem, but it was abandoned later because we already have the fix [8], that's also why I am trying to backport it.
FYI, the root cause of your problem is a bug in multipath-tools [9], you can also fix the problem by upgrading multipath-tools.

[1] https://review.openstack.org/#/c/366065
[2] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L925
[3] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L1200
[4] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L935
[5] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L124
[6] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L263
[7] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L288
[8] https://review.openstack.org/#/c/374421/
[9] https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1621340
[10] https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L918

Hi Joshua, I'll check this and give you feedback. Thanks!

On Wed, 22 Mar 2017 at 09:31, Hua Zhang <email address hidden> wrote:

> @Gustavo,
>
> Liang's comment in patch set 3 [1] can explain your problem, he said:
>
> The dev can disappear momentarily right after 'multipath -r "dev"'. So
> it doesn't happen for every single path. If it did, it would cause a lot
> more issues. The multipath dev removal path reloads the dev near the
> beginning of the operation (_rescan_multipath). Thus the "stat" here can
> fail if it is executed before the dev node being re-created.
>
> 1, 'multipath -r' in _rescan_multipath() [10] can make the multipath dev
> disappear momentarily [9] due to the bug [9].
>
> 2, _get_multipath_device_name() [2] uses 'multipath -ll' command to find
> multipath device name [3], so we saw:
>
> Jan 25 09:24:40 Lock "connect_volume" acquired by
> "os_brick.initiator.connector.disconnect_volume" :: waited 0.000s
> ...
> Jan 25 09:24:40 multipath ['-ll', u'/dev/sdr']:
> stdout=360080e5000297ea40000050658885f45 dm-6 NETAPP,INF-01-00#012
>
> 3, _linuxscsi.remove_multipath_device() [4] will invoke
> remove_multipath_device() [4], so we saw:
>
> Jan 25 09:24:40 remove multipath device /dev/sdr'
>
> 4, then find_multipath_device() will be invoked [5], then 'multipath -l'
> will be invoked [6]
>
> 5, the "stat" right after 'multipath -r' here [7] can fail if it is
> executed before the dev node being re-created. so we saw:
>
> Jan 25 09:24:40 Couldn't find multipath device
> /dev/mapper/360080e5000297ea40000050658885f45
>
> So the fix [1] was trying to fix this problem, but it was abandoned later
> because we already have the fix [8], that's also why I am trying to
> backport it.
> FYI, the root cause of your problem is a bug in multipath-tools [9], you
> can also fix the problem by upgrading multipath-tools.
>
> [1] https://review.openstack.org/#/c/366065
> [2]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L925
> [3]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L1200
> [4]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L935
> [5]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L124
> [6]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L263
> [7]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/linuxscsi.py#L288
> [8] https://review.openstack.org/#/c/374421/
> [9] https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1621340
> [10]
> https://github.com/openstack/os-brick/blob/stable/mitaka/os_brick/initiator/connector.py#L918
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1623700
>
> Title:
> [SRU] multipath iscsi does not logout of sessions on xenial
>
> Status in Ubuntu Cloud Archive:
> Fix Released
> Status in Ubuntu Cloud Archive mitaka series:
> Triaged
> Status in Ubuntu Cloud Archive newton series:
> Triaged
> Status in os-brick:
> Fix Released
> Status in python-os-brick package ...

Hi Joshua, I've done some tests and it's OK!
Thank you

Thanks zhhuabj and gustavo! I've uploaded the patches to yakkety and xenial.

Hello Patrick, or anyone else affected,

Accepted python-os-brick into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-brick/1.2.0-2ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Patrick, or anyone else affected,

Accepted python-os-brick into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-brick/1.6.1-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified successfully on both xenial-proposed and yakkety-proposed. http://paste.ubuntu.com/24365563/

It is worth mentioning that just now our customer also give me feedback that the package in xenial-proposed solved their problem. thanks.

Hello Patrick, or anyone else affected,

Accepted python-os-brick into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Patrick, or anyone else affected,

Accepted python-os-brick into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package python-os-brick - 1.2.0-2ubuntu0.3

---------------
python-os-brick (1.2.0-2ubuntu0.3) xenial; urgency=medium

  * d/p/Stop-calling-multipath-r-when-attaching-detaching-iS.patch: Backport
    Backport fix for stopping calling 'multipath -r' (LP: #1623700)

 -- Zhang Hua <email address hidden> Fri, 17 Mar 2017 19:23:13 +0800

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 1.6.1-0ubuntu1.2

---------------
python-os-brick (1.6.1-0ubuntu1.2) yakkety; urgency=medium

  * d/p/stop-calling-multipath-r-when-attaching-detaching-iS.patch
   Backport fix for stopping calling 'multipath -r' (LP: #1623700)

 -- Hua Zhang <email address hidden> Wed, 22 Mar 2017 18:04:15 +0800

I've verified that the new package (1.6.1-0ubuntu1.2~cloud0 and 1.2.0-2ubuntu0.3~cloud0) doesn't break anything and fix the problem. see https://pastebin.canonical.com/186800/ and https://pastebin.canonical.com/186797/

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 1.2.0-2ubuntu0.3~cloud0
---------------

 python-os-brick (1.2.0-2ubuntu0.3~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-os-brick (1.2.0-2ubuntu0.3) xenial; urgency=medium
 .
   * d/p/Stop-calling-multipath-r-when-attaching-detaching-iS.patch: Backport
     Backport fix for stopping calling 'multipath -r' (LP: #1623700)

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 1.6.1-0ubuntu1.2~cloud0
---------------

 python-os-brick (1.6.1-0ubuntu1.2~cloud0) xenial-newton; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-os-brick (1.6.1-0ubuntu1.2) yakkety; urgency=medium
 .
   * d/p/stop-calling-multipath-r-when-attaching-detaching-iS.patch
    Backport fix for stopping calling 'multipath -r' (LP: #1623700)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/447018
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.