[SRU] iSCSI+Multipath: Volume attachment hungs if sessiong scanning fails

patch: https://review.opendev.org/c/openstack/os-brick/+/775545

This issue was fixed in the openstack/os-brick 4.3.0 release.

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/os-brick/+/792359

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/792359
Committed: https://opendev.org/openstack/os-brick/commit/c336cb76b318f987d43ff908a5b7c9d40b5c4dd6
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit c336cb76b318f987d43ff908a5b7c9d40b5c4dd6
Author: Takashi Kajinami <email address hidden>
Date: Mon Feb 15 10:58:13 2021 +0900

    Avoid unhandled exceptions during connecting to iSCSI portals

    Currently we don't properly catch some possible exceptions during
    connectiing to iSCSI portals, like failures in "iscsiadm -m session".
    Because of this _connect_vol threads can abort unexpectedly in some
    failure patterns, and this abort causes hung in subsequent steps
    waiting for results from _connct_vol threads.

    This change ensures that any exceptions during connecting to iSCSI
    portals are handled in the _connect_vol method corectly, to avoid
    unexpected abort without updating thread results.

    Conflicts:
            os_brick/tests/initiator/connectors/test_iscsi.py

    Closes-Bug: #1915678
    Change-Id: I0278c502806b99f8ec65cb146e3852e43031e9b8
    (cherry picked from commit 4478433550ddab970f106636a4ba5fbba82c5e39)
    (cherry picked from commit 57c8f4334c0e58b151f449f3316bb9e1c8ed6da4)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/os-brick/+/796671

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/796671
Committed: https://opendev.org/openstack/os-brick/commit/d09c385c24288dced7bd685072fe850422dd14d9
Submitter: "Zuul (22348)"
Branch: stable/train

commit d09c385c24288dced7bd685072fe850422dd14d9
Author: Takashi Kajinami <email address hidden>
Date: Mon Feb 15 10:58:13 2021 +0900

    Avoid unhandled exceptions during connecting to iSCSI portals

    Currently we don't properly catch some possible exceptions during
    connectiing to iSCSI portals, like failures in "iscsiadm -m session".
    Because of this _connect_vol threads can abort unexpectedly in some
    failure patterns, and this abort causes hung in subsequent steps
    waiting for results from _connct_vol threads.

    This change ensures that any exceptions during connecting to iSCSI
    portals are handled in the _connect_vol method corectly, to avoid
    unexpected abort without updating thread results.

    Conflicts:
            os_brick/tests/initiator/connectors/test_iscsi.py

    Closes-Bug: #1915678
    Change-Id: I0278c502806b99f8ec65cb146e3852e43031e9b8
    (cherry picked from commit 4478433550ddab970f106636a4ba5fbba82c5e39)
    (cherry picked from commit 57c8f4334c0e58b151f449f3316bb9e1c8ed6da4)
    (cherry picked from commit c336cb76b318f987d43ff908a5b7c9d40b5c4dd6)

The fix[1] was already released in os-brick 4.3.0 .

[1] https://review.opendev.org/c/openstack/os-brick/+/792359

This issue was fixed in the openstack/os-brick 3.0.7 release.

Thanks for backporting the fix to Train release!
Can we please generate a new patch (2.10.8) for this issue, since currently Openstack Train upper-constraints file is pointing to 2.10.7 which doesn't have this fix.

This issue was fixed in the openstack/os-brick 4.0.4 release.

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/os-brick/+/814840

Hi @kajinamit, I'm proposing your fix be back ported to Stein:

https://review.opendev.org/c/openstack/os-brick/+/814840

> Can we please generate a new patch (2.10.8) for this issue, since currently Openstack Train upper-constraints file is pointing to 2.10.7 which doesn't have this fix.

My understanding is that Train has already entered into EM phase and a new release is no longer created. I understand your point but I'm not sure whether/how we can raise an exception.

> Hi @kajinamit, I'm proposing your fix be back ported to Stein:
Thanks. I think the issue affects older branches as well so I agree it'd make sense to backport that to older releases.

Thanks Brett. I've uploaded to the focal and bionic unapproved queues where it is awaiting review by the SRU team, as well as to the cloud archive staging PPAs.

Hello Takashi, or anyone else affected,

Accepted python-os-brick into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Takashi, or anyone else affected,

Accepted python-os-brick into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-brick/3.0.1-0ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Takashi, or anyone else affected,

Accepted python-os-brick into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-brick/2.3.0-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Takashi, or anyone else affected,

Accepted python-os-brick into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Takashi, or anyone else affected,

Accepted python-os-brick into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Takashi, or anyone else affected,

Accepted python-os-brick into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi Corey,

I've tested python-os-brick package 2.8.1-0ubuntu1~cloud0.2 from cloud-archive:stein-proposed as per the Test Case and can confirm this fixes the issue for me.

Hey Brett,

Thanks for testing on stein!

Would it be possible for you to test other/newer releases as well?

I'm asking because fixes for older releases have to land on newer releases as well, if applicable, which needs verification too, but it seems testing for this bug is on short supply; so if you're still running hot with it, that would be great, if at all possible.

And I'd imagine that releases on UCA that still have an Ubuntu release supported (e.g., ussuri -> focal) depend on the fix landing on the Ubuntu release as well.

(Sorry, I know it's a lot to ask; don't shoot the messenger :) just trying to get this fix released, and I realize it's been sitting/waiting on testing for a long while, until you showed up! ;)

Thanks!
Mauricio

P.S.: if that helps, it looks like Train isn't needed/supported anymore [1], unless the fix is required for upgrades.

[1] https://ubuntu.com/about/release-cycle#ubuntu-openstack-release-cycle

Hi Mauricio,

I intend to get there, this was just the first one I had ready to go :) I'll update shortly.

Hi Corey,

I've tested python-os-brick package 4.0.1-0ubuntu1~cloud1 from cloud-archive:victoria-proposed as per the Test Case and can confirm this fixes the issue for me.

Hey Brett, awesome! Thanks a bunch :)

I've tested python-os-brick package 3.0.1-0ubuntu1.4~cloud0 from cloud-archive:ussuri-proposed as per the Test Case and can confirm this fixes the issue for me.

I've tested python-os-brick package 2.10.0-0ubuntu1~cloud0.2 from cloud-archive:train-proposed as per the Test Case and can confirm this fixes the issue for me.

I've tested python-os-brick package 3.0.1-0ubuntu1.4 from focal proposed as per the Test Case and can confirm this fixes the issue for me.

I've tested python-os-brick package 2.3.0-0ubuntu1.3 from bionic proposed as per the Test Case and can confirm this fixes the issue for me.

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 4.0.1-0ubuntu1~cloud1
---------------

 python-os-brick (4.0.1-0ubuntu1~cloud1) focal-victoria; urgency=medium
 .
   [ Brett Milford ]
   * Avoid unhandled exceptions during connecting to iSCSI portals
     (LP: #1915678)
     - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/victoria branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

This bug was fixed in the package python-os-brick - 3.0.1-0ubuntu1.4

---------------
python-os-brick (3.0.1-0ubuntu1.4) focal; urgency=medium

  [ Brett Milford ]
  * Avoid unhandled exceptions during connecting to iSCSI portals
    (LP: #1915678)
    - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

 -- Brett Milford <email address hidden> Fri, 05 Nov 2021 10:55:06 +1000

This bug was fixed in the package python-os-brick - 2.3.0-0ubuntu1.3

---------------
python-os-brick (2.3.0-0ubuntu1.3) bionic; urgency=medium

  [ Brett Milford ]
  * Avoid unhandled exceptions during connecting to iSCSI portals
    (LP: #1915678)
    - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

 -- Brett Milford <email address hidden> Fri, 05 Nov 2021 10:55:06 +1000

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 3.0.1-0ubuntu1.4~cloud0
---------------

 python-os-brick (3.0.1-0ubuntu1.4~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-os-brick (3.0.1-0ubuntu1.4) focal; urgency=medium
 .
   [ Brett Milford ]
   * Avoid unhandled exceptions during connecting to iSCSI portals
     (LP: #1915678)
     - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 2.10.0-0ubuntu1~cloud0.2
---------------

 python-os-brick (2.10.0-0ubuntu1~cloud0.2) bionic-train; urgency=medium
 .
   [ Brett Milford ]
   * Avoid unhandled exceptions during connecting to iSCSI portals
     (LP: #1915678)
     - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

The verification of the Stable Release Update for python-os-brick has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-os-brick - 2.8.1-0ubuntu1~cloud0.2
---------------

 python-os-brick (2.8.1-0ubuntu1~cloud0.2) bionic-stein; urgency=medium
 .
   [ Brett Milford ]
   * Avoid unhandled exceptions during connecting to iSCSI portals
     (LP: #1915678)
     - d/p/lp1915678-avoid-unhandled-exceptions-during-connecting-to-iscsi-portals.patch
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/os-brick/+/814840
Reason: Stein transitioned to End of Life by change Icf9a539a7b8b and is accepting no more changes.

This issue was fixed in the openstack/os-brick train-eol release.