masakari

Failed to recover stopped instance

Bug #1782517 reported by Rikimaru Honjo
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Stein
Fix Committed
High
Unassigned
Train
Fix Released
High
Unassigned
Ussuri
Fix Released
High
Unassigned
Victoria
Fix Released
High
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
masakari
Fix Released
Undecided
takahara.kengo
Train
Fix Released
Undecided
Unassigned
Ussuri
Fix Released
Undecided
Unassigned
Victoria
Fix Released
Undecided
Unassigned
Wallaby
Fix Released
Undecided
takahara.kengo
masakari (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
High
Unassigned
Hirsute
Fix Released
Undecided
Unassigned

Bug Description

[Error]
Recovering host-failure was failed when there was stopped state instance on the failed host.
As a result, notification status became "failed".
(Instance's vm_state after evacuation became "stopped".)

I used the latest version of masakari.

[Cause of error]
Masakari will try to call stop API after evacuating.
But, evacuate API stops the instance at the end if the original vm_state is stopped.
So 409 error was occurred when masakari called stop API after evacuating.

== Ubuntu SRU Details below ==
[Impact]
See above

[Test Case]
For focal:
Test with an actual juju deployed masakari openstack deployment and ensure the reported bug is fixed on host failure.

For all other releases the fix can be verified with an LXD container for the corresponding release:
$ sudo apt install python3-masakari
$ cd /usr/lib/python3/dist-packages
$ python3 -m unittest masakari.tests.unit.engine.drivers.taskflow.test_host_failure_flow.HostFailureTestCase.test_host_failure_flow_for_stopped_instances

The unit test will be successful on a patched deployment and will fail with a mismatch error in test_host_failure_flow_for_stopped_instances.

[Where problems coud occur]
Any regressions in this fix will likely result in similar failures to what was reported in this bug, resulting in a failure to recover an instance on host failure. The patch is a small, targeted change with a good unit test and the code is unchanged across the backports which helps mitigate regression potential.

See original description
takahara.kengo (takahara.kengo)
description: updated
takahara.kengo (takahara.kengo)
Changed in masakari:
assignee: nobody → takahara.kengo (takahara.kengo)
Revision history for this message
takahara.kengo (takahara.kengo) wrote :

Posted a patch.
https://review.openstack.org/585625

Changed in masakari:
status: New → In Progress
suzhengwei (sue.sam)
Changed in masakari:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/masakari 9.1.1

This issue was fixed in the openstack/masakari 9.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/masakari 10.0.1

This issue was fixed in the openstack/masakari 10.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/masakari 8.1.1

This issue was fixed in the openstack/masakari 8.1.1 release.

Billy Olsen (billy-olsen)
Changed in masakari (Ubuntu Hirsute):
status: New → Fix Released
Corey Bryant (corey.bryant)
description: updated
Changed in masakari (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in masakari (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant)
description: updated
Revision history for this message
Corey Bryant (corey.bryant) wrote :

For Ubuntu and the cloud archive this has been uploaded to groovy and focal unapproved queues as well as the staging PPAs for train and stein:

https://launchpad.net/ubuntu/groovy/+queue?queue_state=1&queue_text=masakari
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=masakari
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/train-staging/+packages?field.name_filter=masakari
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/stein-staging/+packages?field.name_filter=masakari

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/masakari 11.0.0.0rc1

This issue was fixed in the openstack/masakari 11.0.0.0rc1 release candidate.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Rikimaru, or anyone else affected,

Accepted masakari into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/masakari/10.0.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in masakari (Ubuntu Groovy):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-groovy
Changed in masakari (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Rikimaru, or anyone else affected,

Accepted masakari into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/masakari/9.0.0-0ubuntu0.20.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Rikimaru, or anyone else affected,

Accepted masakari into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-victoria-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Rikimaru, or anyone else affected,

Accepted masakari into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Rikimaru, or anyone else affected,

Accepted masakari into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Rikimaru, or anyone else affected,

Accepted masakari into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Revision history for this message
Rikimaru Honjo (honjo-rikimaru-c6) wrote :

Hi,

I confirmed that this issue was fixed in the following version masakari. I change tags after that.

10.0.0-0ubuntu2
9.0.0-0ubuntu0.20.04.4
latest stable/victoria branch(with devstack)
latest stable/ussuri branch(with devstack)
latest stable/train branch(with devstack)
latest stable/stein branch(with devstack)

(Sorry, I don't have time to test victoria/ussuri/train/stein deb packages. I used stable branches instead of packages.)

tags: added: verification-done-focal verification-done-groovy verification-stein-done verification-train-done verification-ussuri-done verification-victoria-done
removed: verification-needed-focal verification-needed-groovy verification-stein-needed verification-train-needed verification-ussuri-needed verification-victoria-needed
tags: added: verification-stein-needed
removed: verification-stein-done
Revision history for this message
Rikimaru Honjo (honjo-rikimaru-c6) wrote :

Sorry, I made a mistake.

> latest stable/stein branch(with devstack)

This patch was not ported to stein.

So I reverted "verification-stein-done" tag.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package masakari - 10.0.0-0ubuntu2

---------------
masakari (10.0.0-0ubuntu2) groovy; urgency=medium

  * d/gbp.conf: Create stable/victoria branch.
  * d/p/modified-recovery-flow-for-stopped-instance.patch: Cherry-picked
    from upstream stable branch to fix recovery of stopped instances
    (LP: #1782517).

 -- Corey Bryant <email address hidden> Thu, 25 Mar 2021 16:21:08 -0400

Changed in masakari (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for masakari has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package masakari - 9.0.0-0ubuntu0.20.04.4

---------------
masakari (9.0.0-0ubuntu0.20.04.4) focal; urgency=medium

  * d/p/modified-recovery-flow-for-stopped-instance.patch: Cherry-picked
    from upstream stable branch to fix recovery of stopped instances
    (LP: #1782517).

 -- Corey Bryant <email address hidden> Thu, 25 Mar 2021 16:25:14 -0400

Changed in masakari (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for masakari has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package masakari - 10.0.0-0ubuntu2~cloud0
---------------

 masakari (10.0.0-0ubuntu2~cloud0) focal-victoria; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 masakari (10.0.0-0ubuntu2) groovy; urgency=medium
 .
   * d/gbp.conf: Create stable/victoria branch.
   * d/p/modified-recovery-flow-for-stopped-instance.patch: Cherry-picked
     from upstream stable branch to fix recovery of stopped instances
     (LP: #1782517).

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for masakari has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package masakari - 8.0.0-0ubuntu1~cloud2
---------------

 masakari (8.0.0-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   * d/p/modified-recovery-flow-for-stopped-instance.patch: Cherry-picked
     from upstream stable branch to fix recovery of stopped instances
     (LP: #1782517).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.