Stack Abandon is unsafe

Bug #1353670 reported by Zane Bitter
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Jason Dunsmore

Bug Description

When abandoning a stack, we return the critical data to the user only *after* commencing a destructive delete of it. If the user fails to get the data for any reason (the connection to the API drops, network connectivity to Qpid is lost, an error occurs in heat-api...) then the user will have no (automated) way of recovering the data necessary to recreate the stack.

Revision history for this message
Vijendar Komalla (vijendar-komalla) wrote :

Zane,
I am thinking about fixing following way. Let me know if you have any comments.

1. Introduce a new 'pre-abandon' (or may be called 'freeze') step and in this step return the stack data without performing any destructive operation. And also set stack status as ['pre-abandon', 'complete']
2. Only operations allowed on a stack in 'pre-abandon' state are 'adopt' and 'pre-abandon'. So if for some reason data was lost, then user can again issue 'pre-abandon' and get the data back
3. In stack-abandon step do the actual stack abandon which is destructive

Revision history for this message
Zane Bitter (zaneb) wrote :

Sounds good. To make sure that it is used correctly though you will need to make the actual abandon step not return any useful data (otherwise people will keep on using it the unsafe way).

So maybe the two steps should be:

1) Abandon
    - like the pre-abandon you mention
    - Moves the stack to ABANDON_COMPLETE state
    - Valid actions in ABANDON_COMPLETE are ADOPT and DELETE

2) Delete
    - Doesn't remove the underlying physical resources if the current state is ABANDON_COMPLETE
    - Refuses to run if the current state is ABANDON_FAILED

Revision history for this message
Vijendar Komalla (vijendar-komalla) wrote :

Thanks Zane for your inputs.

Angus Salkeld (asalkeld)
Changed in heat:
status: New → Triaged
Revision history for this message
Anant Patil (ananta) wrote :

May be we should have stack-abandon-data or stack-abandon-show which retrieves data after the stack is in ABANDON_COMPLETE state. This new call should be a idempotent call after the stack is abandoned. Just my 2 cents!

Anant Patil (ananta)
Changed in heat:
assignee: Vijendar Komalla (vijendar-komalla) → Anant Patil (ananta)
Zane Bitter (zaneb)
tags: added: abandon-adopt
Revision history for this message
Anant Patil (ananta) wrote :

Hi Zane,
Do we need to split abandon into two user visible calls: abandon and delete?

Revision history for this message
Zane Bitter (zaneb) wrote :

IMHO yes

Anant Patil (ananta)
Changed in heat:
assignee: Anant Patil (ananta) → nobody
Angus Salkeld (asalkeld)
Changed in heat:
importance: High → Medium
milestone: none → liberty-1
Changed in heat:
assignee: nobody → Kanagaraj Manickam (kanagaraj-manickam)
Revision history for this message
Kanagaraj Manickam (kanagaraj-manickam) wrote :

abandon action is essentially doing the export of stack template + data and forcible delete. So could we convert the abandon option in to below new+old actions:

1. [new] export: export the stack template + all resources with it's data (abandon first step)
2. [old] delete: delete the stack. (abandon second step)

so abandon = export + delete

now user would be able to get the data first using export action, once data is completely available, user could go for delete action directly. so abandon could be renamed to export and remove the deletion (second) part, which can be achieved by using 'delete' action.

User could use the Exported data for adopting later.

Revision history for this message
Zane Bitter (zaneb) wrote :

The trick is knowing when 'delete' deletes the underlying physical resources (normal case) and when it doesn't (abandon case). That's requires either a state change in the export phase or a parameter passed to the delete phase. I'm relaxed about which it is, although I think it's somewhat unusual to be passing data with a DELETE request in a ReST API.

Revision history for this message
Kanagaraj Manickam (kanagaraj-manickam) wrote :

zaneb, thanks for pointing out the diff.

based on your inputs, i tried to frame the following state flow:

export -> will set stack to (EXPORT, [COMPLETED|FAILED])
abandon -> ONLY on (EXPORT, COMPLETED) allow abandon
                      set to (ABANDON, [COMPLETED|FAILED])
delete -> on (ABANDON, FAILED), (EXPORT, [COMPLETED|FAILED]) don't allow to delete, otherwise, like current behavior
delete (with force_delete parameter) -> even on on (ABANDON, FAILED) allow to force delete

Here, the last option force_delete helps to delete the stack, in case user need to delete forcefully.

kindly provide your feedback. Thanks.

Revision history for this message
Zane Bitter (zaneb) wrote :

I don't understand why you're suggesting 3 states: EXPORT/ABANDON/DELETE. Did you read my comment #2? I don't care about names, but what functionality do you think was missing from that?

As I mentioned, options to DELETE are unusual in ReST, so force_delete is questionable.

Revision history for this message
Kanagaraj Manickam (kanagaraj-manickam) wrote :

Zane, I am with your approach mentioned at comment #2. and I named the pre-abandon step as 'export' as its nature of the task is exporting the stack data. (And there is no other diff). so first, user would export the data and on successful export, user would abandon the stack

In the previous comment, For the DELETE action, i tried to mention in the context of abandon. but its not as part of abandon actions (kindly ignore it if it created an confusion)

Thanks for your input.

Revision history for this message
Kanagaraj Manickam (kanagaraj-manickam) wrote :
Changed in heat:
status: Triaged → In Progress
Changed in heat:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190517

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/190518

Changed in heat:
milestone: liberty-1 → liberty-2
Changed in heat:
milestone: liberty-2 → liberty-3
Changed in heat:
milestone: liberty-3 → next
Changed in heat:
assignee: Kanagaraj Manickam (kanagaraj-manickam) → Jason Dunsmore (jasondunsmore)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/181880
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=bbb770841ad5e684be21925a249faa8301e7f614
Submitter: Jenkins
Branch: master

commit bbb770841ad5e684be21925a249faa8301e7f614
Author: Kanagaraj Manickam <email address hidden>
Date: Mon May 11 17:34:43 2015 +0530

    Split abandon into pre-abandon(export) and abandon

    Splits the existing abandon action into two actions as
    below:
    1. export: User is recommended to run export successfully
       before doing the abandon action. It returns the stack
       data, which can be used for adopting stack.
    2. abandon: Abandons the given stack

    Partial-Bug: #1353670

    Co-Authored-By: Jason Dunsmore <email address hidden>
    Change-Id: I65264d91b1378df9cb3e492bc6ccaa778940dd6b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Jason Dunsmore (<email address hidden>) on branch: master
Review: https://review.openstack.org/195053
Reason: This patch series took a different approach (no is_forced param), so this patch is no longer valid.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/250951

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Jason Dunsmore (<email address hidden>) on branch: master
Review: https://review.openstack.org/190517
Reason: Abandoning in favor of https://review.openstack.org/#/c/190518

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/190518
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=47d59f2b1e5d0e1d0e2861f3e58e1a04e962d902
Submitter: Jenkins
Branch: master

commit 47d59f2b1e5d0e1d0e2861f3e58e1a04e962d902
Author: Kanagaraj Manickam <email address hidden>
Date: Thu Jun 11 09:13:43 2015 +0530

    Adds RPC api for export action

    Adds RPC client api for export action and test cases.

    RPC_VERSION = 1.22
    Partial-Bug: #1353670

    Co-Authored-By: Jason Dunsmore <email address hidden>
    Change-Id: I88ef9bba073bf4e730e1dae114a9c5d3b856ca13

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/250951
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c63411eef60f6f851649544599ce39018a6aad3c
Submitter: Jenkins
Branch: master

commit c63411eef60f6f851649544599ce39018a6aad3c
Author: Jason Dunsmore <email address hidden>
Date: Fri Nov 27 15:19:46 2015 -0600

    Add REST API for stack export

    APIImpact
    Closes-Bug: #1353670
    Change-Id: I94d5abf5bfe148b4f25a2a8891e3cf4d1774e373

Changed in heat:
milestone: next → mitaka-3
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/heat 6.0.0.0b3

This issue was fixed in the openstack/heat 6.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.