Cancel update action for stack is failed with nova server

Bug #1570908 reported by Oleksii Chuprykov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Anant Patil

Bug Description

1. Create new stack
2. Launch heat stack-update stack_name
3. Launch heat stack-cancel-update stack_name while update operation is in progress
4. Check state of stack after cancel update

Is failed on step 3 with error ROLLBACK_FAILED.

Stack is created with following template:

description: 'Template which creates single instance
 '
heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
 port:
   type: OS::Neutron::Port
   properties:
     network: {get_param: network}
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: port}
   type: OS::Nova::Server

During update the instance type is changed from tiny to small

-------------------------------------------------------------------------

The same bug was reproduced with template, where used just network property. In this case nova will create port in neutron.
During update replace heat detaches ports and it has downside for nova created ports - they will be deleted by nova during this detach operation.
So we need to handle this case separately and try to attach new port after cancel-update as it was done by nova.

1. Create new stack
2. Launch heat stack-update stack_name
3. Launch heat stack-cancel-update stack_name while update operation is in progress
4. Check state of stack after cancel update

Is failed on step 3 with error ROLLBACK_FAILED.

Stack is created with following template:

description: 'Template which creates single instance
 '
heat_template_version: '2013-05-23'
outputs:
 servers:
   value: {get_resource: Server}
parameters:
 ImageId: {type: string}
 InstanceType: {type: string}
 network: {type: string}
resources:
 Server:
   properties:
     flavor: {get_param: InstanceType}
     flavor_update_policy: REPLACE
     image: {get_param: ImageId}
     name: ost1-test_heat
     networks:
     - network: {Ref: network}
   type: OS::Nova::Server

During update the instance type is changed from tiny to small

Changed in heat:
assignee: nobody → Oleksii Chuprykov (ochuprykov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/306490

Changed in heat:
status: New → In Progress
Changed in heat:
milestone: none → newton-1
importance: Undecided → Medium
Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

incomplete on fuel-9.0-223-2016-04-20_02-00-00.iso

Changed in heat:
importance: Medium → High
Rabi Mishra (rabi)
Changed in heat:
milestone: newton-1 → newton-2
Thomas Herve (therve)
Changed in heat:
milestone: newton-2 → newton-3
Thomas Herve (therve)
Changed in heat:
milestone: newton-3 → newton-rc1
Changed in heat:
assignee: Oleksii Chuprykov (ochuprykov) → Thomas Herve (therve)
Changed in heat:
assignee: Thomas Herve (therve) → Zane Bitter (zaneb)
Changed in heat:
assignee: Zane Bitter (zaneb) → Anant Patil (ananta)
Thomas Herve (therve)
Changed in heat:
milestone: newton-rc1 → none
milestone: none → ocata-1
milestone: ocata-1 → nexton-rc2
Revision history for this message
Anant Patil (ananta) wrote :

There were two reasons for this test to fail:
1. An issue with re-triggering new traversal as described in https://bugs.launchpad.net/heat/+bug/1625073
2. The restore_prev_resource in server_network_mixin has to care of a situation mentioned above bug, i.e. a replacement is created but it could never take off because the stack was rolled back. So the replacement resource exists in DB but there is no physical resource. I have updated the patch https://review.openstack.org/306490

Zane Bitter (zaneb)
tags: added: newton-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/306490
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=584efe3329143e28fdb42b5d9496977f5cdf275a
Submitter: Jenkins
Branch: master

commit 584efe3329143e28fdb42b5d9496977f5cdf275a
Author: Oleksii Chuprykov <email address hidden>
Date: Fri Apr 29 17:03:17 2016 +0300

    Fix cancel update for nova server with defined port

    This particular patch fixes a behaviour of cancel update for
    nova server with defined port, so there are no ports manageable
    by nova. We have these issues while restoring ports after rollback:
    1) We doesn't detach any ports from current server, because we
    doesn't save them to resoruce data. (we store this data after
    succesfull create of the server)
    2) Detaching an interface from current server will fail, if the server
    will be in building state, so we need to wait until server will be
    in active or in error state.
    Refresh ports list to solve problem (1).
    Wait until nova moves to active/error state to solve (2).
    A functional test to prove the fix was added. Note, that this test is
    skipped for convergence engine tests until cancel update will work
    properly in convergence mode (see bug 1533176).
    Partial-Bug: #1570908
    Change-Id: If6fd916068a425eea6dc795192f286cb5ffcb794

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/373615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/newton)

Reviewed: https://review.openstack.org/373615
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9ece778613d67feff35342aeaa6618aaa5f6fc85
Submitter: Jenkins
Branch: stable/newton

commit 9ece778613d67feff35342aeaa6618aaa5f6fc85
Author: Oleksii Chuprykov <email address hidden>
Date: Fri Apr 29 17:03:17 2016 +0300

    Fix cancel update for nova server with defined port

    This particular patch fixes a behaviour of cancel update for
    nova server with defined port, so there are no ports manageable
    by nova. We have these issues while restoring ports after rollback:
    1) We doesn't detach any ports from current server, because we
    doesn't save them to resoruce data. (we store this data after
    succesfull create of the server)
    2) Detaching an interface from current server will fail, if the server
    will be in building state, so we need to wait until server will be
    in active or in error state.
    Refresh ports list to solve problem (1).
    Wait until nova moves to active/error state to solve (2).
    A functional test to prove the fix was added. Note, that this test is
    skipped for convergence engine tests until cancel update will work
    properly in convergence mode (see bug 1533176).

    Partial-Bug: #1570908
    Change-Id: If6fd916068a425eea6dc795192f286cb5ffcb794
    (cherry picked from commit 584efe3329143e28fdb42b5d9496977f5cdf275a)

tags: added: in-stable-newton
Thomas Herve (therve)
Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/316047
Reason: Hi, this patch stays untouched for few months, would like to abandon for now. Feel free to restore it if you still working on this. Thank you!

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/415815

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/mitaka)

Change abandoned by Sergey Kraynev (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/415815
Reason: Agree, I will abandone it. It was proposed before phase 3 support :) however now it's outdated

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.