OpenStack Heat

Stack update on "OS::Nova::Server" resource fails due instance port still in use

Bug #1301486 reported by Nick Martin
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steve Baker
OpenStack Heat 2014.2 "juno"

Bug Description

Here's the resource:
"instance": {
            "Type": "OS::Nova::Server",
            "Properties": {
                "name": {"Fn::Join" : ["-", [ {"Ref" : "AWS::StackName"}, "cloudview01"]]},
                "availability_zone": "az1",
                "image": { "Ref" : "PSOStandardUbuntuLinuxImageId" },
                "flavor": "standard.medium",
                "key_name": { "Ref" : "StackProvisioningSSHKeyName" },
                "networks": [{"port": { "Ref" : "port_floating" }}],
                "metadata": {"tags": {"Fn::Join" : ["", ["Role_role::infra_services::cloudview,Stack_", {"Ref" : "AWS::StackName"}, ",Provisioner_Auto"]]}},
  "flavor_update_policy" : "REPLACE"
            }
        }

The update changed the instance's flavor from standard.small to standard.mendium.

The final failing Heat event:
            "logical_resource_id": "instance",
            "resource_status_reason": "Conflict: Port 3f1e9c8e-ac1c-4d78-95b9-b10301a2690c is still in use. (HTTP 409) (Request-ID: req-46f28a5c-d795-4bb7-8d2c-8edc30ca6256)",
            "resource_status": "CREATE_FAILED"

I believe this bug is related to, but possibly distinct from this defect: https://bugs.launchpad.net/heat/+bug/1280151

It's distinct because it affects "OS::Nova::Server" resources instead of "ec2::instance" resources. In addition, in my update I am trying to change the flavor, not add a new port.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

I have worked with Nick on this and asked him to report this. It seems that it is a legitimate problem and likely just a cargo-culted copy of bug 1280151.

Changed in heat:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

It could be the fix for bug #1285067 was incomplete and my comment #2 could point the way to a fix https://bugs.launchpad.net/heat/+bug/1285067/comments/2

tags: added: icehouse-rc-potential
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

This can be reproduced with the attached template and the following commands:

  heat stack-create -f private/bug-1301486-pre.yaml -P flavor=m1.small -P network=8f150111-8a59-4284-b690-890187e0f8de a
  heat stack-update -f private/bug-1301486-pre.yaml -P flavor=m1.medium -P network=8f150111-8a59-4284-b690-890187e0f8de a
  heat resource-show a the_server

Changed in heat:
assignee: nobody → Steve Baker (steve-stevebaker)
milestone: none → juno-1
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I'm coming to the conclusion that a neutron port cannot be modelled as a heat resource, at least for server interfaces.

Calling nova interface-detach doesn't only detach the port from the server, it also deletes the port! Also, deleting a nova server will delete associated ports even when the port was passed in by ID.

At this point I think this needs to be added to the list of things which doesn't work during stack-update. The fix will be to extend the server networks property schema so that port properties can be specified inline, and port lifecycle is managed entirely within the server resource.

Steve Baker (steve-stevebaker)
tags: removed: icehouse-rc-potential
Changed in heat:
assignee: Steve Baker (steve-stevebaker) → nobody
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-1 → juno-2
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

This bug can be revisited once this nova port fix lands https://review.openstack.org/#/c/77043/

Zane Bitter (zaneb)
Changed in heat:
milestone: juno-2 → juno-3
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-3 → juno-rc1
Steve Baker (steve-stevebaker)
Changed in heat:
assignee: nobody → Steve Baker (steve-stevebaker)
Steve Baker (steve-stevebaker)
Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/121693

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/122568

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/121693
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=30ece56a2118bff073e36a108324d12ee646fab6
Submitter: Jenkins
Branch: master

commit 30ece56a2118bff073e36a108324d12ee646fab6
Author: Steve Baker <email address hidden>
Date: Mon Sep 15 16:37:29 2014 +1200

    Default port policy to force replacement

    This change adds a replacement_policy property to OS::Neutron::Port
    which defaults to always replacing the port on stack-update
    regardless of any changes to the template.

    Currently when a server is replaced, the new server is booted with
    the port that is still attached to the old server, which raises a
    port-still-in-use error.

    Even if heat managed to detach and attach a single port, Nova currently
    deletes all ports on interface-detach and server delete, so the port
    would no longer be available anyway.

    This change ensures that a new server is always booted with a new port,
    so it fixes the above 2 issues, however there are implications for
    other scenarios.

    If the server is not replaced during the stack-update, the server
    handle_update will detach/attach to the new port, so this is fine.

    If the port specifies a fixed_ips ip_address which doesn't change
    during stack-update, an error will be raised that 2 ports exist
    with the same IP address. The only current workaround would be
    to set update_policy:AUTO and not make any changes to the server
    which results in server replacement (or do 2 stack updates using
    a transition ip_address).

    Likewise, update_policy:AUTO will need to be set on any port
    consumed by resources which don't support update without replacement
    (such as OS::Neutron::FloatingIP and OS::Trove::Instance)

    Change-Id: I558db6bac196f49e5c488a577f0580c934b06747
    Closes-Bug: #1301486
    Related-Bug: #1158684
    Related-Bug: #1369748

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/122568
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d0d925f45170634a7ef8b11f15b7971513fa9227
Submitter: Jenkins
Branch: master

commit d0d925f45170634a7ef8b11f15b7971513fa9227
Author: Steve Baker <email address hidden>
Date: Fri Sep 19 11:43:11 2014 +1200

    FloatingIP updateable port_id, fixed_ip_address

    This change implements handle_update for FloatingIP.

    FloatingIPAssociation already has handle_update implemented, so
    this change is just fixing a gap.

    Updateable FloatingIP will be vital if ports end up getting replaced
    on every update as per I558db6bac196f49e5c488a577f0580c934b06747

    Change-Id: I0a05b8cf916267338e4bcb6a3e12bc2cdcdc3d48
    Closes-Bug: #1369748
    Related-Bug: #1301486

Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-rc1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.