test_stack_update_from_failed_patch is too fast

Bug #1583630 reported by Thomas Herve
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Jaime Guerrero

Bug Description

It's a bit weird to say, but I think we're too fast in some cases, and our testing isn't able to handle it. Seeing the failure at http://logs.openstack.org/02/317702/1/gate/gate-heat-dsvm-functional-convg-mysql-lbaasv1/bf3009c/ it seems updated_time stays the same after the successful update, presumably because we manage to update it twice in second? Considering the stack has been created just 1 second before, and that we only use TestResource, that's not impossible (we only have 1 second resolution).

As a short term fix we may need to sleep for 1 second in the test.

Revision history for this message
Thomas Herve (therve) wrote :

Note: we can simply use wait_secs on the resource.

Jaime Guerrero (jg3755)
Changed in heat:
assignee: nobody → Jaime Guerrero (jg3755)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/320162

Changed in heat:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/320162
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4c4301fbe0b3396ec2bd5492d5d5b600e51c8d4c
Submitter: Jenkins
Branch: master

commit 4c4301fbe0b3396ec2bd5492d5d5b600e51c8d4c
Author: Jaime Guerrero <email address hidden>
Date: Mon May 23 21:31:43 2016 +0000

    Add wait_secs to test_stack_update_from_failed_patch

    Add a 1 second pause to test_stack_update_from_failed_patch. If a stack is
    updated more than once per second, there is a possibility for busy waiting if
    _verify_status neither returns nor raises an exception.

    Please see
    https://github.com/openstack/heat/blob/master/heat_integrationtests/common/test.py#L277
    for _verify_status.

    Closes bug: 1583630

    Change-Id: Ifa9a5e9d69b47fe07fbc9ffcbe8f03479a24bb6c

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b1

This issue was fixed in the openstack/heat 7.0.0.0b1 development milestone.

Revision history for this message
Zane Bitter (zaneb) wrote :

Saw this again, this time in heat_integrationtests.functional.test_update_restricted.UpdateRestrictedStackTest.test_replace

http://logs.openstack.org/98/323998/2/gate/gate-heat-dsvm-functional-orig-mysql-lbaasv2/e43b113/console.html

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/329433

Revision history for this message
Rabi Mishra (rabi) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/329748

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/329433
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c85cae58bf2acfd32b6b745d7e218ba8fe05e604
Submitter: Jenkins
Branch: master

commit c85cae58bf2acfd32b6b745d7e218ba8fe05e604
Author: Zane Bitter <email address hidden>
Date: Tue Jun 14 15:09:19 2016 +0200

    Fix test_update_restricted timeouts due to timestamp resolution

    Updating a stack where the only resource needs replacement but is
    restricted from being replaced is *very* fast. Since the test framework
    relies on seeing a change in the updated_time timestamp, which has only 1s
    resolution, it can easily be missed causing the test to timeout. This adds
    a 1s sleep between updates that have the same expected status to ensure
    that the timestamp changes.

    Change-Id: Ief845a21108e925325b1b3e8f090038ccdf73b8e
    Closes-Bug: #1554375
    Related-Bug: #1583630

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/329748
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9101cd4a5427ed1525e7d9a37983ff945299b5e4
Submitter: Jenkins
Branch: master

commit 9101cd4a5427ed1525e7d9a37983ff945299b5e4
Author: Rabi Mishra <email address hidden>
Date: Wed Jun 15 08:41:29 2016 +0530

    Add wait_sec for test_stack_update_from_failed

    Similar to some other failures where the updated_time does not
    change, _verify_status ignores it and the test times out.

    Change-Id: Ifdf8c1e5e129fdd6c3592ffaca307e43c1317f70
    Related-Bug: #1583630

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/498657

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/498657
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8d0a96f7afeb0169eb551f1a9d139ca70e512341
Submitter: Zuul
Branch: master

commit 8d0a96f7afeb0169eb551f1a9d139ca70e512341
Author: rabi <email address hidden>
Date: Mon Aug 28 18:36:27 2017 +0530

    Fix race conditions in test_create_update

    If two updates happen in the same second, stack updated_time
    does not change. This adds 'action_wait_secs' to have a
    different updated_time.

    Change-Id: Ie9d1c7fe83abfc7c9431e2d341392296cfd1f8df
    Closes-Bug: #1745276
    Related-Bug: #1583630

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.