Race condition with stack_lock_create

Bug #1523910 reported by Thomas Herve
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Medium
Unassigned

Bug Description

I got a IntegrityError while testing reset_stack_status, cf http://paste.openstack.org/show/481134/.

I believe the issue is inherent to stack_lock_create, as it properly uses a transaction to query for existing locks, but if 2 transactions start at the same time, both of them will try to create the StackLock, and one of them will fail. When we get an IntegrityError, we should close the transaction, try again, and hopefully the lock will be in the database.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

This should be fixed, but we also need to avoid the thundering herd problem of all the workers on a node attempting to reset the same stacks on startup.

I think a random sleep before the first stack is reset would be enough to avoid this issue.

Changed in heat:
milestone: mitaka-2 → mitaka-3
Changed in heat:
milestone: mitaka-3 → mitaka-rc1
Changed in heat:
milestone: mitaka-rc1 → newton-1
Thomas Herve (therve)
Changed in heat:
assignee: Thomas Herve (therve) → nobody
milestone: newton-1 → next
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.