python-swiftclient

Container delete race condition causes timeout/retry

Bug #1032879 reported by Darrell Bishop
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-swiftclient
Fix Released
Undecided
Darrell Bishop

Bug Description

In _delete_container(), we wait for object_queue.empty() before continuing on to the conn.delete_container(container) call. But Queue.empty() will return True after the last object_queue item has been dequeued but before the object delete has actually been carried out. This is the cause of 409 results which the subsequent code catches and reties 10 times with a 1-second sleep between each retry.

QueueFunctionThread is already correctly calling self.queue.task_done() in a finally: block, so _delete_container() should call object_queue.join() instead of object_queue.empty().

In my testing, this does appear to clear up the 409 errors.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-swiftclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/10817

Changed in python-swiftclient:
assignee: nobody → Darrell Bishop (darrellb)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-swiftclient (master)

Reviewed: https://review.openstack.org/10817
Committed: http://github.com/openstack/python-swiftclient/commit/c87458e229281a050ecca11a0b99fa55c07ad29c
Submitter: Jenkins
Branch: master

commit c87458e229281a050ecca11a0b99fa55c07ad29c
Author: Darrell Bishop <email address hidden>
Date: Fri Aug 3 19:27:07 2012 -0700

    Fix container delete throughput and 409 retries.

    Fix race condition in _delete_container() where all elements of
    object_queue have been removed, but the last one (per thread) may not
    have actually been deleted yet when the container deletion thread calls
    conn.delete_container(container). Fixes bug 1032879.

    Improves container deletion throughput by immediately deleting
    containers with no objects instead of waiting for all pending object
    deletes to complete. Fixes bug 1032878.

    Change-Id: I404229a4c608995294e0ada77724ac8afe8d6f3c

Changed in python-swiftclient:
status: In Progress → Fix Committed
John Dickinson (notmyname)
Changed in python-swiftclient:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.