Quota handing is broken for multiple delete requests will reduce the

Bug #1027187 reported by Phil Day
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Eoghan Glynn

Bug Description

In the new reservation based quota handling code, the logic for deleting an instance creates a negative reservation at the start of the api call and commits it once the request has been cast to the compute manager:

...
            self.compute_rpcapi.terminate_instance(context, instance)

            QUOTAS.commit(context, reservations) ...
...

The code does include checks for instance['host'] beign null and InstanceNotFound to catch race conditions if the instance has been deleted

But in the case where a second delete request were processed in the API server before the first request has been processed in the compute manager (for example consider that the compute manager is down and the user makes multiple requests because the first didn’t seem to finish) these checks won’t fail, which will lead to the quota being released each time the commit is processed in the API server.

The fix would seem to be to pass the negative reservations down into the compute manager so that they only get committed when the delete is processed.

In the compute manager the delete requests for an instance are synchronised on the uuid, so a second request won’t run until the first has finished, hence a check at the start to see if the instance still exists and do the rollback there should work OK.

Eoghan Glynn (eglynn)
Changed in nova:
assignee: nobody → Eoghan Glynn (eglynn)
Eoghan Glynn (eglynn)
Changed in nova:
status: New → In Progress
status: In Progress → Confirmed
Revision history for this message
Eoghan Glynn (eglynn) wrote :

Interestingly, in the special case where the double accounting for the repeated delete causes the in_use count to go negative, then a subsequent attempt to reserve further instance quota will cause the inappropriate in_use value to be healed via the sync logic.

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
Eoghan Glynn (eglynn) wrote :

Fix merged to master:

  https://review.openstack.org/11044

Changed in nova:
status: In Progress → Fix Committed
Eoghan Glynn (eglynn)
Changed in nova:
importance: Undecided → Medium
milestone: none → folsom-3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.