Nova fails to correctly update the quota when deleting lots of VMs, some of which previously failed due to a quota error

Thats expected, you have to delete the instance to remove the quota usage.

That is by design, as its very possible for an instance that is in the ERROR state to be running and user hypervisor resources, IPs, volumes, etc.

You must delete to remove the quota usage.

I don't believe thats a recent change. Did you see different behaviour on a previous release?

Please read the step 3

Where it mentioned delete all vms

On 27-Feb-2017 7:41 PM, "John Garbutt" <email address hidden> wrote:

> Thats expected, you have to delete the instance to remove the quota
> usage.
>
> That is by design, as its very possible for an instance that is in the
> ERROR state to be running and user hypervisor resources, IPs, volumes,
> etc.
>
> You must delete to remove the quota usage.
>
> I don't believe thats a recent change. Did you see different behaviour
> on a previous release?
>
> ** Tags added: quotas
>
> ** Changed in: nova
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1668267
>
> Title:
> Nova ignores to update the quota for VMs in error
>
> Status in OpenStack Compute (nova):
> Incomplete
>
> Bug description:
> When VMs goes to ERROR state after the block device mapping nova updates
> the quota for adding the instance/CPU/RAM to the quota
> but when we delete those VMs nova forgets the release/update the
> CPU/RAM/Instance quota
>
> Steps to reproduce:
> 1. Set the quota limit for volumes to 10 and VMs to 50
> 2. create 20 VMs by using option boot from volume from horizon
> Observation:
> 10 VMs comes to active state and 10 goes to ERROR state (As expected)
>
> 3. delete all VMs
>
> Expectation: CPU/RAM/Instance used quota will come to "0"
> Actual: CPU/RAM/Instance quota is not coming to "0"
> CPU used quota shows 10 (As the VM is created from M1.SMALL flavor)
> Instance quota shows 10
> RAM quota shows 20480MB
>
> which is quite unexpected
>
> Its seems it is a blocker for Ocata release
>
> I am running from the stable/Ocata
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1668267/+subscriptions
>

OK, if you delete them, we should correct the title of the bug.

What state are those deleted instances in now? Are they still in the ERROR state?

nova list
/usr/lib/python2.7/site-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

while deleting the VM

select all 20 VMs from horizon and click on delete from horizon

to create boot from volume from horizon

No VMs present after deletion but quota shows 10 VMs present

Can we retry this with 15 instances instead of 20, just to double check whats going on.

It would be great if we could get logs from the system where the deletes occurred (nova-api and nova-compute logs), to make sure there are no obvious exceptions, etc, that might help give a hint about what is going on here.

Sure working on it

will be able to give you the details in 1hr

Confirmed on my setup as well.
Set instances quota to 10, volumes to 5.
Spawned 7 VMs with volumes attached - 2 of them failed as expected.
Used resources reported 5/10 VMs and 5/5 volumes.
Deleted all instances and the used resources are still the same.

Compute log from host 1

http://paste.openstack.org/show/600711/

API log from host 1

http://paste.openstack.org/show/600712/

Compute log from host 2

http://paste.openstack.org/show/600713/

API log from host 2

http://paste.openstack.org/show/600714/

any update on this?

tested the fix https://review.openstack.org/#/c/437222

still having same issue

For master (pike) this is the fix:

https://review.openstack.org/#/c/443403/

For ocata, the fix will be a backport of this:

https://review.openstack.org/#/c/443395/