Nova boot fails with Error Code 500, if quota_port is < -1 in neutron.conf

Bug #1438738 reported by Sudipta Biswas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Low
Sudipta Biswas
neutron
Fix Released
Medium
Sudipta Biswas

Bug Description

I am running a scale environment where I needed to exceed the port quota count to unlimited.

The neutron.conf documentation for the quota_port parameter states the following:

# Number of ports allowed per tenant. A negative value means unlimited.

Looking at this, I had set the value to -2 as:

quota_port = -2

After this, the nova boot started failing with Error code 500.

The error stack is the following:

 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 1016, in create
 TRACE nova.api.openstack server = self._view_builder.create(req, instances[0])
 TRACE nova.api.openstack IndexError: list index out of range
 TRACE nova.api.openstack

The error is seen because in the /nova/network/neutronv2/api.py - validate_networks() method, the quota check is very strict in terms for the unlimited check:

            if quotas.get('port', -1) == -1:
                # Unlimited Port Quota
                return num_instances
            else:
                free_ports = quotas.get('port') - len(ports)
                ports_needed = ports_needed_per_instance * num_instances
                if free_ports >= ports_needed:
                    return num_instances
                else:
                    return free_ports // ports_needed_per_instance

The above code would return free_ports - and that value is negative.

Filing this bug to change the above check to something like:

            if quotas.get('port', -1) <= -1:
                # Unlimited Port Quota
                return num_instances

This will allow nova to react correctly with the documentation in neutron as well.

Tags: network
Changed in nova:
assignee: nobody → Sudipta Biswas (sbiswas7)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/169605

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Low
tags: added: network
Revision history for this message
Sean Dague (sdague) wrote :

I do not believe this is an issue that should be fixed in Nova. -1 has a well understood 'unlimitted' value in openstack, and neutron definitely shouldn't ever return a -2 over the network. Any data scrubbing here should be on the neutron side.

Changed in nova:
status: In Progress → Won't Fix
Changed in neutron:
assignee: nobody → Sudipta Biswas (sbiswas7)
Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

I spoke with Sean and Dan about this bug in openstack-nova, and agreed to fix it before realizing the original submitter assigned the bug to himself for neutron as well.
I am not going to push the patch to gerrit because this would be unfair to the current assignee.

The patch is attached here, so that the assignee can compare it with his own, or perhaps just push it to gerrit if he feels it's a decent one.

Changed in neutron:
importance: Undecided → Medium
milestone: none → kilo-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/170061

Changed in neutron:
status: New → In Progress
Revision history for this message
Sudipta Biswas (sbiswas7) wrote :

I thought I'd fix this in neutron today and assigned this to my name before going off to bed. I didn't quite realize that you were also looking into it. The patch above looks pretty good. I just added a couple of more lines of checks with respect to network/subnet negative quotas. I pushed the patch in and marked Salvatore Orlando as the co-author. Please let me know if it's not the right way - I'd be absolutely OK with abandoning my review :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/170061
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2041ead12d1bf4c2b03fd980fd2a6ce5f653dcfb
Submitter: Jenkins
Branch: master

commit 2041ead12d1bf4c2b03fd980fd2a6ce5f653dcfb
Author: Sudipta Biswas <email address hidden>
Date: Thu Apr 2 15:06:35 2015 +0530

    Treat all negative quota values as -1

    Currently if the quota_port, quota_network, quota_subnet values
    in the neutron.conf are set to a negative value not equal to -1,
    neutron reports the values as is to consumers like Nova.
    Nova treats -1 as the infinite quota indicator and doesn't expect
    neutron to return any other non-negative value.

    The fix allows the flexibility of having any negative number for the
    quota parameters in the neutron.conf file and allows the nova boot
    to succeed subsequently. The fix would report any negative value
    as -1 for port, subnet and network.

    Change-Id: Ib9a7136b0bfd01bdf04a5d0937854590029b1010
    Closes-Bug: 1438738
    Co-Authored-By: Salvatore Orlando <email address hidden>

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-rc1 → 2015.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/169605
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (neutron-pecan)

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.