Created subnets on ONE network that duplicated CIDR in case of neutron server active-active

Confirmed on devstack all-in-one setup with one neutron-server and api_workers > 0, which is the default devstack setup.

http://paste.openstack.org/show/483424/

It failed once (As expected) with a duplicate CIDR error, but the second time I ran it I didn't get an error and had the same CIDR twice on the same network.

Hi Assaf Muller. Thank you for your check. I would like to explain to more detail about my case:

- First: I build three controller nodes by Devstack. It has services include: nova, neutron, glance, keystone, horizon. After done, I turn off all of services.

- Second: I configure MariaBD Galera to cluster database, configuration to sync time.

- Third: I configure pacemaker to create VIP (Virtual IP) and setup haproxy to load balancing requests and configure cluster Rabbitmp

- Finally: I edit the files configure of services in Openstack so that they will point to VIP then I change the endpoint of services to VIP in the keystone database.

Next step: I run step by step as I mentioned as above and the I received result as I reported.

Do you have suggest how to fix it.

Ohh Sorry land, Because I'm newbie so I dont understand mean tab "Assigned to". Now I understood

One work-around to consider is to use subnetpools exclusively. They are pretty robust in preventing overlapping subnets.

Normally, we rely on database constraints to avoid duplicate entries. However, this is difficult to do with subnet allocation because they can vary in size and overlap without being exactly equal.

For example, you might try to add 192.160.0.0/24 and 192.168.0.192/27. These overlap and we try not to let this happen. But, how do you get the DB to handle this with a constraint? I don't know how to do this all within the DB where I'd ideally like it to be constrained [2]. If only SQL had some sort of unique range constraint where a row could allocate as much of an available range as it wanted as long as no other row's range overlapped [3].

We had to think about this in the subnet allocation code in subnet_alloc.py [1]. Take a look at the _lock_subnetpool method. We iterated a fair amount on this. I'm fairly confident that we ended up with something that would disallow overlapping subnets within a subnetpool.

However, this strategy wasn't applied to the legacy subnet code because we are a lot more loose about how we allow overlap in this IP space. It is really only within the context of a Network or a Router that we check for overlap in the legacy code.

That said, I think the same kind of strategy could be applied to fix this bug. But, it must be applied a bit differently taking in to consideration the different scope where address overlap should be prevented.

[1] https://github.com/openstack/neutron/blob/68276dc9614d47d028e46c35bf62668e253af18b/neutron/ipam/subnet_alloc.py#L44
[2] Actually, there is a way but it could be expensive and ugly. First, you define a minimum size subnet, say /30 or /32. Then you break each subnet in to all the fragments of that size that make up the larger subnet.

For example, 192.168.0.192/27 could break up in to 192.168.0.192/30, 192.168.0.196/30, ..., 192.168.0.220/30 and insert a row in to the DB for each. That table would have a unique constraint for (network_id, subnet_fragment).

The problem is that a small fragment size ends up creating many many DB entries (especially with ipv6) and people complain that a large fragment size is too restrictive.
[3] This isn't the first time I've thought of this before. I've even wanted multi-dimensional range queries before but I don't know of any such feature available in sql.

@Carl, if solving this via solely the DB is difficult (And I understand why it is), it sounds like a good candidate for application level distributed locking. I was there for the Tokyo design session where it was decided to use a library called Tooz [1], which as I understand is an abstraction layer over distributed locking mechanisms. The default that was chosen in that session is Zookeeper. We could lock the method that writes a CIDR to a DB, the lock name would be composed of the method name + the network_id to solve this particular bug.

[1] https://pypi.python.org/pypi/tooz

@Assaf, I don't think we need to go that far. Take a look at [1]. I think something similar could be done. Like I said, we solved this within the context of a subnetpool and the solution could be extended to the context of a Network (this bug) or a router.

My point was that we can't just use a traditional DB constraint for this. A little extra work needs to be done to lock and synchronize between workers around the network.

[1] https://github.com/openstack/neutron/blob/68276dc9614d47d028e46c35bf62668e253af18b/neutron/ipam/subnet_alloc.py#L44

Hi Assaf and Carl Baldwin. Thank you so much for your comments. Please could you review my ideal about fixing this bug.
Link: https://review.openstack.org/#/c/267470/

Fix proposed to branch: master
Review: https://review.openstack.org/314054

Fix proposed to branch: master
Review: https://review.openstack.org/350953

Reviewed: https://review.openstack.org/350953
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5264ab966d3db7b1cb698190872cbe6feaf48464
Submitter: Jenkins
Branch: master

commit 5264ab966d3db7b1cb698190872cbe6feaf48464
Author: Nam Nguyen Hoai <email address hidden>
Date: Fri Aug 5 09:46:43 2016 +0700

    Using revision_number to ensure no overlap in *one* network

    This patch uses revision_number in database. When creating
    a subnet in a network, the revision_number of the network
    will be increased. That will prevent overlapping CIDR
    (overlapping CIDR means some subnets' cidrs are overlapping)
    on *one* network.

    Basically, in case of concurrent requests creating subnets
    on *one* network, only one request successes, other requests
    needs retrying request.

    Change-Id: Id6548535075bed87a4b36e1462db546ab9163f29
    Closes-Bug: #1532695

Change abandoned by Nam Nguyen Hoai (<email address hidden>) on branch: master
Review: https://review.openstack.org/267470

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

Change abandoned by Nam Nguyen Hoai (<email address hidden>) on branch: master
Review: https://review.openstack.org/314054
Reason: This bug can be fixed by this patch set [1]

[1] https://review.openstack.org/#/c/350953/