delete_subnet update_port appears racey with ipam

Bug #1622616 reported by Kevin Benton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Armando Migliaccio

Bug Description

failure spotted in a patch on a delete_subnet call:

2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource traceback.format_exc())
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "port %s"), port_id)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.update_port(context, port_id, data)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource port)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port=new_port)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port.get('mac_address'))
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_mac)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource changes.remove)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "external system for %s"), addresses)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise()
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ipam_subnet.deallocate(ip['ip_address'])
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ip_address=address)
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55
2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource

http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452

Changed in neutron:
milestone: none → newton-rc1
importance: Undecided → High
tags: added: l3-ipam-dhcp
Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

It is tempting to put try/except handler it for this and just ignore it as Kevin suggested to me in the email where he pointed this out. But, let me look at it for a little bit to wrap my head around what is leading up to this.

Changed in neutron:
assignee: nobody → Carl Baldwin (carl-baldwin)
Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

Okay, it looks like there is a race between a couple of requests trying to delete this allocation. The first one [1] is a port update that ends of changing the IP allocated to the port (10.1.0.19 to 10.1.0.18). I'm not sure why yet. The second one [2] is the delete subnet which is trying to auto-deallocate 10.1.0.19 but it is gone already. I wonder what the DB isolation level is here.

I'm not sure that it is wise to simply catch the IpAddressAllocationNotFound error and ignore it because I suspect that 10.1.0.18 would get left behind if we do that.

I'm still trying to learn a little bit more about the context. I'm not really sure why the first request [1] is being made. Why is it changing the IP address? The port update just specifies the subnet id. You'd think it would leave the existing IP address on that subnet alone in this case. Why would it throw away one and got get another one? Still looking in to it.

[1] http://paste.openstack.org/show/572415/
[2] http://paste.openstack.org/show/572416/

Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

Looking deeper in to why the port update mentioned in comment #2 changed the IP address for the DHCP port. It looks like because it doesn't supply the existing IP address with the port update request, it is not matched up with it and so the old one is de-allocated and then a new different IP address is allocated for the subnet.

I'm looking now at this logic [1] and the method around it. This is where I think the existing IP address gets lost and a new one is allocated with it.

I'm thinking that the right thing to do is when a subnet is supplied on a port update -- and no specific IP address is supplied -- then we should try to match it up with an existing IP allocation with the same subnet.

I'll write a unit test to expose this problem.

[1] https://github.com/openstack/neutron/blob/cb99ce5c4465dbe8b63a5d2444ed6a6099b472a7/neutron/db/ipam_backend_mixin.py#L446-L448

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/369051

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/369051
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f07c07b16fb0858c45f6cef135a8d8c07a16c505
Submitter: Jenkins
Branch: master

commit f07c07b16fb0858c45f6cef135a8d8c07a16c505
Author: Carl Baldwin <email address hidden>
Date: Mon Sep 12 15:31:08 2016 -0600

    Don't allocate IP on port update when existing subnet specified

    If a port update specifies only a subnet_id for a fixed_ip then we
    want to look at existing fixed_ips to see if that subnet_id is already
    there. This avoids allocating a new IP address on the subnet and
    deallocating the old one.

    Without some special care, this breaks the code path for prefix
    delegation. One could argue that PD needs reworking. However, as a
    stop-gap measure, we still run the old code path if the address is an
    EUI-64 address. This allows PD to continue to work as it was
    originally written and it doesn't do any harm because allocating
    EUI-64 addresses is repeatable.

    This commit removes a test case from the DNS integration tests. The
    test was specifically testing that DNS records we updated in the case
    where a subnet id was passed to re-allocate a fixed_ip. Since the use
    case no longer works, the test doesn't make sense.

    Change-Id: I887cd23cf65a1df9bd1d59ab897a1ecd005a588c
    Closes-Bug: #1622616

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Kevin Benton (kevinbenton) wrote :

@Carl,

Still happening on rally runs with lots of concurrent subnet deletions in the same network:

http://logs.openstack.org/17/369417/5/check/gate-rally-dsvm-neutron-rally/b0d5b03/logs/screen-q-svc.txt.gz#_2016-09-14_10_58_28_812

Changed in neutron:
status: Fix Released → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/370251

Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

Looking through the logs a bit. This one does look different. It looks like two delete subnet requests are competing to delete the same port. The second one comes around 10 seconds after the first one. It looks like the first one hits a number of retry exceptions.

    req-ac41a239-eb65-42a9-bade-497e78844501
    req-f261ade2-dd11-44c5-abff-67b5546ab587

They appear to be deleting different subnets

    ef901e57-4b3b-48c1-8d59-0f8fd7cdca50
    2c7bf6a9-1811-4111-b692-fd888266823d

Looks like on about the 4th retry of the first request, it decides that it wants to deallocate port 6d1f3f30-4370-4f7f-a886-7b3efc0a011e (Before that, it didn't want to deallocate any). The other request wants to deallocate the same port id (but for a different IP address). It looks like the first request got in there and deleted the port before the second request could get around to it and boom.

Still needs further investigation.

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

This is getting bad. I see a lot of traces associated with delete_subnet API calls:

[1] http://logs.openstack.org/09/368709/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/79edca3/

Changed in neutron:
importance: High → Medium
importance: Medium → Critical
tags: added: newton-rc-potential
Changed in neutron:
milestone: newton-rc1 → ocata-1
Changed in neutron:
milestone: ocata-1 → newton-rc2
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :
Revision history for this message
Kevin Benton (kevinbenton) wrote :

that last occurrence is interesting because there are no concurrent API operations for the subnet.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/370251
Reason: i think the issue has been resolved enough that this isn't necessary

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/372236

Changed in neutron:
assignee: Carl Baldwin (carl-baldwin) → Kevin Benton (kevinbenton)
status: New → In Progress
Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

This new occurrence looks the same as the first one reported on this bug. The DHCP agent is calling update port (req-06c10180-b446-4905-b3d5-ae9e27ca8766) with the subnet_id and the IP gets de-allocated. I see 2003::3 getting deleted by that update port (18:54:33.191) from the DHCP agent that came in before the delete subnet request (req-d0f74aa2-a45a-4fb3-9757-a9f0b06329d9) identified which ports to auto-delete (18:54:33.225). (If I'm reading this log right).

I'm going to push a new version of my original fix that will be more robust.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/372794

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

12 occurrences during the past 7 with this query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20delete_subnet%5C%22%20AND%20build_status%3A%5C%22FAILURE%5C%22%20AND%20build_queue%3A%5C%22gate%5C%22%20AND%20build_name%3A%5C%22gate-tempest-dsvm-neutron-full-ubuntu-xenial%5C%22

Basically any time we get a test failure during delete subnet on full tempest job on the gate queue. It does not include DVR, which is failing more.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/372236
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=673abd56c906836d1e9aa14536139537100900c0
Submitter: Jenkins
Branch: master

commit 673abd56c906836d1e9aa14536139537100900c0
Author: Kevin Benton <email address hidden>
Date: Sun Sep 18 22:50:16 2016 -0700

    Don't immediately restart in DHCP agent on port change

    Now that the agent will receive port update events for
    all port changes[1], we need to avoid immediately restarting
    when the subnets on the agent's port changes. Otherwise
    the restart may request ports on a subnet which is in the
    process of being deleted. While the server is equipped to
    handle this, it makes subnet deletion much more contentious
    than it needs to be.

    This alters the logic to schedule a resync for later if the
    agent's port has had its subnets changed rather than restarting
    right away. Then by the time the agent eventually syncs the
    server should have finished deleting the subnet. Even if it hasn't,
    it spaces out the request from the agent for the network far enough
    that the operation will be much less frequent to avoid racing
    with the server.

    1. I607635601caff0322fd0c80c9023f5c4f663ca25

    Partial-Bug: #1622616
    Change-Id: I98761a7e3f4bce8d5485c885f03c6bfdde246802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/373566
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9019ff2f744ae91839d05a188358ef92343add43
Submitter: Jenkins
Branch: master

commit 9019ff2f744ae91839d05a188358ef92343add43
Author: Kevin Benton <email address hidden>
Date: Mon Sep 19 18:44:02 2016 -0700

    Make DHCP agent use 'revision_number'

    Change I445974b0e0dabb762807c6f318b1b44f51b3fe15 updated the
    'revision' field to 'revision_number' but it missed the DHCP
    agent and subsequently broke it's ability to detect stale updates.

    This fixes the name in the agent.

    This is marked as a partial for 1622616 because one of the reasons
    the agent was frequently updating the DHCP port was in reaction
    to stale port update messages for its own port.

    Partial-Bug: #1622616
    Closes-Bug: #1625867
    Change-Id: Id41000127e1084f7ff243f8dc9c399999fbdaab4

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/373885

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/373887

Revision history for this message
Kevin Benton (kevinbenton) wrote :

With the latest merged fixes, we got a clean run on rally with 400 iterations of the create_and_update_subnets task in rally.[1] I think we can downgrade the severity from critical.

1. http://logs.openstack.org/23/372223/1/check/gate-rally-dsvm-neutron-neutron/3c5405f/rally-plot/results.html.gz#/NeutronNetworks.create_and_update_subnets/overview

Changed in neutron:
assignee: Kevin Benton (kevinbenton) → nobody
Revision history for this message
Kevin Benton (kevinbenton) wrote :

We still need an underlying fix for two concurrent port updates that both try to set IPs. Our port update is a PUT action of the port body so we shouldn't throw errors to the user do to an internal race condition of IPAM.

For example, I shouldn't get an error if the port has IPs A and B and two concurrent port updates set the fixed_ips to just A.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/373885
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=72f964a20ddac5f0d192eb7324c663fff877fb6e
Submitter: Jenkins
Branch: stable/newton

commit 72f964a20ddac5f0d192eb7324c663fff877fb6e
Author: Kevin Benton <email address hidden>
Date: Sun Sep 18 22:50:16 2016 -0700

    Don't immediately restart in DHCP agent on port change

    Now that the agent will receive port update events for
    all port changes[1], we need to avoid immediately restarting
    when the subnets on the agent's port changes. Otherwise
    the restart may request ports on a subnet which is in the
    process of being deleted. While the server is equipped to
    handle this, it makes subnet deletion much more contentious
    than it needs to be.

    This alters the logic to schedule a resync for later if the
    agent's port has had its subnets changed rather than restarting
    right away. Then by the time the agent eventually syncs the
    server should have finished deleting the subnet. Even if it hasn't,
    it spaces out the request from the agent for the network far enough
    that the operation will be much less frequent to avoid racing
    with the server.

    1. I607635601caff0322fd0c80c9023f5c4f663ca25

    Partial-Bug: #1622616
    Change-Id: I98761a7e3f4bce8d5485c885f03c6bfdde246802
    (cherry picked from commit 673abd56c906836d1e9aa14536139537100900c0)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/373887
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c11b08a66887490a4643690bd0a4360a3b0a9215
Submitter: Jenkins
Branch: stable/newton

commit c11b08a66887490a4643690bd0a4360a3b0a9215
Author: Kevin Benton <email address hidden>
Date: Mon Sep 19 18:44:02 2016 -0700

    Make DHCP agent use 'revision_number'

    Change I445974b0e0dabb762807c6f318b1b44f51b3fe15 updated the
    'revision' field to 'revision_number' but it missed the DHCP
    agent and subsequently broke it's ability to detect stale updates.

    This fixes the name in the agent.

    This is marked as a partial for 1622616 because one of the reasons
    the agent was frequently updating the DHCP port was in reaction
    to stale port update messages for its own port.

    Partial-Bug: #1622616
    Closes-Bug: #1625867
    Change-Id: Id41000127e1084f7ff243f8dc9c399999fbdaab4
    (cherry picked from commit 9019ff2f744ae91839d05a188358ef92343add43)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/372794
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=14fee9cfcc816128606fc11d33800af90a0b1859
Submitter: Jenkins
Branch: master

commit 14fee9cfcc816128606fc11d33800af90a0b1859
Author: Carl Baldwin <email address hidden>
Date: Mon Sep 12 15:31:08 2016 -0600

    Try to reuse existing IPs when a port update specifies subnet

    If a port update specifies only a subnet_id for a fixed_ip then we
    want to look at existing fixed_ips to see if that subnet_id is already
    there. This avoids allocating a new IP address on the subnet and
    deallocating the old one.

    Without some special care, this breaks the code path for prefix
    delegation. One could argue that PD needs reworking. However, as a
    stop-gap measure, we still run the old code path if the address is an
    EUI-64 address. This allows PD to continue to work as it was
    originally written and it doesn't do any harm because allocating
    EUI-64 addresses is repeatable.

    This commit removes a test case from the DNS integration tests. The
    test was specifically testing that DNS records we updated in the case
    where a subnet id was passed to re-allocate a fixed_ip. Since the use
    case no longer works, the test doesn't make sense.

    This commit also preserves the ability to add an additional IP from a
    subnet for which the port already has IPs.

    Change-Id: Iba5d54efa7f99ed82275ffc8e5be975b373c29d3
    Related-Bug: #1622616
    Closes-Bug: #1625334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/374689

Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Lowering severity as per Kevin's comments.

Changed in neutron:
importance: Critical → High
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/374689
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c201439a94d9249d1c3aefbf404304ffa1690be0
Submitter: Jenkins
Branch: stable/newton

commit c201439a94d9249d1c3aefbf404304ffa1690be0
Author: Carl Baldwin <email address hidden>
Date: Mon Sep 12 15:31:08 2016 -0600

    Try to reuse existing IPs when a port update specifies subnet

    If a port update specifies only a subnet_id for a fixed_ip then we
    want to look at existing fixed_ips to see if that subnet_id is already
    there. This avoids allocating a new IP address on the subnet and
    deallocating the old one.

    Without some special care, this breaks the code path for prefix
    delegation. One could argue that PD needs reworking. However, as a
    stop-gap measure, we still run the old code path if the address is an
    EUI-64 address. This allows PD to continue to work as it was
    originally written and it doesn't do any harm because allocating
    EUI-64 addresses is repeatable.

    This commit removes a test case from the DNS integration tests. The
    test was specifically testing that DNS records we updated in the case
    where a subnet id was passed to re-allocate a fixed_ip. Since the use
    case no longer works, the test doesn't make sense.

    This commit also preserves the ability to add an additional IP from a
    subnet for which the port already has IPs.

    Change-Id: Iba5d54efa7f99ed82275ffc8e5be975b373c29d3
    Related-Bug: #1622616
    Closes-Bug: #1625334
    (cherry picked from commit 14fee9cfcc816128606fc11d33800af90a0b1859)

Changed in neutron:
assignee: nobody → Armando Migliaccio (armando-migliaccio)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/373536
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=873b5ac837220c11e204c6610782f8b86c90bf03
Submitter: Jenkins
Branch: master

commit 873b5ac837220c11e204c6610782f8b86c90bf03
Author: Armando Migliaccio <email address hidden>
Date: Tue Sep 20 14:23:40 2016 -0700

    Retry port update on IpAddressAllocationNotFound

    If a port update and a subnet delete interleave, there is a
    chance that the IPAM update operation raises this exception.
    Rather than throwing that up to the user under some sort of
    conflict, bubble up a retry instead; that should bring things
    back to sanity.

    Closes-bug: #1622616

    Change-Id: Ia8cac09349d4cb722737bdf0bec6c54b9e77f31d

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375867

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0rc1

This issue was fixed in the openstack/neutron 9.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/375867
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=37dc141da1a5bbede7b7d131454c54b515074e6b
Submitter: Jenkins
Branch: stable/newton

commit 37dc141da1a5bbede7b7d131454c54b515074e6b
Author: Armando Migliaccio <email address hidden>
Date: Tue Sep 20 14:23:40 2016 -0700

    Retry port update on IpAddressAllocationNotFound

    If a port update and a subnet delete interleave, there is a
    chance that the IPAM update operation raises this exception.
    Rather than throwing that up to the user under some sort of
    conflict, bubble up a retry instead; that should bring things
    back to sanity.

    Closes-bug: #1622616

    Change-Id: Ia8cac09349d4cb722737bdf0bec6c54b9e77f31d
    (cherry picked from commit 873b5ac837220c11e204c6610782f8b86c90bf03)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0rc2

This issue was fixed in the openstack/neutron 9.0.0.0rc2 release candidate.

tags: removed: newton-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0rc1

This issue was fixed in the openstack/neutron 9.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0b1

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/834536

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/834650

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/834651

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/834652

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/834653

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/834656

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/834657

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834536
Committed: https://opendev.org/openstack/neutron/commit/83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd
Submitter: "Zuul (22348)"
Branch: master

commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834650
Committed: https://opendev.org/openstack/neutron/commit/2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834651
Committed: https://opendev.org/openstack/neutron/commit/d2b1e3edaafdf5a2fe4b64457a4caa57a194f858
Submitter: "Zuul (22348)"
Branch: stable/xena

commit d2b1e3edaafdf5a2fe4b64457a4caa57a194f858
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)
    (cherry picked from commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834652
Committed: https://opendev.org/openstack/neutron/commit/b053dfe3844aaf024668847aab3498c3af559dc0
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b053dfe3844aaf024668847aab3498c3af559dc0
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)
    (cherry picked from commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834653
Committed: https://opendev.org/openstack/neutron/commit/884418183a6e518d755f4e4dd6e05db7e7eef0ab
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 884418183a6e518d755f4e4dd6e05db7e7eef0ab
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)
    (cherry picked from commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94)

tags: added: in-stable-victoria
tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834656
Committed: https://opendev.org/openstack/neutron/commit/4f03e76aaafbbc871a6f5c3fc4ecca4298635d6f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 4f03e76aaafbbc871a6f5c3fc4ecca4298635d6f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)
    (cherry picked from commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834657
Committed: https://opendev.org/openstack/neutron/commit/1254f2371864e10dc66f5aa27031088209f74e9e
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1254f2371864e10dc66f5aa27031088209f74e9e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Mar 16 16:32:31 2022 +0000

    Remove exception ``IpAddressAllocationNotFound``

    This patch removes the ``IpAddressAllocationNotFound`` exception. This
    exception was raised when a IPAM register was called to be deleted
    but not found.

    As reported in the LP bug, this IPAM register deletion can be called
    several times if a port fails during the creation. The IPAM register
    deletion calls the DB deletion but doesn't raise any exception if the
    register does not exist. The code ensures the IPAM register is
    deleted and there is no need to fail if it is not present anymore.

    This patch also removes the exception catch and try in "update_port",
    that was added in [0] as a fix for [1]. That was added because the
    subnet deletion code involved a port update call [2] during the
    IP allocation deletion, if any port was still present in the subnet.
    Since [3], this code is not needed because the subnet deletion does
    not call a port update anymore.

    [0]https://review.opendev.org/c/openstack/neutron/+/373536
    [1]https://bugs.launchpad.net/neutron/+bug/1622616
    [2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
    [3]https://review.opendev.org/c/openstack/neutron/+/713045

    Closes-Bug: #1965807
    Related-Bug: #1954763
    Related-Bug: #1622616

    Conflicts:
        neutron/ipam/exceptions.py

    Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
    (cherry picked from commit 83b6ce9e9ec2f0261fed630e18cbe9cfa363a0dd)
    (cherry picked from commit 2a6a4ab5dacc2e92c24d68774bf286c9e6ae2d94)

tags: added: in-stable-train
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.