Pluggable IPAM rollback fails with reference driver

Bug #1603162 reported by Carl Baldwin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Carl Baldwin

Bug Description

This bug is visible when pluggable IPAM is active. It can be seen with this patch [1]. It does not cause gate failures but it is still something that should be understood. This logstash query [2] seems to find where they occur. It is helpful to look at the DEBUG level logging around the time of the error. For example see this paste [3].

It seems that the session gets broken with an exception that causes a rollback. Then, the IPAM rollback attempts to use the same session for rollback which fails. Should the reference pluggable IPAM driver be using a different session? Or, should it call rollback?

[1] https://review.openstack.org/#/c/181023
[2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22IP%20deallocation%20failed%20on%20external%20system%5C%22
[3] http://paste.openstack.org/show/532891/

Tags: l3-ipam-dhcp
Changed in neutron:
importance: Undecided → High
tags: added: l3-ipam-dhcp
Revision history for this message
Carl Baldwin (carl-baldwin) wrote :

Can we write a test that tickles this problem? Could it be as simple as passing two fixed_ips to a port, one that is available and allocation would succeed and then a second that conflicts with an IP that is already allocated? The fixed IPs would get sorted here [1] which could make the test fail or pass intermittently.

[1] https://github.com/openstack/neutron/blob/b4a3745d1837d6f3a1b16929cd9ea83321e26492/neutron/db/ipam_pluggable_backend.py#L105

Changed in neutron:
assignee: nobody → Pavel Bondar (pasha117)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/348956

Changed in neutron:
assignee: Pavel Bondar (pasha117) → Carl Baldwin (carl-baldwin)
status: New → In Progress
summary: - IP deallocation failed on external system with pluggable IPAM
+ Pluggable IPAM rollback fails with reference driver
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/348956
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=67984850228f6f26a72504b9f464a5fbcaac59e6
Submitter: Jenkins
Branch: master

commit 67984850228f6f26a72504b9f464a5fbcaac59e6
Author: Carl Baldwin <email address hidden>
Date: Fri Jul 29 10:10:48 2016 -0600

    Avoid IPAM driver reusing a session that has been rolled back

    With the in-tree pluggable IPAM driver, IPAM rollback tries to use the
    DB session after it has been rolled back due to an exception. This
    driver doesn't need roll back, so fix this by adding a method to the
    driver signalling that rollback shouldn't be attempted.

    Change-Id: Ic254789e58a8a51cd1aa943cb71de12410f4c0a7
    Closes-Bug: #1603162
    Related-Bug: #1516156

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.0.0.0b3

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.