bulk delete of ports cost iptables-firewall too much time

I suspect this does affect Liberty as well, so if we fix it, lets plan for a backport.

kyle, thanks for your confirm, it affects Liberty as well

I'm afraid using a kind of async mechanism for cleaning conntrack will create possibility for race conditions for conntrack zones.
What time does it take to cleanup conntrack for a 100 VMs?

What is the issue that this bug is reporting? Taking time to cleanup after a deleted VM is only an issue if it's blocking the wiring of new or updated ports. Is that what is happening? If so, I would rather just have us de-prioritizing the cleanup so this doesn't impact anything important.

@Eugene Nikanorov, it depends on the the number of VMs running on a compute node, for example, if a default security group has 100 VMs, and one compute node has 15 VMs, when we delete the other 85 VMs, this compute node need at least 5 minutes to clean the ip_conntrack

@kevin, when the ovs-agent was cleaning the ip_conntrack, it block the wiring of new or updated ports, because ovs-agent just run with one process.
I have test with eventlet.GreenPool(size=10), as above case, the time can be reduced to 2 minutes, but I think some user can't accept it, so I want to add a config option, the default value is not cleaning the ip_conntrack, what do you think?

Fix proposed to branch: master
Review: https://review.openstack.org/243484

Can confirm that conntrack cleanup seems to be a blocking operation. Can also attest to poor performance. I witnessed my system spend well over 10 minutes spinning on conntrack cleanup of 50 instances and a security group that allows IP traffic from 0.0.0.0, which seems like fairly basic scale. I can only imagine what happens with a more complex security group. Haven't spent much time in the code, and it did appear that there were attempts to clean up conntrack state that had already been cleaned up. I've lost the log file, but I can reset and attempt to reproduce.

@kevin, Ryan Tidwell, do you have good idea to solve this problem?

@Ryan Tidwell, I have test case that a security group that allows IP traffic from 0.0.0.0, but not found problems, can you provide some details?

Fix proposed to branch: master
Review: https://review.openstack.org/243994

I'm rising importance to High, as this seems to be really widespread (as per comment #8) and probably turning some clouds into no functional at scale.

Change abandoned by shihanzhang (<email address hidden>) on branch: master
Review: https://review.openstack.org/243484
Reason: please review this patch https://review.openstack.org/#/c/243994/

Adding the linuxbridge tag, and pinging the relevant people, since I believe this also affects linux bridge implementation due to them sharing the iptables firewall driver.

@shihanzang, another question, did we already have conntrack manipulations on kilo? or that was only introduced in liberty?.

Miguel, I believe we do, so it would affect Kilo too.
However I wonder, how such a cleanup could take so much time.

this feature implement in Liberty, so it does not affect kilo

Let's see if there's any chance to nuke this. Assessing...

After looking closer, this seems like a blocker to me.

I am not sure how it's a release blocker if it was always the case since the feature introduction.

OVS agent is blocked on conntrack call because it's not using thread pool. Optimization for security group updates will get us that far where we could optimize for some specific use cases; but it won't tackle the global issue of the agent being completely blocked by external process calls.

Fix proposed to branch: master
Review: https://review.openstack.org/293239

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/293286

Reviewed: https://review.openstack.org/293239
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7aeb8dd4b1d122e17eef8687192cd122b79fd6e
Submitter: Jenkins
Branch: master

commit d7aeb8dd4b1d122e17eef8687192cd122b79fd6e
Author: Kevin Benton <email address hidden>
Date: Mon Mar 14 13:19:54 2016 -0700

    De-dup conntrack deletions before running them

    During a lot of port deletions, the OVS agent will
    build up a lot of remote security group member updates
    for a single device. Once the call to delete all
    of the removed remote IP conntrack state gets issued,
    there will be many duplicated entries for the same
    device in the devices_with_updated_sg_members dicionary
    of lists.

    This results in many duplicated calls to remove conntrack
    entries that are just a waste of time. The longer it takes
    to remove conntrack entries, the more of these duplicates
    build up for other pending changes, to the point where there
    can be hundreds of duplicate calls for a single device.

    This just adjusts the conntrack manager clearing logic to
    make sure it de-duplicates all of its delete commands before
    it issues them.

    In a local test on a single host I have 11 threads create
    11 ports each, plug them into OVS, and then delete them.
    Here are the number of conntrack delete calls issued:

    Before this patch - ~232000
    With this patch - ~5200

    While the remaining number still seems high, the agent is now
    fast enough to keep up with all of the deletes.

    Closes-Bug: #1513765
    Change-Id: Icba88ab47ee17bf5d6ccdfc0f78bec911987ca90

This issue was fixed in the openstack/neutron 8.0.0.0rc1 release candidate.

Reviewed: https://review.openstack.org/293286
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e4f590dc45424ddaaec0a23841156ea9d1a8bbb9
Submitter: Jenkins
Branch: stable/liberty

commit e4f590dc45424ddaaec0a23841156ea9d1a8bbb9
Author: Kevin Benton <email address hidden>
Date: Mon Mar 14 13:19:54 2016 -0700

    De-dup conntrack deletions before running them

    During a lot of port deletions, the OVS agent will
    build up a lot of remote security group member updates
    for a single device. Once the call to delete all
    of the removed remote IP conntrack state gets issued,
    there will be many duplicated entries for the same
    device in the devices_with_updated_sg_members dicionary
    of lists.

    This results in many duplicated calls to remove conntrack
    entries that are just a waste of time. The longer it takes
    to remove conntrack entries, the more of these duplicates
    build up for other pending changes, to the point where there
    can be hundreds of duplicate calls for a single device.

    This just adjusts the conntrack manager clearing logic to
    make sure it de-duplicates all of its delete commands before
    it issues them.

    In a local test on a single host I have 11 threads create
    11 ports each, plug them into OVS, and then delete them.
    Here are the number of conntrack delete calls issued:

    Before this patch - ~232000
    With this patch - ~5200

    While the remaining number still seems high, the agent is now
    fast enough to keep up with all of the deletes.

    Closes-Bug: #1513765
    Depends-On: I4041478ca09bd124827782774b8520908ef07be0
    Change-Id: Icba88ab47ee17bf5d6ccdfc0f78bec911987ca90
    (cherry picked from commit d7aeb8dd4b1d122e17eef8687192cd122b79fd6e)

This issue was fixed in the openstack/neutron 7.0.4 release.

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/243994
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Fix proposed to branch: master
Review: https://review.openstack.org/388490

Change abandoned by shihanzhang (<email address hidden>) on branch: master
Review: https://review.openstack.org/243994

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/388490
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.