OVS polling loop created by ovsdbapp and os-vif starving n-cpu threads

This is caused by the ovs polling loop in ovsdbapp as a result of the connection crated in os-vif

Another possible hit in https://bugs.launchpad.net/nova/+bug/1863889/comments/3 ?

There are several things that I think need to happen:

1) This patch needs to be merged in python-ovs that stops using a non-monkeypatched select.poll() when checking for ovsdb connection completion - http://patchwork.ozlabs<email address hidden>/

2) os-vif should limit the tables that it registers here: https://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/ovsdb/impl_idl.py#L26 to only the tables that it uses. On connection it currently pulls in the whole database at connection, which takes a long time

3) In the next line, it could pass a probe_interval argument to idl.Idl which will increase the amount of time from the default of 5s for sending echo probes to the server. If downloading the entire DB from a busy ovsdb-server with lots of connections takes longer than 5s, it'll currently disconnect and try again w/o ever successfully connecting

4) Something like https://review.opendev.org/c/openstack/neutron/+/794892 can be done, overriding the python-ovs streams to use TCP keepliaves instead of ovsdb echo requests, making reconnections less likely on a loaded single-threaded ovsdb-server since the kernel will take care of them. Note that that patch probably should have configured TCP_KEEPIDLE, TCP_KEEPCNT, and TCP_KEEPINTVL instead of relying on whatever is configured with sysctl as the defaults are very long.

setting to invalid for nova as the error is in the ovs python bindings.
marked as triaged for os-vif to track the enhancements proposed in comment 3 above.

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/796813

https://review.opendev.org/c/openstack/devstack/+/796826

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-vif/+/805223

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-vif/+/805625

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/805223
Committed: https://opendev.org/openstack/os-vif/commit/09c0629bb728ad342a41d844143d8e7437c925c4
Submitter: "Zuul (22348)"
Branch: master

commit 09c0629bb728ad342a41d844143d8e7437c925c4
Author: Sean Mooney <email address hidden>
Date: Thu Aug 19 14:32:42 2021 +0100

    Use TCP keepalives for ovsdb connections

    Ultimately, this is something that should be fixed in python-ovs,
    but setting the SO_KEEPALIVE socket option benefits the client by
    removing the need to send 'echo' requests, which can time out on
    an overloaded ovsdb-server, which causes a disconnection which then#
    adds even more load on the ovsdb-server as it has to send the entire
    db contents over the wire after the connection is restored.

    This patch ports the optimisation form neutron to reduce the likelyhood
    of a reconnection which can cause the nova compute agent to hang
    temporarily while the connection is reestablished.

    Change-Id: I984ec62730276f8ee60d71a02a98fbfc4c37f7d8
    Related-Bug: #1930926
    Partial-Bug: #1929446

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/805625
Committed: https://opendev.org/openstack/os-vif/commit/e4dc8b5664ccee8bde9e90fc9e618d6b705a0b68
Submitter: "Zuul (22348)"
Branch: master

commit e4dc8b5664ccee8bde9e90fc9e618d6b705a0b68
Author: Sean Mooney <email address hidden>
Date: Mon Aug 23 13:11:16 2021 +0100

    only register tables used by os-vif

    This change limits the tables registered in the native driver
    to the set actully used by os-vif. This will shorten the inital
    startup time and reconnection time if the ovs db connection is dropped.
    as a result this will help mitigate bug #1929446 where on reconnection
    the nova compute agent can stall until reconnection is completed.

    Change-Id: I635dff2b4fcff905ca8f431eb7e928265200f92a
    Partial-Bug: #1929446

Change abandoned by "Terry Wilson <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/795789
Reason: This should be fixed in upstream python-ovs by adding a cooperative_yield() method that can be overriden. ovsdbapp may go ahead and also add a default cooperative_yield() that is just time.sleep(0).

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/818446

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/818446
Committed: https://opendev.org/openstack/ovsdbapp/commit/a2d3ef2a6491eb63b5ee961fc930070207a79d84
Submitter: "Zuul (22348)"
Branch: master

commit a2d3ef2a6491eb63b5ee961fc930070207a79d84
Author: Terry Wilson <email address hidden>
Date: Fri Nov 12 10:58:22 2021 -0600

    Add cooperative_yield() to OvsdbIdl

    On python-ovs 2.16.0+, overriding cooperative_yield() will allow
    potentially long-running CPU-intensive methods to cooperatively
    yield to greenthreads. This patch + ovs 2.16.0+ will resolve
    the related bug. I'll see if I can get cooperative_yield
    backported in OVS as well.

    Related-Bug: #1929446
    Change-Id: Ibd3c7427cbcab81253e0ed700174be09908bdef7

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/841738

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/841739

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/841740

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/841741

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/os-vif/+/841771

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/os-vif/+/841772

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/os-vif/+/841773

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/os-vif/+/841774

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/os-vif/+/841775

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/os-vif/+/841776

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/os-vif/+/841777

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/os-vif/+/841778

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/841779

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/841780

this is basiclaly as fixed as it is going to be in os-vif and ovsdbapp

the actual fix was in ovs
the mitigation for ti are also merged in os-vif and ovsdbapp master (since xena)

i have proposed backports to os-vif down to train
and we can review them as normal.

by the way the os-vif patches jsut reduce the time it takes to reconnect and makes it happen less often but that is only a mitigation for the issue not a fix.

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/841738
Committed: https://opendev.org/openstack/ovsdbapp/commit/62ee94741206de8340eed16e89e4ddb55a8723d3
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 62ee94741206de8340eed16e89e4ddb55a8723d3
Author: Terry Wilson <email address hidden>
Date: Fri Nov 12 10:58:22 2021 -0600

    Add cooperative_yield() to OvsdbIdl

    On python-ovs 2.16.0+, overriding cooperative_yield() will allow
    potentially long-running CPU-intensive methods to cooperatively
    yield to greenthreads. This patch + ovs 2.16.0+ will resolve
    the related bug. I'll see if I can get cooperative_yield
    backported in OVS as well.

    Related-Bug: #1929446
    Change-Id: Ibd3c7427cbcab81253e0ed700174be09908bdef7
    (cherry picked from commit a2d3ef2a6491eb63b5ee961fc930070207a79d84)

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/841740
Committed: https://opendev.org/openstack/ovsdbapp/commit/84c07ca8a3bd827406edb3275a829f5feba5a54e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 84c07ca8a3bd827406edb3275a829f5feba5a54e
Author: Terry Wilson <email address hidden>
Date: Fri Nov 12 10:58:22 2021 -0600

    Add cooperative_yield() to OvsdbIdl

    On python-ovs 2.16.0+, overriding cooperative_yield() will allow
    potentially long-running CPU-intensive methods to cooperatively
    yield to greenthreads. This patch + ovs 2.16.0+ will resolve
    the related bug. I'll see if I can get cooperative_yield
    backported in OVS as well.

    Related-Bug: #1929446
    Change-Id: Ibd3c7427cbcab81253e0ed700174be09908bdef7
    (cherry picked from commit a2d3ef2a6491eb63b5ee961fc930070207a79d84)

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/841741
Committed: https://opendev.org/openstack/ovsdbapp/commit/4d9ea84cbafa9cb15d8d91aae29672e98473c3de
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4d9ea84cbafa9cb15d8d91aae29672e98473c3de
Author: Terry Wilson <email address hidden>
Date: Fri Nov 12 10:58:22 2021 -0600

    Add cooperative_yield() to OvsdbIdl

    On python-ovs 2.16.0+, overriding cooperative_yield() will allow
    potentially long-running CPU-intensive methods to cooperatively
    yield to greenthreads. This patch + ovs 2.16.0+ will resolve
    the related bug. I'll see if I can get cooperative_yield
    backported in OVS as well.

    Related-Bug: #1929446
    Change-Id: Ibd3c7427cbcab81253e0ed700174be09908bdef7
    (cherry picked from commit a2d3ef2a6491eb63b5ee961fc930070207a79d84)

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/841739
Committed: https://opendev.org/openstack/ovsdbapp/commit/e1a0d7c85783e87be6ca440f223d37ba2e2a888f
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e1a0d7c85783e87be6ca440f223d37ba2e2a888f
Author: Terry Wilson <email address hidden>
Date: Fri Nov 12 10:58:22 2021 -0600

    Add cooperative_yield() to OvsdbIdl

    On python-ovs 2.16.0+, overriding cooperative_yield() will allow
    potentially long-running CPU-intensive methods to cooperatively
    yield to greenthreads. This patch + ovs 2.16.0+ will resolve
    the related bug. I'll see if I can get cooperative_yield
    backported in OVS as well.

    Related-Bug: #1929446
    Change-Id: Ibd3c7427cbcab81253e0ed700174be09908bdef7
    (cherry picked from commit a2d3ef2a6491eb63b5ee961fc930070207a79d84)

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/841771
Committed: https://opendev.org/openstack/os-vif/commit/26f073f0969cfd9719375a4edfc16ce4ec139c4f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 26f073f0969cfd9719375a4edfc16ce4ec139c4f
Author: Sean Mooney <email address hidden>
Date: Thu Aug 19 14:32:42 2021 +0100

    Use TCP keepalives for ovsdb connections

    Ultimately, this is something that should be fixed in python-ovs,
    but setting the SO_KEEPALIVE socket option benefits the client by
    removing the need to send 'echo' requests, which can time out on
    an overloaded ovsdb-server, which causes a disconnection which then#
    adds even more load on the ovsdb-server as it has to send the entire
    db contents over the wire after the connection is restored.

    This patch ports the optimisation form neutron to reduce the likelyhood
    of a reconnection which can cause the nova compute agent to hang
    temporarily while the connection is reestablished.

    Change-Id: I984ec62730276f8ee60d71a02a98fbfc4c37f7d8
    Related-Bug: #1930926
    Partial-Bug: #1929446
    (cherry picked from commit 09c0629bb728ad342a41d844143d8e7437c925c4)

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/841772
Committed: https://opendev.org/openstack/os-vif/commit/e6e791bd0af560ac501e383814fa7a5bce004536
Submitter: "Zuul (22348)"
Branch: stable/xena

commit e6e791bd0af560ac501e383814fa7a5bce004536
Author: Sean Mooney <email address hidden>
Date: Mon Aug 23 13:11:16 2021 +0100

    only register tables used by os-vif

    This change limits the tables registered in the native driver
    to the set actully used by os-vif. This will shorten the inital
    startup time and reconnection time if the ovs db connection is dropped.
    as a result this will help mitigate bug #1929446 where on reconnection
    the nova compute agent can stall until reconnection is completed.

    Change-Id: I635dff2b4fcff905ca8f431eb7e928265200f92a
    Partial-Bug: #1929446
    (cherry picked from commit e4dc8b5664ccee8bde9e90fc9e618d6b705a0b68)

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/841773
Committed: https://opendev.org/openstack/os-vif/commit/700b8cc377c6caa79e8d98cd55b34d77f23ff780
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 700b8cc377c6caa79e8d98cd55b34d77f23ff780
Author: Sean Mooney <email address hidden>
Date: Thu Aug 19 14:32:42 2021 +0100

    Use TCP keepalives for ovsdb connections

    Ultimately, this is something that should be fixed in python-ovs,
    but setting the SO_KEEPALIVE socket option benefits the client by
    removing the need to send 'echo' requests, which can time out on
    an overloaded ovsdb-server, which causes a disconnection which then#
    adds even more load on the ovsdb-server as it has to send the entire
    db contents over the wire after the connection is restored.

    This patch ports the optimisation form neutron to reduce the likelyhood
    of a reconnection which can cause the nova compute agent to hang
    temporarily while the connection is reestablished.

    Change-Id: I984ec62730276f8ee60d71a02a98fbfc4c37f7d8
    Related-Bug: #1930926
    Partial-Bug: #1929446
    (cherry picked from commit 09c0629bb728ad342a41d844143d8e7437c925c4)
    (cherry picked from commit 26f073f0969cfd9719375a4edfc16ce4ec139c4f)

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/841774
Committed: https://opendev.org/openstack/os-vif/commit/bac7f7d9d4337c60bf684d5b518c79f9aff95771
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit bac7f7d9d4337c60bf684d5b518c79f9aff95771
Author: Sean Mooney <email address hidden>
Date: Mon Aug 23 13:11:16 2021 +0100

    only register tables used by os-vif

    This change limits the tables registered in the native driver
    to the set actully used by os-vif. This will shorten the inital
    startup time and reconnection time if the ovs db connection is dropped.
    as a result this will help mitigate bug #1929446 where on reconnection
    the nova compute agent can stall until reconnection is completed.

    Change-Id: I635dff2b4fcff905ca8f431eb7e928265200f92a
    Partial-Bug: #1929446
    (cherry picked from commit e4dc8b5664ccee8bde9e90fc9e618d6b705a0b68)
    (cherry picked from commit e6e791bd0af560ac501e383814fa7a5bce004536)

just to be clear this fix required a new enough version fo OVS specific the ovs python binding that supports cooperative_yield() introduced in 2.16.0+

locally i have

(venv) stack@upstream-devstack:~$ pip freeze | grep ovs
ovs==2.17.1.post1
ovsdbapp==2.4.0

which should be patched but I'm still seeing the old behavior on ubuntu 22.04

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/841780
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/841779
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/os-vif/+/841777
Reason: stable/ussuri branch of openstack/os-vif transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/os-vif/+/841778
Reason: stable/ussuri branch of openstack/os-vif transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/os-vif/+/841776
Reason: stable/victoria branch of openstack/os-vif is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/os-vif/+/841775
Reason: stable/victoria branch of openstack/os-vif is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.