Missing neutron-ovs-cleanup service

related to bug 1091605

On Mon, Dec 9, 2013 at 5:28 PM, James Page <email address hidden> wrote:

> ** Changed in: cloud-archive
> Status: New => Triaged
>
> ** Changed in: cloud-archive
> Importance: Undecided => Medium
>
> ** Also affects: neutron (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: neutron (Ubuntu)
> Status: New => Triaged
>
> ** Changed in: neutron (Ubuntu)
> Importance: Undecided => High
>
> ** Changed in: neutron (Ubuntu)
> Assignee: (unassigned) => James Page (james-page)
>
> --
> You received this bug notification because you are subscribed to ubuntu-
> cloud-archive.
> Matching subscriptions: cloud archive
> https://bugs.launchpad.net/bugs/1249708
>
> Title:
> Missing neutron-ovs-cleanup service
>
> Status in Ubuntu Cloud Archive:
> Triaged
> Status in “neutron” package in Ubuntu:
> Triaged
>
> Bug description:
> When the neutron network node is rebooted, the "neutron-ovs-cleanup"
> command must be run before the neutron services start or several
> services like L3 and DHCP won't work properly .
>
> The RHEL OpenStack packages install a system service that
> automatically runs this command, but on Ubuntu the sysadmin is
> responsible for running this command manually or setting up a system
> service manually.
>
> It would make sysadmin life easier if the Ubuntu packages also
> installed a service that invoked neutron-ovs-cleanup at the
> appropriate time.
>
> References:
>
> * https://bugs.launchpad.net/openstack-manuals/+bug/1156861
> * https://bugs.launchpad.net/neutron/+bug/1084355
> * https://ask.openstack.org/en/question/2605/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-archive/+bug/1249708/+subscriptions
>

--
Cheers,
Jian

I started working on this and I agree that this tool needs to be run; but for me its unclear as to when this should happen.

Does this need to happen on network forwarding nodes only (l3 and dhcp services) and for the openvswitch/ml2 plugins only? In which case it might make sense to run this in the pre-start block of the neutron-plugin-openvswitch-agent upstart configuration.

I believe it's an issue whenever you have a Neutron service that creates tap devices (openvswitch internal ports). I think only the l3 and dhcp services do this currently,

I think this tool needs to be run on l3 nodes and dhcp nodes.
It should only be run when rebooting a node.
And it should finish the cleaning before l3 agent or dhcp agent starts.

In some environments, nova compute, l3 agent and dhcp agent are on the same node [1].
We need to make sure that the tool finishes the cleaning before nova compute starts.

It's kind of complicated. That's why I don't like the idea that using a upstart/init job to do the cleaning.
Maybe it's better that l3 agent and dhcp agent replug the ports themselves during starting.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1010941

I agree its complicated; I discussed with upstream PTL and its only needed for l3 and dhcp agents with the openvswitch plugin; the dhcp agent is also used by other plugins so the dependency on cleanup completing first needs to be soft (this can be done).

My current thing is to add this to the neutron-plugin-openvswitch-agent package, add conditional waits to the l3 and dhcp agent configurations and a hard wait for the neutron-plugin-openvswitch-agent.

We can also add the same soft wait to nova-compute if this is required - that creates tap devices as well so I think Jian is correct in #4.

It should start on every compute and network node. It clear up all strange artefacts saved by ovs-vsctl.

How to get 'strange artifacts':
On any working compute/network node with active networking on instances perform any operations on ovs-vsctl, f.e.:
ovs-vsctl add-br just_empty_bridge
ovs-vsctl del-br just_empty_bridge

ovs will save all current tap/tun qr-* interfaces in br-int/br-ext bridges and restore them on next boot inside own config, even if they are not available at server start.

This will cause problems for existing instances during their startup.

neutron-ovs-cleanup should run during startup on all non-diskless nodes to ensure nothing strange is exists in bridges prior fists instance/agent start.

This bug was fixed in the package neutron - 1:2014.1~b2-0ubuntu2

---------------
neutron (1:2014.1~b2-0ubuntu2) trusty; urgency=medium

  * debian/patches/skip-lb-test.patch: Skipped lb configuration
    test.
 -- Chuck Short <email address hidden> Mon, 27 Jan 2014 12:01:50 -0500