Internal interfaces defined via OVS are not brought up properly after a reboot

Bug #1091605 reported by Maru Newby
40
This bug affects 8 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Gary Kotton
Folsom
Fix Released
High
Gary Kotton
quantum (Ubuntu)
Won't Fix
Medium
Jian Wen

Bug Description

The L3 agents and DHCP agents both define internal (qg-, qr-, tap-) ports via OVS. In both cases, the agents call plug() to configure and bring the device up if it does not exist. If the device does exist, however, the agents neither call plug nor do they ensure the link is up (OVS ensures that the devices survive a reboot but does not ensure that they are brought up on boot).

The responsibility for bringing devices up should probably remain in quantum/agent/linux/interface.py, so a suggested implementation would be delegating the device existence check to the driver's plug() method, which could then ensure that the device was brought up if necessary.

This bug reveals a hole in our current testing strategy. Most developers presumably work on devstack rather than installed code. Since devstack agents don't survive a reboot, most developers would never have the chance to validate whether a quantum agent node still works after a reboot. Documenting use-cases that need to be tested (e.g. quantum agent nodes need to work properly after a reboot) is a good first step - is this currently captured somewhere or can we find a place to do so?

Gary Kotton (garyk)
Changed in quantum:
assignee: nobody → Gary Kotton (garyk)
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Gary Kotton (garyk) wrote :

This is an interesting problem.
Ideally it would be best if the OVS did not have the persistant devices after reboot. I am not sure if it is possible to configure it in a way that the devices will not be recreated after reboot.
One alternative is provide a script that will purge the ovs of devices created by quantum (this can be run prior to running the agents that make use of the ovs
Thoughts?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/18302

Changed in quantum:
status: Confirmed → In Progress
Gary Kotton (garyk)
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (master)

Reviewed: https://review.openstack.org/18302
Committed: http://github.com/openstack/quantum/commit/767859ddb947f281c4ada930ccc21cc6f0ba6806
Submitter: Jenkins
Branch: master

commit 767859ddb947f281c4ada930ccc21cc6f0ba6806
Author: Gary Kotton <email address hidden>
Date: Tue Dec 18 12:20:50 2012 +0000

    Add OVS cleanup utility

    Fixes bug 1091605

    The utility should be called after rebooting an appliance. This
    will purge the openvswicth of configured tap devices.

    A configuration variable quantum_ports has been added. This is
    by default True which indicates that only Quantum ports will be
    deleted from the OVS. If this is set as False then all ports on the
    bridge will be deleted.

    Change-Id: I442f64cf82f95bfa99d7765eb09db1ce2ecf602e

Changed in quantum:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/18638

Revision history for this message
Akihiro Motoki (amotoki) wrote :

OVS interface driver has two modes: the one uses interval OVS port and the other uses a veth pair (if ovs_use_veth is True).
In the former case OVS keeps ports across reboot. In the latter case OVS keeps port information but veth device itself disappears after reboot. I agree with Gary's approach.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (stable/folsom)

Reviewed: https://review.openstack.org/18638
Committed: http://github.com/openstack/quantum/commit/43ece8fb4c1bf77b662d28fa3bff42342b2a7a41
Submitter: Jenkins
Branch: stable/folsom

commit 43ece8fb4c1bf77b662d28fa3bff42342b2a7a41
Author: Gary Kotton <email address hidden>
Date: Tue Dec 18 12:20:50 2012 +0000

    Add OVS cleanup utility

    Fixes bug 1091605

    The utility should be called after rebooting an appliance. This
    will purge the openvswitch of configured tap devices.

    A configuration variable quantum_ports has been added. This is
    by default True which indicates that only Quantum ports will be
    deleted from the OVS. If this is set as False then all ports on the
    bridge will be deleted.

    Change-Id: I442f64cf82f95bfa99d7765eb09db1ce2ecf602e

tags: added: in-stable-folsom
Gary Kotton (garyk)
tags: removed: folsom-backport-potential
Thierry Carrez (ttx)
Changed in quantum:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Mark McLoughlin (markmc)
tags: removed: in-stable-folsom
Thierry Carrez (ttx)
Changed in quantum:
milestone: grizzly-2 → 2013.1
Jian Wen (wenjianhn)
Changed in quantum (Ubuntu):
status: New → In Progress
assignee: nobody → Jian Wen (wenjianhn)
Revision history for this message
Jian Wen (wenjianhn) wrote :
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "bug_1091605.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Jian Wen (wenjianhn)
tags: removed: patch
Revision history for this message
Dmitry Shachnev (mitya57) wrote :

Thanks for your work here. Please do a proper changelog entry and forward this change to Debian (upstart exists in Debian as well).

James Page (james-page)
Changed in quantum (Ubuntu):
status: In Progress → Fix Released
status: Fix Released → Triaged
importance: Undecided → Medium
Revision history for this message
Luke Faraone (lfaraone) wrote :

I similarly recommend this patch be forwarded to Debian; the "submittodebian" command can help with the process.

If you want your patch included in Ubuntu, please create a separate changelog entry for it instead of appending to the current entry.

Unsubscribing ~ubuntu-sponsors.

Changed in quantum (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Jian Wen (wenjianhn) wrote :

The OVS cleanup utility introduces another bug that I think is important:
  https://bugzilla.redhat.com/show_bug.cgi?id=1010941

Maybe purging the ovs of devices created by quantum after
reboot is not a good idea.

Revision history for this message
James Page (james-page) wrote :

quantum is no longer found in any supported Ubuntu release

Changed in quantum (Ubuntu):
status: Incomplete → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.