octavia

Failover fails with haproxy network namespaces

Bug #1577963 reported by Michael Johnson
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
octavia
Fix Released
Critical
Michael Johnson

Bug Description

With the new network namespace for HAProxy inside the amphora, when a failover is triggered the failover flow calls ListenersUpdate before the network ports are fully plugged into the amphora(before AmphoraPostVIPPlug and AmphoraPostNetworkPlug are called). This causes HAProxy to fail to restart (called by listenerupdate) because the /var/lib/octavia/plugged_interfaces file is not yet present. The file should normally be present before ListenersUpdate is called as the VIP should be plugged.

Either the failover flow needs to be re-ordered or the haproxy startup scripts need to be smarter about the situation where the plugged_interfaces file is missing

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to octavia (master)

Fix proposed to branch: master
Review: https://review.openstack.org/312270

Changed in octavia:
status: New → In Progress
Revision history for this message
Michael Johnson (johnsom) wrote :

Ok, I have been peeling back the onion on failover recently.

First issue, neutron reverted enabling the DNS integration by default which caused our port detach code to fail.

This was reported as bug: https://review.openstack.org/341102 and fixed in: https://review.openstack.org/#/c/337939/

Revision history for this message
Michael Johnson (johnsom) wrote :

Next layer of the onion, nova made a recent change that broke the ability to reattach a port to a new instance.

The change that broke us: https://review.openstack.org/#/c/264017/
The bug I opened: https://launchpad.net/bugs/1602357
The proposed fix: https://review.openstack.org/#/c/341102/
A tempest test to catch this in the future: https://review.openstack.org/#/c/341191/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to octavia (master)

Reviewed: https://review.openstack.org/312270
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=7ba33e6ee4dd19757fe6f3a6bae513db49240c5e
Submitter: Jenkins
Branch: master

commit 7ba33e6ee4dd19757fe6f3a6bae513db49240c5e
Author: Michael Johnson <email address hidden>
Date: Tue May 3 22:30:42 2016 +0000

    Fixes failover flow with namespace driver

    This patch updates the haproxy service scripts to handle the case
    where the network interfaces have not yet been plugged. This can
    occur in a failover situation.
    This patch also makes sure we don't move the management lan interface
    into the network namespace.

    Closes-Bug: #1509706
    Closes-Bug: #1577963
    Change-Id: I04d267bd3cdedca11f0350c5255086233cba14ec

Changed in octavia:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to octavia (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/361520

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/octavia 0.9.0

This issue was fixed in the openstack/octavia 0.9.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on octavia (stable/mitaka)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/361520
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/octavia 0.9.0

This issue was fixed in the openstack/octavia 0.9.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.