Add process monitor for haproxy

Bug #1565801 reported by Nir Magnezi
20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
octavia
Fix Released
Low
Nir Magnezi

Bug Description

Bug 1565511 aims to solve cases where the lbaas agent goes offline.
To have a complete high-availability solution for lbaas agent with haproxy running in namespace, we would also want to handle a case where the haproxy process itself stopped.

This[1] neutron spec offers the following approach:
"We propose monitoring those processes, and taking a configurable action, making neutron more resilient to external failures."

[1] http://specs.openstack.org/openstack/neutron-specs/specs/juno/agent-child-processes-status.html

Tags: lbaas
Revision history for this message
Assaf Muller (amuller) wrote :

I'm not sure if we want to use the RFE process here. On paper it should be less than 20 lines of code to implement this.

Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

Well, on paper yes, but if I don't remember it wrong the driver is not using our linux.utils.ProcessManager to handle the processes. Although if it uses a class, they just need to provide a couple of attributes to make it possible.

Akihiro Motoki (amotoki)
Changed in neutron:
status: New → Confirmed
importance: Undecided → Wishlist
Revision history for this message
Michael Johnson (johnsom) wrote :
Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

@michael, upstart is ubuntu specific, and we should aim for something distro-agnostic.

Revision history for this message
Akihiro Motoki (amotoki) wrote :

It is a reasonable request. The implementation would be small.

One thing we need to check is the destiny of haproxy agent in LBaaS
whether we accept new features for LBaaS v2 haproxy namespace driver.
We need to check this to Doug.

Moving this to Triaged.

Changed in neutron:
status: Confirmed → Triaged
Revision history for this message
Nir Magnezi (nmagnezi) wrote :

@Doug

Could you please address amotoki's comment about the future of haproxy within namespace in LBaaSv2[1]?

[1] https://bugs.launchpad.net/neutron/+bug/1565801/comments/5

Nir Magnezi (nmagnezi)
Changed in neutron:
assignee: nobody → Nir Magnezi (nmagnezi)
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I agree this can be dealt with a simple bug fix, assumed that there's a desire to continue to support this driver.

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

let's discuss it at the drivers meeting so that we can reject or bless this one.

Revision history for this message
Doug Wiegley (dougwig) wrote :

The lbaas team decided to not let the old reference driver rot, but also to not add new features. If you need an official answer, I'd say it should be, "no", on that basis.

If it truly is a trivial amount of code, which is used elsewhere, and a review appeared, I expect it'd get looked at, though.

tags: removed: rfe
summary: - [RFE] Add process monitor for haproxy
+ Add process monitor for haproxy
Changed in neutron:
importance: Wishlist → Low
Changed in neutron:
status: Triaged → In Progress
Revision history for this message
Nir Magnezi (nmagnezi) wrote :

Reviews needed for this fix: https://review.openstack.org/#/c/327966/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lbaas (master)

Reviewed: https://review.openstack.org/327966
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=56795d73094832b58b4804007ed31b5e896f59fc
Submitter: Jenkins
Branch: master

commit 56795d73094832b58b4804007ed31b5e896f59fc
Author: Nir Magnezi <email address hidden>
Date: Thu Jun 9 23:36:19 2016 +0300

    Implements ProcessMonitor in the haproxy driver

    The ProcessMonitor class will monitor spawned external processes.

    This patch enhances the HaproxyNSDriver class (v2) to utilize the
    external_process module in order to monitor and respawn the haproxy processes
    if and when needed.

    With this patch the LBaaS agent (v2) will load external_process related options
    in order to take a configured action when haproxy process dies unexpectedly.

    Closes-Bug: #1565801

    Change-Id: I420ca20b2620487909885e0e9f08dae60ebec2bf

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lbaas (master)

Fix proposed to branch: master
Review: https://review.openstack.org/344658

Nir Magnezi (nmagnezi)
Changed in neutron:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/345263

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/345263
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ec929d9b1caaf970f431df10b10dc5fed6a9cc9f
Submitter: Jenkins
Branch: master

commit ec929d9b1caaf970f431df10b10dc5fed6a9cc9f
Author: Nir Magnezi <email address hidden>
Date: Thu Jul 21 11:46:49 2016 +0300

    Adds a default reload callback to ProcessManager

    Currently, process reload in ProcessManager is being done with 'HUP'.
    Some processes, such as haproxy[1], use a different method for reloading
    configuration updates.

    This patch addes a default reload callback option to handle such cases, while
    keeping 'HUP' as the default method, in case a default reload callback
    was not provided.

    [1] See -sf flags: http://www.haproxy.org/download/1.2/doc/haproxy-en.txt

    Related-Bug: #1565801

    Change-Id: Ia2f67e742778cd99f2211c94782635652d9032c7

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-lbaas 9.0.0.0b3

This issue was fixed in the openstack/neutron-lbaas 9.0.0.0b3 development milestone.

affects: neutron → octavia
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lbaas (master)

Reviewed: https://review.openstack.org/344658
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=8c01eff53b2d399fe63d62a55b13487877e8a0b7
Submitter: Jenkins
Branch: master

commit 8c01eff53b2d399fe63d62a55b13487877e8a0b7
Author: Nir Magnezi <email address hidden>
Date: Wed Jul 20 10:31:28 2016 +0300

    Implements ProcessMonitor in the haproxy driver

    The ProcessMonitor class will monitor spawned external processes.

    This patch enhances the HaproxyNSDriver class (v2) to utilize the
    external_process module in order to monitor and respawn the haproxy
    processes if and when needed.

    With this patch the LBaaS agent (v2) will load external_process related
    options in order to take a configured action when haproxy process
    dies unexpectedly.

    This is the second attempt to introduce ProcessMonitor in the haproxy
    driver. The first attempt I420ca20b2620487909885e0e9f08dae60ebec2bf
    caused an issue where haproxy failed to acquire new configuration, due
    to the lack of -sf flags which should be provided when haproxy.conf is
    being updated. More information about the flags here[1]

    [1] http://www.haproxy.org/download/1.2/doc/haproxy-en.txt

    Co-Authored-By: Thomas Herve <email address hidden>
    Closes-Bug: #1565801
    Related-Bug: #1603860

    Change-Id: Ia47b1586be17be421e01c131859dd0d50b1d7db6

Changed in octavia:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lbaas (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/425084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron-lbaas (stable/newton)

Change abandoned by Nir Magnezi (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/425084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron-lbaas 10.0.0.0b3

This issue was fixed in the openstack/neutron-lbaas 10.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.