Excessive CPU usage in ProcessLauncher()'s wait loop

Bug #1095346 reported by Mark McLoughlin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Maru Newby
oslo-incubator
Fix Released
Medium
Maru Newby

Bug Description

See https://review.openstack.org/18689 for some background

We can't use os.wait() to block until a child exited so, instead, we're busy-looping

We should be able to come up with another way of doing this - e.g. using pipes provided to each child to give us a selectable handle we can block on

Changed in oslo:
assignee: nobody → Davanum Srinivas (dims-v)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/18689
Committed: http://github.com/openstack/oslo-incubator/commit/ffeb0855085617095f19296770a1223cb5641d1c
Submitter: Jenkins
Branch: master

commit ffeb0855085617095f19296770a1223cb5641d1c
Author: Davanum Srinivas <email address hidden>
Date: Thu Dec 27 12:22:56 2012 -0500

    Don't rely on os.wait() blocking

    Before this fix which was included in eventlet 0.10.0:

    https://bitbucket.org/which_linden/eventlet/issue/92/eventletgreen-override-of-oswaitpid

    the os.wait() wrapper in eventlet would not block if no child processes
    had exited. This was clearly buggy behaviour and os.wait() no blocks
    correctly. However, it appears it now does not return if the parent process
    was interrupted by a signal. This means that you can't kill the parent process
    and have the child processes cleaned up gracefully.

    To avoid all these issues, switch to non-blocking wait behaviour. We use
    WNOHANG with waitpid() to poll for an exited child process and, if there
    is none, we yield to other green threads and sleep for a short period.
    This means excessive CPU usage, though, which will be tracked by
    bug #1095346.

    Check pid to avoid some unnecessary warning/info messages in the logs

    Fixes LP #1094076

    Change-Id: I783fac652376c2daa9d591403b9f4c1fe9523043

Changed in oslo:
status: In Progress → Fix Committed
Mark McLoughlin (markmc)
Changed in oslo:
status: Fix Committed → Triaged
assignee: Davanum Srinivas (dims-v) → nobody
Mark McLoughlin (markmc)
Changed in oslo:
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/63017

Changed in oslo:
assignee: nobody → Maru Newby (maru)
status: Triaged → In Progress
Maru Newby (maru)
Changed in oslo:
assignee: Maru Newby (maru) → nobody
Changed in oslo:
assignee: nobody → Maru Newby (maru)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/63017
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=32cd4c002d3e1074c35032d10366f3a968f8c46f
Submitter: Jenkins
Branch: master

commit 32cd4c002d3e1074c35032d10366f3a968f8c46f
Author: Maru Newby <email address hidden>
Date: Wed Dec 18 23:07:52 2013 +0000

    Allow configurable ProcessLauncher liveness check

    The current interval between ProcessLauncher checks of whether
    a child process has exited - 0.01s - prioritizes fast respawning of
    children at the expense of cpu usage. This tradeoff may not be
    appropriate for all clients of the class, so this patch makes the
    interval configurable. The current interval remains the default
    to avoid changing expected behavior.

    This patch was motivated by unnecessary idle cpu usage of the
    Neutron service when configured to use ProcessLauncher-managed
    worker processes.

    Partial-Bug: #1095346

    Change-Id: I2f5e06f98be1f75c21e6681967b573f5d88a4ed6

Changed in oslo:
milestone: none → icehouse-2
status: In Progress → Fix Committed
Maru Newby (maru)
Changed in neutron:
assignee: nobody → Maru Newby (maru)
milestone: none → icehouse-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/67919

Changed in neutron:
status: New → In Progress
Kyle Mestery (mestery)
Changed in neutron:
importance: Undecided → Medium
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-2 → icehouse-3
Thierry Carrez (ttx)
Changed in oslo:
status: Fix Committed → Fix Released
Maru Newby (maru)
Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo:
milestone: icehouse-2 → 2014.1
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.