Cinder does not shutdown processes gracefully

Bug #1464822 reported by Mitsuhiro Tanino
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Low
Mitsuhiro Tanino
Kilo
Fix Released
Undecided
Unassigned

Bug Description

Currently, cinder services does not shutdown gracefully when they receive signals such as SIGTERM, SIGINT in order to shutdown parent and child processes.

This is because Cinder service stop() method stops processes instantly without waiting their services completion properly. This leads a problem to make odd database records and inconsistency between DB and backend storages.

Nova already fixed same issue.
https://bugs.launchpad.net/nova/+bug/1295625

We need to introduce same logic into Cinder for providing graceful shutdown.

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/191515

Changed in cinder:
status: New → In Progress
Changed in cinder:
importance: Undecided → Low
Revision history for this message
Mitsuhiro Tanino (mitsuhiro-tanino) wrote :

Fix proposed to branch oslo.service (master)
Review: https://review.openstack.org/#/c/189418/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/191515
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fcb5068e79490351359afdec9f05f42ebb022edf
Submitter: Jenkins
Branch: master

commit fcb5068e79490351359afdec9f05f42ebb022edf
Author: Mitsuhiro Tanino <email address hidden>
Date: Sat Jun 13 00:23:40 2015 -0400

    Wait until service thread is done on service stop

    In order to be able to perform graceful shutdown of services, we need
    to wait for all the current service threads to finish before exiting
    from the service process.

    Oslo.service provides this facility through it's wait() method, so we
    need to call it in Cinder Service stop() method which gets called when
    graceful shutdown is requested by sending the process SIGINT or SIGTERM.

    Partial-Bug: 1464822
    Change-Id: I972b80251cebe353f9d89ff4e05db8d029ef5e73

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/191862

Revision history for this message
Mitsuhiro Tanino (mitsuhiro-tanino) wrote :

Waiting the fix will be landed at oslo.service. After introducing oslo.service into Cinder, we need an additional fix to close this bug.

https://bugs.launchpad.net/oslo.service/+bug/1382390
https://review.openstack.org/#/c/190175/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/197369

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/kilo)

Change abandoned by Mitsuhiro Tanino (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/191862
Reason: The fix is posted following review.

https://review.openstack.org/#/c/197369/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/kilo)

Reviewed: https://review.openstack.org/197369
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=70c6a356844c1f0773fdeeb472db634bf8896892
Submitter: Jenkins
Branch: stable/kilo

commit 70c6a356844c1f0773fdeeb472db634bf8896892
Author: Mitsuhiro Tanino <email address hidden>
Date: Mon Jun 15 00:12:17 2015 -0400

    Graceful shutdown WSGI/RPC server

    Currently, termination of WSGI application or RPC server immediately stops
    service and so interrupts in-progress request.

    Graceful handler for SIGTERM signal was added.
    SIGINT signal handler was removed to allow instantaneous termination of
    service.

    DocImpact: graceful termination of process can be done by sending SIGTERM
    signal to parent WSGI process. Graceful termination is not instantaneous.
    To force instantaneous termination SIGINT signal must be sent.

    (cherry picked from commit 13ce823686062d70b268b4d3888849adef07e4ff)
      Additional fix: oslo.service provides this facility through it's wait()
                      method, so we need to call it in Cinder Service wait()
                      method which gets called when graceful shutdown is
                      requested by sending the process SIGTERM.

    Change-Id: Icaedec63c0df0255c6842b688c6d83a496f142b8
    Closes-Bug: 1464822

tags: added: in-stable-kilo
Revision history for this message
Duncan Thomas (duncan-thomas) wrote :

Reverting the fix since it seems to be breaking grenade: https://review.openstack.org/#/c/200134/

Changed in cinder:
assignee: Mitsuhiro Tanino (mitsuhiro-tanino) → Tomoki Sekiyama (tsekiyama)
Changed in cinder:
assignee: Tomoki Sekiyama (tsekiyama) → Mitsuhiro Tanino (mitsuhiro-tanino)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/194929
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=87db01e977e0fe55df762b7f4d26e669d47cbefd
Submitter: Jenkins
Branch: master

commit 87db01e977e0fe55df762b7f4d26e669d47cbefd
Author: Mitsuhiro Tanino <email address hidden>
Date: Tue Jun 23 23:49:50 2015 -0400

    Graceful shutdown WSGI/RPC server

    Change-ID I80487f559302b0e88953835ffb4a3bfe2a347501 introduces
    oslo.service into Cinder. The graceful shutdown of RPC server
    was introduced via merging of oslo.service.
    Actual fix was introduced by https://review.openstack.org/#/c/190175/
    into oslo.service.

    The oslo.service fix provides this facility through it's wait()
    method, so we need to call it in Cinder Service wait() method
    which gets called when graceful shutdown is requested by sending
    the process SIGTERM.

    Change-Id: Iea877e062272103ea48583b5638dec6174e99de0
    Closes-Bug: 1464822

Changed in cinder:
status: In Progress → Fix Committed
Changed in cinder:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: liberty-2 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.