OpenStack Identity (keystone)

Service managers starting keystone-all don't know when its ready

Bug #980037 reported by Derek Higgins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Alan Pevec
OpenStack Identity (keystone) 2012.2 "folsom"

Bug Description

If starting keystone-all with a Service Manager (systemd for example), keystone has no way of reporting back to systemd that it is ready to serve http requests and as a result its possible for systemd to return before keystone is ready.

For example on Fedora
where the systemd process start-up type is set to simple (i.e. just start the process and return)

> /bin/systemctl stop openstack-keystone.service ; /bin/systemctl start openstack-keystone.service ; /usr/bin/keystone --token keystone_admin_token --endpoint http://127.0.0.1:35357/v2.0/ service-list
Unable to communicate with identity service: 'NoneType' object has no attribute 'makefile'. (HTTP 400)

Derek Higgins (derekh)
Changed in keystone:
assignee: nobody → Derek Higgins (derekh)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6489

Changed in keystone:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/6489
Committed: http://github.com/openstack/keystone/commit/6c5f7d9e10f059e2bc1e56ea2d4e32a7e4cadc8b
Submitter: Jenkins
Branch: master

commit 6c5f7d9e10f059e2bc1e56ea2d4e32a7e4cadc8b
Author: Derek Higgins <email address hidden>
Date: Wed May 9 21:17:41 2012 +0000

    notify calling process we are ready to serve

    Fixes bug 980037

    Service managers starting keystone-all have no way of being notified
    when the service is ready to accept connections. This commit allows
    a configurable command to be called when we are ready e.g.
    for systemd setting the statup type of a service unit to "notify" and setting
    onready = systemd-notify --ready
    in keystone.conf, would notify a waiting systemd that we are ready to
    serve

    In an automated envirnment (e.g. puppet) this will allow the startup of
    the keystone-all service (with systemctl for example) directly followed
    by usage of the keystone client without the need for a sleep (or retry)
    while we are waiting for the keystone service to be ready.

    Change-Id: I3f7aafe9837be60a0f35cae1a7db892f6851cc47

Changed in keystone:
status: In Progress → Fix Committed
Joseph Heck (heckj)
Changed in keystone:
milestone: none → folsom-1
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Revision history for this message
Alan Pevec (apevec) wrote :

I'd like to reopen this to address issue with systemd notifications https://bugzilla.redhat.com/show_bug.cgi?id=820448#c2
My proposed patch builds on 6c5f7d9e10f059e2bc1e56ea2d4e32a7e4cadc8b and reuses "onready" configuration parameter like this:
1. try import_module(CONF.onready) and call mod.notify() for in process notification (see above BZ comment for an example)
2. fallback to the current behaviour, running a shell command

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/9219

Changed in keystone:
assignee: Derek Higgins (derekh) → Alan Pevec (apevec)
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/9219
Committed: http://github.com/openstack/keystone/commit/abc06716d027d68f0da3b0f559fa7c85a21804d5
Submitter: Jenkins
Branch: master

commit abc06716d027d68f0da3b0f559fa7c85a21804d5
Author: Alan Pevec <email address hidden>
Date: Mon Jul 2 14:07:06 2012 +0200

    notify calling process we are ready to serve

    Fixes bug 980037 again

    Systemd notification should be sent in-process, otherwise systemd might
    miss the subprocess sending notification.
    See systemd bug https://bugzilla.redhat.com/show_bug.cgi?id=820448

    Change-Id: Iccc51cf77af5598ee6b4c3cd69a12a7ee9fc2eb5

Changed in keystone:
status: In Progress → Fix Committed
Joseph Heck (heckj)
Changed in keystone:
status: Fix Committed → Fix Released
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/13237

Changed in keystone:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/13237
Committed: http://github.com/openstack/keystone/commit/b9a7bd46375268ddc6d8fc1aa035a61e271d940f
Submitter: Jenkins
Branch: master

commit b9a7bd46375268ddc6d8fc1aa035a61e271d940f
Author: Alan Pevec <email address hidden>
Date: Wed Sep 19 00:26:13 2012 +0200

    notify calling process we are ready to serve

    Fixes bug 980037 again and again

    Recent SystemD moved notification socket into abstract namespace:
    http://cgit.freedesktop.org/systemd/systemd/commit/?id=29252e9e5bad3b0bcfc45d9bc761aee4b0ece1da

    Change-Id: Idfb1dfb272f06a8066843f0f5750ff6e70f6bc64

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: folsom-1 → 2012.2
Jakub Libosvar (libosvar)
Changed in neutron:
assignee: nobody → Jakub Libosvar (libosvar)
Mark McClain (markmcclain)
no longer affects: neutron
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.