Searchlight does not receive notification when deployed in same env as Ceilometer

Bug #1470627 reported by Travis Tripp
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Searchlight
Fix Released
High
Lakshmi N Sampath

Bug Description

Currently notifications from other services may be consumed by other listeners and searchlight will not received the notification. This does mean that you can not currently use listener based updates in an environment that has ceilometer deployed. You will need
to periodically perform a few full re-index operation.

Changed in searchlight:
importance: Undecided → High
Changed in searchlight:
assignee: nobody → Lakshmi N Sampath (lakshmi-sampath)
Changed in searchlight:
status: New → In Progress
Revision history for this message
Travis Tripp (travis-tripp) wrote :
Revision history for this message
Lakshmi N Sampath (lakshmi-sampath) wrote :
Download full text (6.5 KiB)

After discussion with ceilometer team, the decision was to use another topic for notifications specific to searchlight.

Here's the IRC meeting transcirpt:
<gordc> #topic Open discussion
* openstack has changed the topic to: Open discussion (Meeting topic: ceilometer)
* sudipto_ has quit (Ping timeout: 264 seconds)
<gordc> does anyone have oslo.messaging 1.16.x + devstack installed?
* ganeshna (~ganeshna@72.163.220.5) has joined
* dprince has quit (Ping timeout: 256 seconds)
<gordc> just for reference, https://bugs.launchpad.net/ceilometer/+bug/1475307
<openstack> Launchpad bug 1475307 in Ceilometer "collector has possible memory leak" [Critical,New]
<lakshmiS> Have a question for ceilometer team. I am core commiter on Openstack searchlight project and we consume notifications from glance but when ceilometer is running, searchlight listener doesn't recieve those notifications.
<lakshmiS> We have to stop ceilometer for searchlight to receive new notifications. Searchlight project has a patchset to include a pool name: https://review.openstack.org/#/c/202392/
<lakshmiS> Can ceilometer do similarly or any other suggestions?
<eglynn> lakshmiS: you need to configure glance to emit on two topics IIRC
<gordc> lakshmiS: ^
<cdent> eglynn: that seems unfair
* moha_hunt has quit (Quit: moha_hunt)
* ganeshna has quit (Read error: Connection reset by peer)
<eglynn> cdent: life is unfair :)
<lakshmiS> can we not use different pool names: atleast that's what the oslo messaging document says
<cdent> you're basically putting a dependency on a source for their to be multiple listeners
<cdent> s/their/there/
<cdent> whereas what we'd like in a distributed system is for there to be N potential listeners without needing to change the source
* ivasev has quit (Read error: Connection reset by peer)
<gordc> lakshmiS: there was an issue with that oslo.messaging
* cdent looks at sileht
* afazekas (~turul@68.152.broadband9.iol.cz) has joined
* pm90_ has quit (Read error: Connection reset by peer)
<lakshmiS> cdent: +1
<sileht> that should just work
<eglynn> cdent: yeap, it's certainly not pretty, I just think that's the current reality
* cdent does not aspire to reality ;)
* ivasev (~<email address hidden>) has joined
* jorge_munoz has quit (Read error: Connection reset by peer)
<eglynn> sileht: a-ha, k ... good to know
* pm90_ (~<email address hidden>) has joined
<lakshmiS> I tried changing ceilometer listener code to have a pool name but it stops working afer a while
<lakshmiS> is there anyone from ceilometer team I can discuss this offline?
<gordc> sileht: wasn't there an issue with pools, something about it not being created at right time so we'd lose messages?
* jorge_munoz (~jorge@72.32.115.231) has joined
* ganeshna (~ganeshna@2001:420:c0e0:1005::2eb) has joined
* _nadya_ has quit (Remote host closed the connection)
<sileht> kmartin,you need to take a look to your rabbitmq queues and find those prefixed with searchlight-listener
<gordc> lakshmiS: we're all over at #openstack-ceilometer (more euro based) but there's a few of us (me) in americas tz
* _nadya_ (~<email address hidden>) has joined
* vishwanathj (~vishwanat...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to searchlight (master)

Reviewed: https://review.openstack.org/202392
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=080c8993d0fcfd7d473bd3ee5f611938eb61e518
Submitter: Jenkins
Branch: master

commit 080c8993d0fcfd7d473bd3ee5f611938eb61e518
Author: Lakshmi N Sampath <email address hidden>
Date: Wed Jul 15 22:44:53 2015 -0700

    Fix issue with multiple notification listeners

    Adds searchlight specific topic for glance which
    resolves the conflict with ceilometer and other
    services which also consume glance notifications

    Closes-Bug: #1470627
    Change-Id: Iae28a1527fa93bca4de59fd272eb29b5cbb30fee

Changed in searchlight:
status: In Progress → Fix Committed
Changed in searchlight:
milestone: none → liberty-rc1
Changed in searchlight:
status: Fix Committed → Fix Released
Changed in searchlight:
milestone: liberty-rc1 → 0.1.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.