OpenStack Searchlight

Bug #1470627
Comment #2

Comment 2 for bug 1470627

Revision history for this message

Lakshmi N Sampath (lakshmi-sampath) wrote on 2015-07-30:

After discussion with ceilometer team, the decision was to use another topic for notifications specific to searchlight.

Here's the IRC meeting transcirpt:
<gordc> #topic Open discussion
* openstack has changed the topic to: Open discussion (Meeting topic: ceilometer)
* sudipto_ has quit (Ping timeout: 264 seconds)
<gordc> does anyone have oslo.messaging 1.16.x + devstack installed?
* ganeshna (~ganeshna@72.163.220.5) has joined
* dprince has quit (Ping timeout: 256 seconds)
<gordc> just for reference, https://bugs.launchpad.net/ceilometer/+bug/1475307
<openstack> Launchpad bug 1475307 in Ceilometer "collector has possible memory leak" [Critical,New]
<lakshmiS> Have a question for ceilometer team. I am core commiter on Openstack searchlight project and we consume notifications from glance but when ceilometer is running, searchlight listener doesn't recieve those notifications.
<lakshmiS> We have to stop ceilometer for searchlight to receive new notifications. Searchlight project has a patchset to include a pool name: https://review.openstack.org/#/c/202392/
<lakshmiS> Can ceilometer do similarly or any other suggestions?
<eglynn> lakshmiS: you need to configure glance to emit on two topics IIRC
<gordc> lakshmiS: ^
<cdent> eglynn: that seems unfair
* moha_hunt has quit (Quit: moha_hunt)
* ganeshna has quit (Read error: Connection reset by peer)
<eglynn> cdent: life is unfair :)
<lakshmiS> can we not use different pool names: atleast that's what the oslo messaging document says
<cdent> you're basically putting a dependency on a source for their to be multiple listeners
<cdent> s/their/there/
<cdent> whereas what we'd like in a distributed system is for there to be N potential listeners without needing to change the source
* ivasev has quit (Read error: Connection reset by peer)
<gordc> lakshmiS: there was an issue with that oslo.messaging
* cdent looks at sileht
* afazekas (~turul@68.152.broadband9.iol.cz) has joined
* pm90_ has quit (Read error: Connection reset by peer)
<lakshmiS> cdent: +1
<sileht> that should just work
<eglynn> cdent: yeap, it's certainly not pretty, I just think that's the current reality
* cdent does not aspire to reality ;)
* ivasev (~<email address hidden>) has joined
* jorge_munoz has quit (Read error: Connection reset by peer)
<eglynn> sileht: a-ha, k ... good to know
* pm90_ (~<email address hidden>) has joined
<lakshmiS> I tried changing ceilometer listener code to have a pool name but it stops working afer a while
<lakshmiS> is there anyone from ceilometer team I can discuss this offline?
<gordc> sileht: wasn't there an issue with pools, something about it not being created at right time so we'd lose messages?
* jorge_munoz (~jorge@72.32.115.231) has joined
* ganeshna (~ganeshna@2001:420:c0e0:1005::2eb) has joined
* _nadya_ has quit (Remote host closed the connection)
<sileht> kmartin,you need to take a look to your rabbitmq queues and find those prefixed with searchlight-listener
<gordc> lakshmiS: we're all over at #openstack-ceilometer (more euro based) but there's a few of us (me) in americas tz
* _nadya_ (~<email address hidden>) has joined
* vishwanathj (~vishwanat@144.49.132.22) has joined
* ujuc (~ujuc@211.219.170.176) has joined
* isviridov_away is now known as isviridov
<lakshmiS> gordc: sileht: can you point me to the issue you had previously with oslo.messaging pools?
<sileht> gordc, yes but you just lost message between the client side restart and the first server side start, that doesn't occurs offen
<sileht> lakshmiS, it not a issue, this is a historical behavior :p
<lakshmiS> ok
<sileht> unsure it's cleanly documented
<lakshmiS> so the solution/alternative is to have multiple topics for notification.info from glance?
<gordc> lakshmiS: post your issue over at #openstack-ceilometer and someone will reply. if not, we're sleeping and ML is good.
* Guest29403 has quit (Remote host closed the connection)
* vishwana_ has quit (Ping timeout: 255 seconds)
<lakshmiS> gordc: i wil do ML since I already did on IRC with no response
<gordc> lakshmiS: that is the alternative yes... that's typically what is done. (ie. stacktach)
<lakshmiS> gordc: thx
* arvinchou_1 (~Adium@61-222-64-29.HINET-IP.hinet.net) has joined
<gordc> sileht: for the above, do ceilometer and searchlight need to create pool or just one? the problem right is we're both using hte same default pool right?
* isviridov is now known as isviridov_away
* _nadya_ has quit (Ping timeout: 256 seconds)
<sileht> gordc, I think the safer ways is to emit the notification twice on the other side
<lakshmiS> sileht: that would be a change on each team(glance, nova) etc right?
<sileht> lakshmiS, yes a configuration change
<sileht> gordc, this method ensure that no notifications will be missed
* ujuc has quit (Ping timeout: 264 seconds)
<gordc> sileht: got it. that's what i understood.
* PrashantS (~PerfBeing@50.56.228.68) has joined
<lakshmiS> sileht: do you have any previous reviews doint this cofig change ?
<lakshmiS> s/doint/doing
* hashar_ (~sempitern@mediawiki/hashar) has joined
<gordc> lakshmiS: its either or i guess. pro/con either way.
* shashank_hegde_ (~<email address hidden>) has joined
* annegentle has quit (Remote host closed the connection)
* PrashantS has quit (Remote host closed the connection)
<gordc> lakshmiS: let me dig up stacktach docs, they might reference it
<lakshmiS> thanks gordc:
* PrashantS (~PerfBeing@50.56.229.9) has joined
* annegentle (~annegentl@50.57.17.244) has joined
* arvinchou_ has quit (Ping timeout: 256 seconds)
<gordc> anything other topics?
* adiantum has quit (Ping timeout: 256 seconds)
<gordc> lakshmiS: i'll send you link over in openstack-ceilometer unless you have another preference
<gordc> 30s countdown
* leeantho (~<email address hidden>) has joined
* hashar has quit (Ping timeout: 240 seconds)
* vivek_ebay is now known as vivek_
<lakshmiS> gordc: please send it to <email address hidden> if possbile too
<gordc> lakshmiS: kk
<gordc> ending meeting. thanks folks
<gordc> #endmeeting
* openstack has changed the topic to: OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings
<openstack> Meeting ended Thu Jul 16 15:55:28 2015 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
<openstack> Minutes: http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.html
<openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.txt
<openstack> Log: http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.log.html

After discussion with ceilometer team, the decision was to use another topic for notifications specific to searchlight.

Here's the IRC meeting transcirpt:
<gordc> #topic Open discussion
* openstack has changed the topic to: Open discussion (Meeting topic: ceilometer)
* sudipto_ has quit (Ping timeout: 264 seconds)
<gordc> does anyone have oslo.messaging 1.16.x + devstack installed?
* ganeshna (~ganeshna@72.163.220.5) has joined
* dprince has quit (Ping timeout: 256 seconds)
<gordc> just for reference, https://bugs.launchpad.net/ceilometer/+bug/1475307
<openstack> Launchpad bug 1475307 in Ceilometer "collector has possible memory leak" [Critical,New]
<lakshmiS> Have a question for ceilometer team. I am core commiter on Openstack searchlight project and we consume notifications from glance but when ceilometer is running, searchlight listener doesn't recieve those notifications. 
<lakshmiS> We have to stop ceilometer for searchlight to receive new notifications. Searchlight project has a patchset to include a pool name: https://review.openstack.org/#/c/202392/
<lakshmiS> Can ceilometer do similarly or any other suggestions?
<eglynn> lakshmiS: you need to configure glance to emit on two topics IIRC
<gordc> lakshmiS: ^
<cdent> eglynn: that seems unfair
* moha_hunt has quit (Quit: moha_hunt)
* ganeshna has quit (Read error: Connection reset by peer)
<eglynn> cdent: life is unfair :)
<lakshmiS> can we not use different pool names: atleast that's what the oslo messaging document says
<cdent> you're basically putting a dependency on a source for their to be multiple listeners
<cdent> s/their/there/
<cdent> whereas what we'd like in a distributed system is for there to be N potential listeners without needing to change the source
* ivasev has quit (Read error: Connection reset by peer)
<gordc> lakshmiS: there was an issue with that oslo.messaging
* cdent looks at sileht 
* afazekas (~turul@68.152.broadband9.iol.cz) has joined
* pm90_ has quit (Read error: Connection reset by peer)
<lakshmiS> cdent: +1
<sileht> that should just work
<eglynn> cdent: yeap, it's certainly not pretty, I just think that's the current reality
* cdent does not aspire to reality ;)
* ivasev (~ivasev@secure-bastion.dfw1.rackspace.com) has joined
* jorge_munoz has quit (Read error: Connection reset by peer)
<eglynn> sileht: a-ha, k ... good to know
* pm90_ (~pm90_@of1-nat2.aus1.rackspace.com) has joined
<lakshmiS> I tried changing ceilometer listener code to have a pool name but it stops working afer a while
<lakshmiS> is there anyone from ceilometer team I can discuss this offline?
<gordc> sileht: wasn't there an issue with pools, something about it not being created at right time so we'd lose messages?
* jorge_munoz (~jorge@72.32.115.231) has joined
* ganeshna (~ganeshna@2001:420:c0e0:1005::2eb) has joined
* _nadya_ has quit (Remote host closed the connection)
<sileht> kmartin,you need to take a look to your rabbitmq queues and find those prefixed with searchlight-listener
<gordc> lakshmiS: we're all over at #openstack-ceilometer (more euro based) but there's a few of us (me) in americas tz
* _nadya_ (~nadya@bb-46-254-83-78.pppoe.fannet.ru) has joined
* vishwanathj (~vishwanat@144.49.132.22) has joined
* ujuc (~ujuc@211.219.170.176) has joined
* isviridov_away is now known as isviridov
<lakshmiS> gordc: sileht: can you point me to the issue you had previously with oslo.messaging pools?
<sileht> gordc, yes but you just lost message between the client side restart and the first server side start, that doesn't occurs offen
<sileht> lakshmiS, it not a issue, this is a historical behavior :p
<lakshmiS> ok
<sileht> unsure it's cleanly documented
<lakshmiS> so the solution/alternative is to have multiple topics for notification.info from glance?
<gordc> lakshmiS: post your issue over at #openstack-ceilometer and someone will reply. if not, we're sleeping and ML is good.
* Guest29403 has quit (Remote host closed the connection)
* vishwana_ has quit (Ping timeout: 255 seconds)
<lakshmiS> gordc: i wil do ML since I already did on IRC with no response
<gordc> lakshmiS: that is the alternative yes... that's typically what is done. (ie. stacktach)
<lakshmiS> gordc: thx
* arvinchou_1 (~Adium@61-222-64-29.HINET-IP.hinet.net) has joined
<gordc> sileht: for the above, do ceilometer  and searchlight need to create pool or just one? the problem right is we're both using hte same default pool right?
* isviridov is now known as isviridov_away
* _nadya_ has quit (Ping timeout: 256 seconds)
<sileht> gordc, I think the safer ways is to emit the notification twice on the other side
<lakshmiS> sileht: that would be a change on each team(glance, nova) etc right?
<sileht> lakshmiS, yes a configuration change
<sileht> gordc, this method ensure that no notifications will be missed
* ujuc has quit (Ping timeout: 264 seconds)
<gordc> sileht: got it. that's what i understood.
* PrashantS (~PerfBeing@50.56.228.68) has joined
<lakshmiS> sileht: do you have any previous reviews doint this cofig change ?
<lakshmiS> s/doint/doing
* hashar_ (~sempitern@mediawiki/hashar) has joined
<gordc> lakshmiS: its either or i guess. pro/con either way.
* shashank_hegde_ (~shashank_@c-98-234-177-153.hsd1.ca.comcast.net) has joined
* annegentle has quit (Remote host closed the connection)
* PrashantS has quit (Remote host closed the connection)
<gordc> lakshmiS: let me dig up stacktach docs, they might reference it
<lakshmiS> thanks gordc:
* PrashantS (~PerfBeing@50.56.229.9) has joined
* annegentle (~annegentl@50.57.17.244) has joined
* arvinchou_ has quit (Ping timeout: 256 seconds)
<gordc> anything other topics?
* adiantum has quit (Ping timeout: 256 seconds)
<gordc> lakshmiS: i'll send you link over in openstack-ceilometer unless you have another preference
<gordc> 30s countdown
* leeantho (~leeantho@swa01cs005-da02-z.ams.hpecore.net) has joined
* hashar has quit (Ping timeout: 240 seconds)
* vivek_ebay is now known as vivek_
<lakshmiS> gordc: please send it to lakshmi.sampath@hp.com if possbile too
<gordc> lakshmiS: kk
<gordc> ending meeting. thanks folks
<gordc> #endmeeting
* openstack has changed the topic to: OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings
<openstack> Meeting ended Thu Jul 16 15:55:28 2015 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
<openstack> Minutes:        http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.html
<openstack> Minutes (text): http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.txt
<openstack> Log:            http://eavesdrop.openstack.org/meetings/ceilometer/2015/ceilometer.2015-07-16-15.00.log.html