Ceilometer

tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate failed for timeout

Bug #1323524 reported by Liusheng on 2014-05-27

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	Ceilometer	Fix Released	Medium	Mehdi Abaakouk	Ceilometer 2014.2 "juno"

Bug Description

http://logs.openstack.org/02/93702/2/check/gate-ceilometer-python27/e8ea708/console.html

Traceback (most recent call last):
  File "/home/jenkins/workspace/gate-ceilometer-python27/ceilometer/tests/alarm/test_rpc.py", line 235, in test_ordination_allocate
    self.coordinator_server.rpc.wait()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/oslo/messaging/server.py", line 143, in wait
    self._executor.wait()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/oslo/messaging/_executors/impl_eventlet.py", line 106, in wait
    self._thread.wait()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
    return self._exit_event.wait()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 236, in run
    self.wait(sleep_time)
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 81, in wait
    sleep(seconds)
  File "/home/jenkins/workspace/gate-ceilometer-python27/.tox/py27/local/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
TimeoutException

Revision history for this message

ZhiQiang Fan (aji-zqfan) wrote on 2014-05-30:

another one: http://logs.openstack.org/69/83369/2/check/gate-ceilometer-python27/4e2715c/

Changed in ceilometer:
status:	New → Confirmed

Eoghan Glynn (eglynn) on 2014-06-01

Changed in ceilometer:
importance:	Undecided → Medium
assignee:	nobody → Eoghan Glynn (eglynn)
milestone:	none → juno-1

Revision history for this message

Mehdi Abaakouk (sileht) wrote on 2014-06-05:

I have introduced the bug when I have updated the testr configuration

I think we can safely removed or increased the 'OS_TEST_TIMEOUT' in .testr.conf

Revision history for this message

ZhiQiang Fan (aji-zqfan) wrote on 2014-06-05:

there may be a further question, why 160s is not enough for our test?

Revision history for this message

Anita Kuno (anteaya) wrote on 2014-06-05:

message:"FAIL: tests.alarm.test_rpc.TestRPCAlarmPartitionCoordination.test_ordination_allocate"

https://review.openstack.org/#/c/98146/2/queries/1323524.yaml

Revision history for this message

Matt Riedemann (mriedem) wrote on 2014-06-05:

This is the change referred to in comment 2:

https://review.openstack.org/#/c/95199/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-09: Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/98820

Changed in ceilometer:
assignee:	Eoghan Glynn (eglynn) → Mehdi Abaakouk (sileht)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-06-10: Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/98820
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=f35a9a6b8c4158c7b060dcb9b3ced4367c46977d
Submitter: Jenkins
Branch: master

commit f35a9a6b8c4158c7b060dcb9b3ced4367c46977d
Author: Mehdi Abaakouk <email address hidden>
Date: Mon Jun 9 17:35:18 2014 +0200

Removes per test testr timeout

    Because testr run tests in parallel and we use fake driver of
    oslo.messaging to make unittests and this one use time.sleep() monkey
    patched by eventlet, a test can take a lot of time, just because
    eventlet have decided to run other tests before instead of finish the
    started one.

So, just remove the per test testr timeout, like we have before.

Change-Id: I26f14937a16b23428f55fc7f42b4066b398c06a8
Closes-bug: #1323524

Changed in ceilometer:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-06-12

Changed in ceilometer:
status:	Fix Committed → Fix Released

Revision history for this message

ZhiQiang Fan (aji-zqfan) wrote on 2014-06-23:

after patch: Reenables the testr per test timeout Ifccef9040529ec1a0b21076ed17d91a4f9737096 is merged, this problem raises again, see http://logs.openstack.org/58/101558/1/check/gate-ceilometer-python27/de6a11d/testr_results.html.gz

Revision history for this message

Mehdi Abaakouk (sileht) wrote on 2014-06-23:

@ZhiQiang Fan: yes this is the desired behavior, all ceilometer tests must run in less then 10 minutes if it take more, we have a issue.

We fixe many ceilometer unit tests timeout issues last week , the per timeout was to help to track which part of unit tests have issue instead of just reach the gate timeout than hide which test is broken.

One remaining issue is https://bugs.launchpad.net/ceilometer/+bug/1331453, you can use this one to recheck bug XXX.

Revision history for this message

Boris Pavlovic (boris-42) wrote on 2014-06-30:

#10

This should be re opened

Revision history for this message

Boris Pavlovic (boris-42) wrote on 2014-06-30:

#11

https://review.openstack.org/#/c/100239/ This patch failed with the similar reason

Thierry Carrez (ttx) on 2014-10-16

Changed in ceilometer:
milestone:	juno-1 → 2014.2

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.