Ceilometer

recent build timeout failure in gate

Bug #1328679 reported by gordon chung
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Critical
Mehdi Abaakouk
Ceilometer 2014.2 "juno"

Bug Description

seeing this pop up quite a bit on python 2.6 and 2.7 gates:

2014-06-10 15:06:17.668 | Error in atexit._run_exitfuncs:
2014-06-10 15:06:17.669 | Traceback (most recent call last):
2014-06-10 15:06:17.669 | File "/usr/lib64/python2.6/atexit.py", line 24, in _run_exitfuncs
2014-06-10 15:06:17.669 | func(*targs, **kargs)
2014-06-10 15:06:17.707 | File "/usr/lib64/python2.6/logging/__init__.py", line 1516, in shutdown
2014-06-10 15:06:17.708 | h.flush()
2014-06-10 15:06:17.709 | File "/usr/lib64/python2.6/logging/__init__.py", line 762, in flush
2014-06-10 15:06:17.709 | self.stream.flush()
2014-06-10 15:06:17.709 | File "/usr/lib64/python2.6/StringIO.py", line 256, in flush
2014-06-10 15:06:17.710 | _complain_ifclosed(self.closed)
2014-06-10 15:06:17.710 | File "/usr/lib64/python2.6/StringIO.py", line 40, in _complain_ifclosed
2014-06-10 15:06:17.710 | raise ValueError, "I/O operation on closed file"
2014-06-10 15:06:17.710 | ValueError: I/O operation on closed file
2014-06-10 15:06:17.710 | Error in sys.exitfunc:
2014-06-10 16:10:16.496 | Build timed out (after 70 minutes). Marking the build as failed.
2014-06-10 16:10:18.924 | ++ rm -rf /tmp/tmp.8k9zVRxU4C
2014-06-10 16:10:18.924 | ++ clean_exit
2014-06-10 16:10:18.924 | ++ local error_code=0
2014-06-10 16:10:18.925 | ++ rm -rf /tmp/CEILO-MONGODB-JH559

ref: http://logs.openstack.org/85/93585/6/check/gate-ceilometer-python26/ad11f5e/

See original description
gordon chung (chungg)
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Lianhao Lu (lianhao-lu) wrote :

looks like we hit this again http://logs.openstack.org/75/98075/2/gate/gate-ceilometer-python26/97ea8b6/console.html

Mehdi Abaakouk (sileht)
Changed in ceilometer:
assignee: nobody → Mehdi Abaakouk (sileht)
Revision history for this message
Lianhao Lu (lianhao-lu) wrote :

Well, I don't think this is related to the case of tests.test_notification.TestRealNotification.test_notification_service, I just found another failure http://logs.openstack.org/88/97588/3/gate/gate-ceilometer-python26/3291e40/ in which that case had been marked successful.

Revision history for this message
Lianhao Lu (lianhao-lu) wrote :

Besides, I also found many may gate tests marked as 'green' have the similar console output, please search "ValueError: I/O operation on closed file" at http://logs.openstack.org/06/88506/54/check/gate-ceilometer-python26/9133fb0/

Lianhao Lu (lianhao-lu)
description: updated
Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

IMHO, ValueError: I/O operation on closed file is something different with build time out, the build time out happens often recently and will cause gate fail, but the ValueError exists for a long time, and will not cause gate fail

Revision history for this message
Lianhao Lu (lianhao-lu) wrote :

agreed, should have another bug track "ValueError: I/O operation", which seems first appeared in https://review.openstack.org/#/c/71688/. Here we need to fight against build time out

summary: - string io errors in unit test
+ recent build timeout failure in gate-py26
summary: - recent build timeout failure in gate-py26
+ recent build timeout failure in gate
Revision history for this message
Lianhao Lu (lianhao-lu) wrote :

https://bugs.launchpad.net/ceilometer/+bug/1329692 is created to track the "ValueError: I/O operation" error

Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

I noticed a stuck in **local tox** with up-to-date master, after wait for 15 minutes, finally I have to press CTRL+C to stop it, see stack trace paste:
http://paste.openstack.org/show/84018/

Revision history for this message
gordon chung (chungg) wrote :

ER query patch here: https://review.openstack.org/#/c/100405/

tags: added: gate-failure testing

Revision history for this message
Joe Gordon (jogo) wrote :

being this is a pretty big issue with the unit tests (42 hits in 48 hours), I think the importance should be higher then medium

Revision history for this message
Eoghan Glynn (eglynn) wrote :

Addressed already by:

  https://review.openstack.org/99891
  https://review.openstack.org/100453

just missing the Closes-Bug tag in the commit msg, so the state of this bug wasn't auto-updated.

Changed in ceilometer:
milestone: none → juno-2
status: Triaged → Fix Committed
importance: Medium → Critical
Revision history for this message
gordon chung (chungg) wrote :

i'm going to reopen this temporarily. just saw it again: https://review.openstack.org/#/c/100297/

Changed in ceilometer:
status: Fix Committed → Triaged
Revision history for this message
Mark McLoughlin (markmc) wrote :

The e-r query for this doesn't match anything in logstash now, probably should mark as Fix Committed again

I see matches for 'ValueError: I/O operation on closed file' only on stable/icehouse, but that suggests a different problem?

Revision history for this message
gordon chung (chungg) wrote :

works for me.

the i/o error is related to: https://bugs.launchpad.net/ceilometer/+bug/1329692

i've added icehouse as affected

Changed in ceilometer:
status: Triaged → Fix Committed
Russell Bryant (russellb)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: juno-2 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.