OK, let me wrap here a few findings.
First, the number of occurences is way lower than what I said in c#25 (fortunately).
Something around 19 occurrences around the last 20 days (which is still pretty big tho)
Secondly, I think I identified the real exception leading to a failed test condition :
2023-01-24 18:29:03.537384 | ubuntu-jammy | During handling of the above exception, another exception occurred:
2023-01-24 18:29:03.537389 | ubuntu-jammy |
2023-01-24 18:29:03.537394 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537400 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10619, in _live_migration
2023-01-24 18:29:03.537405 | ubuntu-jammy | self.live_migration_abort(instance)
2023-01-24 18:29:03.537411 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10087, in live_migration_abort
2023-01-24 18:29:03.537416 | ubuntu-jammy | guest = self._host.get_guest(instance)
2023-01-24 18:29:03.537422 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/host.py", line 673, in get_guest
2023-01-24 18:29:03.537427 | ubuntu-jammy | return libvirt_guest.Guest(self._get_domain(instance))
2023-01-24 18:29:03.537432 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/guest.py", line 112, in __init__
2023-01-24 18:29:03.537438 | ubuntu-jammy | libvirt = importutils.import_module('libvirt')
2023-01-24 18:29:03.537482 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_utils/importutils.py", line 73, in import_module
2023-01-24 18:29:03.537494 | ubuntu-jammy | __import__(import_str)
2023-01-24 18:29:03.537500 | ubuntu-jammy | File
2023-01-24 18:29:03.537505 | ubuntu-jammy | "/home/zuul/src/opendev.org/openstack/nova/nova/tests/fixtures/nova.py", line 1829, in find_spec
2023-01-24 18:29:03.537511 | ubuntu-jammy | raise ImportError(fullname)
2023-01-24 18:29:03.537516 | ubuntu-jammy | ImportError: libvirt
2023-01-24 18:29:03.537521 | ubuntu-jammy |
2023-01-24 18:29:03.537527 | ubuntu-jammy | During handling of the above exception, another exception occurred:
2023-01-24 18:29:03.537532 | ubuntu-jammy |
2023-01-24 18:29:03.537537 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537543 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 8854, in _do_live_migration
2023-01-24 18:29:03.537548 | ubuntu-jammy | self.driver.live_migration(context, instance, dest,
2023-01-24 18:29:03.537553 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10076, in live_migration
2023-01-24 18:29:03.537559 | ubuntu-jammy | self._live_migration(context, instance, dest,
2023-01-24 18:29:03.537564 | ubuntu-jammy | File "/home/zuul/src/opendev.org/openstack/nova/nova/virt/libvirt/driver.py", line 10620, in _live_migration
2023-01-24 18:29:03.537569 | ubuntu-jammy | except libvirt.libvirtError:
2023-01-24 18:29:03.537575 | ubuntu-jammy | AttributeError: 'NoneType' object has no attribute 'libvirtError'
Corresponding logsearch query :
logsearch log --job-group nova-functional --result FAILURE "ImportError: This test imports the 'libvirt' module, which it should not in the test environment. Please add appropriate mocking to this test." --days 20
OK, let me wrap here a few findings.
First, the number of occurences is way lower than what I said in c#25 (fortunately).
Something around 19 occurrences around the last 20 days (which is still pretty big tho)
Secondly, I think I identified the real exception leading to a failed test condition :
2023-01-24 18:29:03.537384 | ubuntu-jammy | During handling of the above exception, another exception occurred: zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ driver. py", line 10619, in _live_migration migration_ abort(instance) zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ driver. py", line 10087, in live_migration_ abort get_guest( instance) zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ host.py" , line 673, in get_guest guest.Guest( self._get_ domain( instance) ) zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ guest.py" , line 112, in __init__ import_ module( 'libvirt' ) zuul/src/ opendev. org/openstack/ nova/.tox/ functional- py310/lib/ python3. 10/site- packages/ oslo_utils/ importutils. py", line 73, in import_module _(import_ str) zuul/src/ opendev. org/openstack/ nova/nova/ tests/fixtures/ nova.py" , line 1829, in find_spec fullname) zuul/src/ opendev. org/openstack/ nova/nova/ compute/ manager. py", line 8854, in _do_live_migration live_migration( context, instance, dest, zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ driver. py", line 10076, in live_migration migration( context, instance, dest, zuul/src/ opendev. org/openstack/ nova/nova/ virt/libvirt/ driver. py", line 10620, in _live_migration libvirtError:
2023-01-24 18:29:03.537389 | ubuntu-jammy |
2023-01-24 18:29:03.537394 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537400 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537405 | ubuntu-jammy | self.live_
2023-01-24 18:29:03.537411 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537416 | ubuntu-jammy | guest = self._host.
2023-01-24 18:29:03.537422 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537427 | ubuntu-jammy | return libvirt_
2023-01-24 18:29:03.537432 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537438 | ubuntu-jammy | libvirt = importutils.
2023-01-24 18:29:03.537482 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537494 | ubuntu-jammy | __import_
2023-01-24 18:29:03.537500 | ubuntu-jammy | File
2023-01-24 18:29:03.537505 | ubuntu-jammy | "/home/
2023-01-24 18:29:03.537511 | ubuntu-jammy | raise ImportError(
2023-01-24 18:29:03.537516 | ubuntu-jammy | ImportError: libvirt
2023-01-24 18:29:03.537521 | ubuntu-jammy |
2023-01-24 18:29:03.537527 | ubuntu-jammy | During handling of the above exception, another exception occurred:
2023-01-24 18:29:03.537532 | ubuntu-jammy |
2023-01-24 18:29:03.537537 | ubuntu-jammy | Traceback (most recent call last):
2023-01-24 18:29:03.537543 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537548 | ubuntu-jammy | self.driver.
2023-01-24 18:29:03.537553 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537559 | ubuntu-jammy | self._live_
2023-01-24 18:29:03.537564 | ubuntu-jammy | File "/home/
2023-01-24 18:29:03.537569 | ubuntu-jammy | except libvirt.
2023-01-24 18:29:03.537575 | ubuntu-jammy | AttributeError: 'NoneType' object has no attribute 'libvirtError'
There are false positives like https:/ /storage. gra.cloud. ovh.net/ v1/AUTH_ dcaab5e32b234d5 6b626f72581e364 4c/zuul_ opendev_ logs_aea/ 850501/ 15/check/ nova-tox- functional- py38/aea02af/ job-output. txt that show that most of the times the eventlet threads are missing the DB, but the overall test isn't failing, unless a specific thread above is running *after* we cleaned up the poisoned libvirt object.
Corresponding logsearch query :
logsearch log --job-group nova-functional --result FAILURE "ImportError: This test imports the 'libvirt' module, which it should not in the test environment. Please add appropriate mocking to this test." --days 20
Builds with matching logs 19/143:
+-- ------- ------- ------- ------- ----+-- ------- ------- -----+- ------- ------- -+----- -----+- ------- ------- ------- ------- ------+ ------- -+----- ------- ------- ------- -+
| uuid | finished | project | pipeline | review | branch | job |
+-- ------- ------- ------- ------- ----+-- ------- ------- -----+- ------- ------- -+----- -----+- ------- ------- ------- ------- ------+ ------- -+----- ------- ------- ------- -+
| 9f2e63e091e74ba 99151a5c8e30890 d1 | 2023-02-07T09:21:45 | openstack/nova | check | https:/ /review. opendev. org/870938 | master | nova-tox- functional- py39 |
| 5a44ba30aec545d e867f2a81c2fb35 b2 | 2023-02-07T08:51:52 | openstack/nova | check | https:/ /review. opendev. org/870932 | master | nova-tox- functional- py38 |
| d00d1ffaa9b34af ba264c32b0bdf6a 7c | 2023-02-06T16:24:57 | openstack/nova | check | https:/ /review. opendev. org/868237 | master | nova-tox- functional- py38 |
| 679799857eee48e 799dd3fdc80712c f0 | 2023-02-03T23:18:23 | openstack/nova | gate | https:/ /review. opendev. org/872432 | master | nova-tox- functional- py38 |
| 2f8f21db817f448 8a24cbc72f8922a 6c | 2023-02-02T18:53:13 | openstack/nova | check | https:/ /review. opendev. org/860287 | master | nova-tox- functional- py38 |
| 3888abbc3b44491 6875966eda45abe 95 | 2023-02-02T12:50:31 | openstack/nova | check | https:/ /review. opendev. org/860288 | master | nova-tox- functional- py310 |
| 003e39baf11248c ca9d4ad857228d7 76 | 2023-02-02T10:36:21 | openstack/nova | check | https:/ /review. opendev. org/872413 | master | nova-tox- functional- py310 |
| efd50b6ce8c7440 89ba3023408449c c1 | 2023-01-31T20:36:40 | openstack/nova | check | https:/ /review. opendev. org/872348 | master | nova-tox- functional- py310 |
| 27395778c2da42f bbef27fb4d1ae17 5d | 2023-01-30T19:22:13 | openstack/nova | check | https:/ /review. opendev. org/863920 | master | nova-tox- functional- py310 |
| 8025cac32927455 e826448d5165800 d2 | 2023-01-25T12:42:42 | openstack/nova | check | https:/ /review. opendev. org/854824 | master | nova-tox- functional- py39 |
| 92daaa41e4404ef dac9186d10dabc2 e0 | 2023-01-25T11:44:18 | openstack/nova | check | https:/ /review. opendev. org/833090 | master | nova-tox- functional- py38 |
| 466e0d714c6d4d7 2b566cdf8fe21d6 50 | 2023-01-24T18:38:42 | openstack/nova | check | https:/ /review. opendev. org/860282 | master | nova-tox- functional- py310 |
| 5e37af39cbeb40f 7b50977d35a96e2 e9 | 2023-01-24T15:27:31 | openstack/nova | check | https:/ /review. opendev. org/871612 | master | nova-tox- functional- py38 |
| 4b02df6e2a92485 194e47859ea9248 0c | 2023-01-23T15:27:17 | openstack/nova | check | https:/ /review. opendev. org/869950 | master | nova-tox- functional- py38 |
| ef9ef691ad2f43b 3bc40800d95c8ee fa | 2023-01-20T23:00:13 | openstack/nova | check | https:/ /review. opendev. org/863920 | master | nova-tox- functional- py310 |
| d926c0602cdf4f9 4a67ef342b63fc8 1c | 2023-01-19T15:20:11 | openstack/nova | check | https:/ /review. opendev. org/867324 | master | nova-tox- functional- py39 |
| 889b15a8573c4ae d9d93c71e5e22be f7 | 2023-01-19T13:43:52 | openstack/nova | check | https:/ /review. opendev. org/870932 | master | nova-tox- functional- py310 |
| fb32a7becc0043e 283cc00c5504ebc 9b | 2023-01-19T13:33:02 | openstack/nova | check | https:/ /review. opendev. org/870933 | master | nova-tox- functional- py310 |
| c763a465e27e4b8 4bcdc7211fa8d61 d3 | 2023-01-19T13:12:54 | openstack/nova | check | https:/ /review. opendev. org/870932 | master | nova-tox- functional- py39 |
+-- ------- ------- ------- ------- ----+-- ------- ------- -----+- ------- ------- -+----- -----+- ------- ------- ------- ------- ------+ ------- -+----- ------- ------- ------- -+