tests: Use GreenThreadPoolExecutor.shutdown(wait=True)
We are still having some issues in the gate where greenlets from
previous tests continue to run while the next test starts, causing
false negative failures in unit or functional test jobs.
This adds a new fixture that will ensure
GreenThreadPoolExecutor.shutdown() is called with wait=True, to wait
for greenlets in the pool to finish running before moving on.
In local testing, doing this does not appear to adversely affect test
run times, which was my primary concern.
As a baseline, I ran a subset of functional tests in a loop
until failure without the patch and after 11 hours, I got a failure
reproducing the bug. With the patch, running the same subset of
functional tests in a loop has been running for 24 hours and has not
failed yet.
Based on this, I think it may be worth trying this out to see if it
will help stability of our unit and functional test jobs. And if it
ends up impacting test run times or causes other issues, we can
revert it.
Reviewed: https:/ /review. opendev. org/c/openstack /nova/+ /883317 /opendev. org/openstack/ nova/commit/ c095cfe04e2c71e fcfbfdd95948af0 80a98065e6
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit c095cfe04e2c71e fcfbfdd95948af0 80a98065e6
Author: melanie witt <email address hidden>
Date: Thu May 11 19:23:52 2023 +0000
tests: Use GreenThreadPool Executor. shutdown( wait=True)
We are still having some issues in the gate where greenlets from
previous tests continue to run while the next test starts, causing
false negative failures in unit or functional test jobs.
This adds a new fixture that will ensure PoolExecutor. shutdown( ) is called with wait=True, to wait
GreenThread
for greenlets in the pool to finish running before moving on.
In local testing, doing this does not appear to adversely affect test
run times, which was my primary concern.
As a baseline, I ran a subset of functional tests in a loop
until failure without the patch and after 11 hours, I got a failure
reproducing the bug. With the patch, running the same subset of
functional tests in a loop has been running for 24 hours and has not
failed yet.
Based on this, I think it may be worth trying this out to see if it
will help stability of our unit and functional test jobs. And if it
ends up impacting test run times or causes other issues, we can
revert it.
Partial-Bug: #1946339
Change-Id: Ia916310522b007 061660172fa4d63 d0fde9a55ac