While debugging the LiveMigrationTest.test_iscsi_volume tempest test, I am seeing this in my local env as one of the causes of intermittent failures with this test. I can get this to fail maybe 2-3x a day. This is what I run to provoke it:
$ tempest run --regex test_live_migration --concurrency=4
I haven't seen it fail on a single threaded test, but it's hard to say since it fails so infrequently.
The failure in n-cpu.log is:
2017-09-20 14:01:50.192 ^[[01;31mERROR oslo_messaging.rpc.server [^[[01;36mreq-6db9664a-4c6e-4d47-bffc-224b45b34eb0 ^[[00;36mtempest-LiveMigrationTest-1489892041 tempest-LiveMigrationTest-1489892041^[[01;31m] ^[[01;35m^[[01;31mException during message handling^[[00m: DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.
The tempest error is:
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 3194b6d9-cb70-4e67-8c28-ebf9ea094297 failed to reach ['available'] status (current in-use) within the required time (196 s).
While debugging the LiveMigrationTe st.test_ iscsi_volume tempest test, I am seeing this in my local env as one of the causes of intermittent failures with this test. I can get this to fail maybe 2-3x a day. This is what I run to provoke it:
$ tempest run --regex test_live_migration --concurrency=4
I haven't seen it fail on a single threaded test, but it's hard to say since it fails so infrequently.
The failure in n-cpu.log is:
2017-09-20 14:01:50.192 ^[[01;31mERROR oslo_messaging. rpc.server [^[[01; 36mreq- 6db9664a- 4c6e-4d47- bffc-224b45b34e b0 ^[[00;36mtempes t-LiveMigration Test-1489892041 tempest- LiveMigrationTe st-1489892041^ [[01;31m] ^[[01;35m^ [[01;31mExcepti on during message handling^[[00m: DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.
The tempest error is:
tempest. lib.exceptions. TimeoutExceptio n: Request timed out cb70-4e67- 8c28-ebf9ea0942 97 failed to reach ['available'] status (current in-use) within the required time (196 s).
Details: volume 3194b6d9-
I am adding more debug code to help debug this.