tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing

It looks like it is some issue with tempest test probably.
For now it looks like it detaches interface and then "too fast" tries to get metadata on instance. It's in tempest/api/compute/servers/test_device_tagging.py

Maybe some waif for detached state first could do the job? I need to test it.

Hi, yes i have tried this and got some test case failed in the tempest side. https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_device_tagging.py#L350

Related fix proposed to branch: master
Review: https://review.openstack.org/578765

After some analysis of logs from failed run it looks for me that this "exit code 137" is because test reached timeout and was just cleaned.
I think that in verify_empty_devices() method there is still some device existing and because f that this test is not finishing.
I don't know yet if that is volume or nic device which wasn't removed from this metadata properly but will investigate...

In verify_empty_devices(), we can check the non presence of specified tags ['nic-tag', 'volume-tag'] instead of no device so that if any other devices is there we do not keep trying in wait loop.

@slaweq This test is now passed on my environment. Can you please check the same on your environment or Is it still reproducable?

python -m testtools.run tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment
Tests running...

Ran 1 test in 61.800s
OK

I see this still happening in the Neutron check jobs. Here's an example:

http://logs.openstack.org/39/577739/2/check/neutron-tempest-linuxbridge/825feb8/job-output.txt.gz#_2018-07-10_03_06_34_759658

Reviewed: https://review.openstack.org/578765
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Submitter: Zuul
Branch: master

commit eb70a96ee3f7e56dbf57aefa2a1aa768aeb559ab
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jun 28 13:03:04 2018 +0200

    Add additional log of meta_data devices content

    In TaggedAttachmentsTest.test_tagged_attachment test there is
    now added some additional debug log to check what
    devices are still in meta_data's devices field when it should
    be empty already.

    This patch changes also from handling all exeptions to only
    AssertionError in verify_empty_devices() method.
    If exeption other than AssertionError occur, it will not be silently
    ignored anymore.

    Change-Id: I95016e35102569fba79fc58f9695e480234619dc
    Related-Bug: #1775947

I just spotted this issue again and it looks that volume device is not removed from this meta_data.json file:
http://logs.openstack.org/69/567369/15/check/tempest-full/528bc75/job-output.txt.gz#_2018-07-19_10_06_09_273919

Related fix proposed to branch: master
Review: https://review.openstack.org/586292

Reviewed: https://review.openstack.org/586292
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Submitter: Zuul
Branch: master

commit 4e2a957bf8aade0a766c9c9a1d6798e4cb71e86d
Author: Matt Riedemann <email address hidden>
Date: Thu Jul 26 12:28:48 2018 -0400

    Skip verifying empty devices in test_tagged_attachment until bug is fixed

    We have a non-trivial amount of failures due to bug 1775947 [1]
    and so far attempts at debugging the issue haven't proved
    fruitful, so let's skip the last (problematic) portion of the
    test until we can get a recreate with proper debug to help
    isolate the problem.

    Related nova debug patch: https://review.openstack.org/584032

    [1] http://status.openstack.org/elastic-recheck/#1775947

    Change-Id: I3104041cb66f912727aabd6c5894600dc87a1229
    Related-Bug: #1775947

Related fix proposed to branch: master
Review: https://review.opendev.org/742444

What's the status here? I see several fixes were merged, is the issue still reproducible? If so, please, share new info (tracebacks, steps to reproduce, links to failed jobs etc) and let's discuss what next, until then I'll mark is as Incomplete.

[Expired for tempest because there has been no activity for 60 days.]