TestMinimumBasicScenario.test_minimum_basic_scenario race fail comparing volume to expected values with updated_at diff

Bug #1838202 reported by Matt Riedemann
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Low
Matt Riedemann

Bug Description

Seen here:

http://logs.openstack.org/51/643451/28/check/nova-next/17c3af7/job-output.txt.gz#_2019-07-27_01_20_36_245189

Looks like the test isn't filtering out updated_at from expected values but probably should to avoid race failures.

2019-07-27 01:20:36.245189 | controller | {1} tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario [21.310866s] ... FAILED
2019-07-27 01:20:36.245278 | controller |
2019-07-27 01:20:36.245325 | controller | Captured traceback:
2019-07-27 01:20:36.245362 | controller | ~~~~~~~~~~~~~~~~~~~
2019-07-27 01:20:36.245428 | controller | b'Traceback (most recent call last):'
2019-07-27 01:20:36.245526 | controller | b' File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 89, in wrapper'
2019-07-27 01:20:36.245611 | controller | b' return f(*func_args, **func_kwargs)'
2019-07-27 01:20:36.245740 | controller | b' File "/opt/stack/tempest/tempest/scenario/test_minimum_basic.py", line 118, in test_minimum_basic_scenario'
2019-07-27 01:20:36.245803 | controller | b' self.cinder_show(volume)'
2019-07-27 01:20:36.246358 | controller | b' File "/opt/stack/tempest/tempest/scenario/test_minimum_basic.py", line 64, in cinder_show'
2019-07-27 01:20:36.246456 | controller | b' self.assertEqual(volume, got_volume)'
2019-07-27 01:20:36.246605 | controller | b' File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/testtools/testcase.py", line 411, in assertEqual'
2019-07-27 01:20:36.247445 | controller | b' self.assertThat(observed, matcher, message)'
2019-07-27 01:20:36.247601 | controller | b' File "/opt/stack/tempest/.tox/tempest/lib/python3.6/site-packages/testtools/testcase.py", line 498, in assertThat'
2019-07-27 01:20:36.247648 | controller | b' raise mismatch_error'
2019-07-27 01:20:36.247704 | controller | b'testtools.matchers._impl.MismatchError: !=:'
2019-07-27 01:20:36.247749 | controller | b"reference = {'attachments': [],"
2019-07-27 01:20:36.248037 | controller | b" 'availability_zone': 'nova',"
2019-07-27 01:20:36.248913 | controller | b" 'bootable': 'false',"
2019-07-27 01:20:36.249002 | controller | b" 'consistencygroup_id': None,"
2019-07-27 01:20:36.249079 | controller | b" 'created_at': '2019-07-27T01:20:28.000000',"
2019-07-27 01:20:36.249134 | controller | b" 'description': None,"
2019-07-27 01:20:36.249184 | controller | b" 'encrypted': False,"
2019-07-27 01:20:36.249275 | controller | b" 'id': '12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.249432 | controller | b" 'links': [{'href': 'https://149.202.184.221/volume/v3/862bd3e1e8b34dfda2413be1ed10ff10/volumes/12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.249510 | controller | b" 'rel': 'self'},"
2019-07-27 01:20:36.249659 | controller | b" {'href': 'https://149.202.184.221/volume/862bd3e1e8b34dfda2413be1ed10ff10/volumes/12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.249869 | controller | b" 'rel': 'bookmark'}],"
2019-07-27 01:20:36.249924 | controller | b" 'metadata': {},"
2019-07-27 01:20:36.249975 | controller | b" 'multiattach': False,"
2019-07-27 01:20:36.250087 | controller | b" 'name': 'tempest-TestMinimumBasicScenario-volume-1950148737',"
2019-07-27 01:20:36.250193 | controller | b" 'os-vol-tenant-attr:tenant_id': '862bd3e1e8b34dfda2413be1ed10ff10',"
2019-07-27 01:20:36.250253 | controller | b" 'replication_status': None,"
2019-07-27 01:20:36.251473 | controller | b" 'size': 1,"
2019-07-27 01:20:36.251540 | controller | b" 'snapshot_id': None,"
2019-07-27 01:20:36.251591 | controller | b" 'source_volid': None,"
2019-07-27 01:20:36.251643 | controller | b" 'status': 'available',"
2019-07-27 01:20:36.251717 | controller | b" 'updated_at': '2019-07-27T01:20:29.000000',"
2019-07-27 01:20:36.251798 | controller | b" 'user_id': '16e9467e3ed74b039c20d6103d69f9c5',"
2019-07-27 01:20:36.251851 | controller | b" 'volume_type': 'lvmdriver-1'}"
2019-07-27 01:20:36.251894 | controller | b"actual = {'attachments': [],"
2019-07-27 01:20:36.251928 | controller | b" 'availability_zone': 'nova',"
2019-07-27 01:20:36.251954 | controller | b" 'bootable': 'false',"
2019-07-27 01:20:36.251986 | controller | b" 'consistencygroup_id': None,"
2019-07-27 01:20:36.252026 | controller | b" 'created_at': '2019-07-27T01:20:28.000000',"
2019-07-27 01:20:36.252052 | controller | b" 'description': None,"
2019-07-27 01:20:36.252078 | controller | b" 'encrypted': False,"
2019-07-27 01:20:36.252119 | controller | b" 'id': '12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.252214 | controller | b" 'links': [{'href': 'https://149.202.184.221/volume/v3/862bd3e1e8b34dfda2413be1ed10ff10/volumes/12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.252250 | controller | b" 'rel': 'self'},"
2019-07-27 01:20:36.252352 | controller | b" {'href': 'https://149.202.184.221/volume/862bd3e1e8b34dfda2413be1ed10ff10/volumes/12d211dd-48b7-45d9-9154-75470931e85b',"
2019-07-27 01:20:36.252387 | controller | b" 'rel': 'bookmark'}],"
2019-07-27 01:20:36.252442 | controller | b" 'metadata': {},"
2019-07-27 01:20:36.253044 | controller | b" 'multiattach': False,"
2019-07-27 01:20:36.253127 | controller | b" 'name': 'tempest-TestMinimumBasicScenario-volume-1950148737',"
2019-07-27 01:20:36.253216 | controller | b" 'os-vol-tenant-attr:tenant_id': '862bd3e1e8b34dfda2413be1ed10ff10',"
2019-07-27 01:20:36.253280 | controller | b" 'replication_status': None,"
2019-07-27 01:20:36.253324 | controller | b" 'size': 1,"
2019-07-27 01:20:36.253373 | controller | b" 'snapshot_id': None,"
2019-07-27 01:20:36.253420 | controller | b" 'source_volid': None,"
2019-07-27 01:20:36.253470 | controller | b" 'status': 'available',"
2019-07-27 01:20:36.253542 | controller | b" 'updated_at': '2019-07-27T01:20:30.000000',"
2019-07-27 01:20:36.253611 | controller | b" 'user_id': '16e9467e3ed74b039c20d6103d69f9c5',"
2019-07-27 01:20:36.253672 | controller | b" 'volume_type': 'lvmdriver-1'}"
2019-07-27 01:20:36.253706 | controller | b''
2019-07-27 01:20:36.254049 | controller | b''

Matt Riedemann (mriedem)
Changed in tempest:
status: New → Confirmed
tags: added: cinder gate-failure volume
Revision history for this message
Matt Riedemann (mriedem) wrote :

It looks like the dict keys are at least sorted so I can do this for a logstash query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22actual%20%20%20%20%3D%20%7B'attachments'%3A%20%5B%5D%2C%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

That shows it's pretty rare (2 hits in 7 days).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tempest (master)

Fix proposed to branch: master
Review: https://review.opendev.org/688737

Changed in tempest:
assignee: nobody → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in tempest:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.opendev.org/688737
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=3ec41c20d67a8f4a861161fa85494709a4503854
Submitter: Zuul
Branch: master

commit 3ec41c20d67a8f4a861161fa85494709a4503854
Author: Matt Riedemann <email address hidden>
Date: Tue Oct 15 11:39:32 2019 -0400

    Relax volume compare in test_minimum_basic_scenario

    Similar to I50d1319b690453923b470733e94f3a11fd1cd249 for
    the server that was just created, this change relaxes the
    volume compare check on the updated_at field since we can
    race between the time the volume was created and when comparing
    to the results of GETing the volume that updated_at could have
    changed.

    Change-Id: Ie666b9ea0e46cf3ee22d07709e5faebafde35325
    Closes-Bug: #1838202

Changed in tempest:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.