test_list_servers_filter_by_error_status intermittently fails with MismatchError on no servers in response

Seen here:

http://logs.openstack.org/59/424759/12/gate/gate-tempest-dsvm-cells-ubuntu-xenial/d7b1311/console.html#_2017-01-31_17_48_34_663273

2017-01-31 17:48:34.663337 | Captured traceback:
2017-01-31 17:48:34.663348 | ~~~~~~~~~~~~~~~~~~~
2017-01-31 17:48:34.663363 | Traceback (most recent call last):
2017-01-31 17:48:34.663393 | File "tempest/api/compute/admin/test_servers.py", line 59, in test_list_servers_filter_by_error_status
2017-01-31 17:48:34.663414 | self.assertIn(self.s1_id, map(lambda x: x['id'], servers))
2017-01-31 17:48:34.663448 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
2017-01-31 17:48:34.663468 | self.assertThat(haystack, Contains(needle), message)
2017-01-31 17:48:34.663502 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
2017-01-31 17:48:34.663515 | raise mismatch_error
2017-01-31 17:48:34.663542 | testtools.matchers._impl.MismatchError: u'108b4797-74fd-4a00-912a-b7fe0e142888' not in []

This test resets the state on a server to ERROR:

2017-01-31 17:48:34.663649 | 2017-01-31 17:28:39,375 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 202 POST http://10.23.154.32:8774/v2.1/servers/108b4797-74fd-4a00-912a-b7fe0e142888/action 0.142s
2017-01-31 17:48:34.663695 | 2017-01-31 17:28:39,376 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-01-31 17:48:34.663714 | Body: {"os-resetState": {"state": "error"}}

Then tries to list servers by that status and expects to get that one back:

2017-01-31 17:48:34.663883 | 2017-01-31 17:28:39,556 504 INFO [tempest.lib.common.rest_client] Request (ServersAdminTestJSON:test_list_servers_filter_by_error_status): 200 GET http://10.23.154.32:8774/v2.1/servers?status=error 0.179s
2017-01-31 17:48:34.663955 | 2017-01-31 17:28:39,556 504 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>', 'Accept': 'application/json', 'Content-Type': 'application/json'}
2017-01-31 17:48:34.663969 | Body: None
2017-01-31 17:48:34.664078 | Response - Headers: {u'x-openstack-nova-api-version': '2.1', u'vary': 'X-OpenStack-Nova-API-Version', u'content-length': '15', 'status': '200', u'content-type': 'application/json', u'x-compute-request-id': 'req-91ef16ab-28c3-47c5-b823-6a321bde5c01', u'date': 'Tue, 31 Jan 2017 17:28:39 GMT', 'content-location': 'http://10.23.154.32:8774/v2.1/servers?status=error', u'openstack-api-version': 'compute 2.1', u'connection': 'close'}
2017-01-31 17:48:34.664094 | Body: {"servers": []}

And the list is coming back empty, intermittently, with cells v1. So there is probably some vm_state change race between the state change in the child cell and reporting that back up to the parent API cell.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Body%3A%20%7B%5C%5C%5C%22servers%5C%5C%5C%22%3A%20%5B%5D%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22gate-tempest-dsvm-cells-ubuntu-xenial%5C%22&from=7d

16 hits in 7 days, check and gate, all failures.