test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"

http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f218227/console.html#_2017-02-01_02_06_33_592237

2017-02-01 02:06:33.592237 | tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_deleted_server[id-581a397d-5eab-486f-9cf9-1014bbd4c984,negative]
2017-02-01 02:06:33.592305 | ------------------------------------------------------------------------------------------------------------------------------------------------------
2017-02-01 02:06:33.592321 |
2017-02-01 02:06:33.592340 | Captured traceback:
2017-02-01 02:06:33.592367 | ~~~~~~~~~~~~~~~~~~~
2017-02-01 02:06:33.592398 | Traceback (most recent call last):
2017-02-01 02:06:33.592453 | File "tempest/api/compute/servers/test_servers_negative.py", line 190, in test_reboot_deleted_server
2017-02-01 02:06:33.593010 | server['id'], type='SOFT')
2017-02-01 02:06:33.593072 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 485, in assertRaises
2017-02-01 02:06:33.593110 | self.assertThat(our_callable, matcher)
2017-02-01 02:06:33.593162 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 496, in assertThat
2017-02-01 02:06:33.593205 | mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
2017-02-01 02:06:33.593266 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 547, in _matchHelper
2017-02-01 02:06:33.593294 | mismatch = matcher.match(matchee)
2017-02-01 02:06:33.593345 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
2017-02-01 02:06:33.593388 | mismatch = self.exception_matcher.match(exc_info)
2017-02-01 02:06:33.593443 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
2017-02-01 02:06:33.593468 | mismatch = matcher.match(matchee)
2017-02-01 02:06:33.593515 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 475, in match
2017-02-01 02:06:33.593544 | reraise(*matchee)
2017-02-01 02:06:33.593597 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
2017-02-01 02:06:33.593618 | result = matchee()
2017-02-01 02:06:33.593667 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 1049, in __call__
2017-02-01 02:06:33.593699 | return self._callable_object(*self._args, **self._kwargs)
2017-02-01 02:06:33.593736 | File "tempest/lib/services/compute/servers_client.py", line 236, in reboot_server
2017-02-01 02:06:33.593777 | return self.action(server_id, 'reboot', **kwargs)
2017-02-01 02:06:33.593814 | File "tempest/lib/services/compute/servers_client.py", line 186, in action
2017-02-01 02:06:33.594170 | post_body)
2017-02-01 02:06:33.594219 | File "tempest/lib/common/rest_client.py", line 275, in post
2017-02-01 02:06:33.594261 | return self.request('POST', url, extra_headers, headers, body, chunked)
2017-02-01 02:06:33.594298 | File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
2017-02-01 02:06:33.594328 | method, url, extra_headers, headers, body, chunked)
2017-02-01 02:06:33.594360 | File "tempest/lib/common/rest_client.py", line 663, in request
2017-02-01 02:06:33.594392 | self._error_checker(resp, resp_body)
2017-02-01 02:06:33.594449 | File "tempest/lib/common/rest_client.py", line 775, in _error_checker
2017-02-01 02:06:33.594496 | raise exceptions.Conflict(resp_body, resp=resp)
2017-02-01 02:06:33.594555 | tempest.lib.exceptions.Conflict: An object with that identifier already exists
2017-02-01 02:06:33.594641 | Details: {u'code': 409, u'message': u"Cannot 'reboot' instance d9978a80-0349-4cb8-84fa-46aa0b8eaa16 while it is in vm_state building"}

http://logs.openstack.org/91/426991/1/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/f218227/logs/screen-n-api.txt.gz#_2017-02-01_01_25_02_367

The test creates a server, then deletes the server and waits for it to be gone and then tries to reboot it and expects a 404 but gets a 409 because the instance is building.

We're probably hitting a window of time between when we have a build request and a real instance due to the recent change of moving instance creation from nova-api to nova-conductor.

Looks like this started on 1/27:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Cannot%20'reboot'%20instance%5C%22%20AND%20message%3A%5C%22while%20it%20is%20in%20vm_state%20building%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

Which is when this merged: https://review.openstack.org/#/c/319379/