Intermittent "KeyError: 'allocations'" in functional tests

I spent a bit of time looking into this, and though I couldn't figure out why it was happening I got closer. Elsewhere in the logs <http://logs.openstack.org/87/472287/1/gate/gate-nova-tox-functional-py35-ubuntu-xenial/79cb96d/console.html#_2017-06-08_17_42_12_498469> we get things like this:

2017-06-08 17:42:12.498469 | b'2017-06-08 17:38:27,167 ERROR [nova.scheduler.client.report] Failed to retrieve filtered list of resource providers from placement API for filters {\'resources\': \'DISK_GB:20,MEMORY_MB:2048,VCPU:1\'}. Got 300: {"choices": [{"id": "v2.0", "status": "SUPPORTED", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "links": [{"rel": "self", "href": "http://127.0.0.1:41392/v2/resource_providers"}]}, {"id": "v2.1", "status": "CURRENT", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "links": [{"rel": "self", "href": "http://127.0.0.1:41392/v2.1/resource_providers"}]}]}.'

That's the output from the nova/api/openstack/compute/versions.py Versions.multi method which a request to the placement service should never see, so somehow things have been confused in the fixtures such that requests that are supposed to be destined for the placement fixture are ending up on the compute fixture. You can see from elsewhere in the logs <http://logs.openstack.org/87/472287/1/gate/gate-nova-tox-functional-py35-ubuntu-xenial/79cb96d/console.html#_2017-06-08_17_42_12_488625> that a request gets a 200 and logs as nova.placement.wsgi.server but then is immediate followed by a 300 and logs as nova.osapi_compute.wsgi.server

wat?

So recording this for posterity in case anyone has some some ideas. The wsgi service fixtures aren't something I'm too familiar with. If I had to guess, eventlet is being odd.

Oh, I forget to mention: the key error is because there is a json body (the weird version doc) but it doesn't have the key...

Related fix proposed to branch: master
Review: https://review.openstack.org/483972

Reviewed: https://review.openstack.org/483972
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d812c7eb3c7f0ce80c3548e9aec10b9fdc650a67
Submitter: Jenkins
Branch: master

commit d812c7eb3c7f0ce80c3548e9aec10b9fdc650a67
Author: Matt Riedemann <email address hidden>
Date: Fri Jul 14 11:40:13 2017 -0400

    Set wsgi.keep_alive=False globally for tests

    This was previously set for all functional API samples
    tests to try and fix a weird issue where requests and responses
    seemed to get mixed up between the compute API wsgi service
    fixture used in the API samples functional tests and the
    Placement API functional tests.

    However, we're also seeing these weird mixup issues in
    non-API samples tests, and there are other wsgi server fixtures
    that should be setting this, like the OSMetadataServer
    fixture. And there are functional tests that don't inherit from
    _IntegratedTestBase so we should just set this globally to be
    safe.

    Related-Bug: #1686109
    Related-Bug: #1696834

    Change-Id: Ie307e4d43be4b3e40dd75384ea6bb77598811713

This is likely fixed by these changes to use wsgi_intercept in our compute API and Placement API fixtures:

https://github.com/openstack/nova/commit/fdf27abf7db233ca51f12e2926d78c272b54935b

https://github.com/openstack/nova/commit/eed6ced78776a3b9a7ada7b0c8ff74eaa376efaf