Intermittent failure running new instances

First occurence seems to be:

https://jenkins.openstack.org/job/gate-integration-tests-devstack-vm/5038/
June 12, 19:01 UTC

Although since the change is intermittent and "devstack-gate tests have passed recently without actually running any tests" this is a fuzzy match.

All fails are on hpcloud machines, All passes are on rackspace machines, so this may actually be something linked to the test setup...

Last hpcloud OK run is 5037(June 12, 18:57 UTC).
First hpcloud fail is 5038 (June 12, 19:01 UTC). Since then they just all failed on hpcloud runs.

Moving to openstack-ci until we can pinpoint to something in Nova.

This is as if virtualization support was removed from hpcloud-precise machines around June 12, 19:00 UTC.
I suggest we disable hpcloud machines while we investigate, to remove the false negatives.

They have been removed from the pool.

Before June 12, a flaw in hpcloud machines was causing tests to pass without being run. That's when we worked around that issue. So the failures could have been happening before then, and possibly since the time we switched to precise images.

Since devstack is configuring bare QEMU virtualization, it's not clear to me what kind of support could be missing to cause that.

It looks like this was due to hpcloud not adding the hostname to /etc/hosts on their precise images. The devstack gate script now does that.

Just to record it for posterity:

cloud-init in Oneiric (only! (see bug 890501 and bug 871966 for context)) would add an entry to /etc/hosts, which is why the problem didn't exist there. In Precise, there's no entry for $fqdn in /etc/hosts, so we rely on DNS to look that up. However, the DNS lookup coincided with the reconfiguration of the network that happens when the first instance is run on a nova-compute node.

The original analysis was thus wrong. Seeing things like:

libvir: QEMU error : Domain not found: no domain with matching name 'instance-00000001'

is perfectly normal and expected behaviour prior to the first launch of an instance.