test_operationalerror_view_integration fails intermittently in parallel tests

I'm not able to dupe this after repeated attempts in a non-parallel lxc lucid container. Intermittent issues often signal timing issues, and the possibility that jumps out to me is that the bouncer isn't quite started yet. In pgbouncer.fixture.PGBouncerFixture.start, the code waits to see the bouncer's pid. Perhaps the presence of the bouncer's pid is not sufficient to indicate that the bouncer is ready. In that case, a proper fix might be to change bouncer code so that the pid is in fact a reliable indicator; or to change the fixture to look for some other more reliable signal. A more temporary fix (and/or an easier indication that this hypothesis has merit) might be to retry the assertion in this test a few time with a time.sleep(0.5) between each attempt. Working with a pgbouncer dev might be helpful.

Starting with a 0.5 sec sleep after pgbouncerFixture.setUp() sounds
like a sensible diagnostic.

Fixed in stable r15086 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15086) by a commit, but not testable.

The error happened again, even with 5 tries spread across .5 seconds between each. See line 7419 and previous of http://pastebin.ubuntu.com/934203/ .

(Additionally the format string uses a Python 2.7 feature, leading to the error seen in the traceback. "bouncer did not come up after {} attempts.".format(retries) needs to be something like "bouncer did not come up after {count} attempts.".format(count=retries) in 2.6.)

Fixed in stable r15157 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15157) by a commit, but not testable.

This still isn't quite addressed. I got another instance today (breaking our 12-run green streak!) http://pastebin.ubuntu.com/975858/ .

Right now, in the pgbouncer fixture, we are saying that we have finished with the "start" method as soon as there is a pgbouncer subprocess and we can poll it. There's no guarantee that this will mean that pgbouncer is actually "open for business". There might or might not be a guarantee that Launchpad is ready to recognize that pgbouncer has returned.

I think we should use the method Brad did initially (retrying the url) but just give it some more time. Let's keep the code in the bouncer's start method too--it's good. But let's re-add this (see https://code.launchpad.net/~bac/launchpad/bug-974617-2/+merge/103339), and increase the retry count to 10.

26 - # bouncer.start() can sometimes return before the service is actually
27 - # available for use. To be defensive, let's retry a few times. See
28 - # bug 974617.
29 - retries = 5
30 - for i in xrange(retries):
31 - try:
32 - urllib2.urlopen(url)
33 - except urllib2.HTTPError as e:
34 - if e.code != httplib.SERVICE_UNAVAILABLE:
35 - raise
36 - else:
37 - break
38 - time.sleep(0.5)
39 - else:
40 - raise TimeoutException(
41 - "bouncer did not come up after {} attempts.".format(retries))

Fixed in stable r15216 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15216>.

Still not resolved. Failed again today with Benji's patch. Benji's patch shows that pgbouncer is ready, but not that LP is ready. I still think we should re-add explicitly what is in comment 7. If that fails, after five seconds of retries and the checks Benji added, I'd say we need to look elsewhere for why we keep getting this failure.