Juju test suite fails sporadically due to low timeouts

Status changed to 'Confirmed' because the bug affects multiple users.

Hi Martin. These are basically timeouts because each test needs to finish in less than 5 seconds. Some of them take 1 - 2 seconds normally, so if your system gets busy for a second, that can be enough to push a few of them over 5 seconds.

A few tips:

1) I use sbuild, not pbuilder. If you use the 'mk-sbuild' script to build your chroot, you can pass it --eatmydata, which will help a lot. The tests run zookeeper, which fsyncs every write to disk. --eatmydata sets up your schroot to run everything inside 'eatmydata' which disables fsync and so de-couples your build from having to flush to disk.

2) If you have access to a machine with an SSD, that also helps a lot. On a non-SSD box, the tests take 7 minutes. With eatmydata, that drops to 5. With an SSD machine those numbers are cut in half. Thus, each tests is faster and so has less chance to cross over 5 seconds.

Anyway, I think this is worth re-titling to "test suite fails sporadically" and mark as an upstream bug.

fwiw, i'm experimenting atm with a small test runner change/diff to disable
zk fsync
http://paste.ubuntu.com/1057627/
-k

On Sun, Jun 24, 2012 at 10:58 AM, Clint Byrum <email address hidden> wrote:

> Hi Martin. These are basically timeouts because each test needs to
> finish in less than 5 seconds. Some of them take 1 - 2 seconds normally,
> so if your system gets busy for a second, that can be enough to push a
> few of them over 5 seconds.
>
> A few tips:
>
> 1) I use sbuild, not pbuilder. If you use the 'mk-sbuild' script to
> build your chroot, you can pass it --eatmydata, which will help a lot.
> The tests run zookeeper, which fsyncs every write to disk. --eatmydata
> sets up your schroot to run everything inside 'eatmydata' which
> disables fsync and so de-couples your build from having to flush to
> disk.
>
> 2) If you have access to a machine with an SSD, that also helps a lot.
> On a non-SSD box, the tests take 7 minutes. With eatmydata, that drops
> to 5. With an SSD machine those numbers are cut in half. Thus, each
> tests is faster and so has less chance to cross over 5 seconds.
>
> Anyway, I think this is worth re-titling to "test suite fails
> sporadically" and mark as an upstream bug.
>
> ** Summary changed:
>
> - FTBFS: Sveral tests fails
> + Juju test suite fails sporadically due to low timeouts
>
> ** Changed in: juju (Ubuntu)
> Status: Confirmed => Triaged
>
> ** Also affects: juju
> Importance: Undecided
> Status: New
>
> ** Changed in: juju
> Status: New => Confirmed
>
> ** Changed in: juju (Ubuntu)
> Importance: Undecided => Medium
>
> ** Changed in: juju
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to juju.
> https://bugs.launchpad.net/bugs/1017113
>
> Title:
> Juju test suite fails sporadically due to low timeouts
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1017113/+subscriptions
>

Ah, right.

Making it less hardware-bound (if that's the issue) sounds good :)

I believe this was fixed for Ubuntu in r559 by allowing the per-test timeout to be raised (and it is raised, to 30 seconds, in quantal's packaging).

Thats really just a workaround for the issue of slow or I/O intensive tests though.. so I'll leave the juju bug report open, but set its importance to Low.