unity trunk failing autopilot tests (02/2014)

Yes, I've been working on Autopilot test failures for the last 4 weeks and having gone from > 100 failures on each machine to ~33-35 on each machine means we are in much better shape.

From my debugging of the remaining issues, many of these failures are not issues in Unity, but issues with Autopilot and the machines themselves. I have entered a bug report for the Autopilot issue with no response on the bug. I've also sent email to Ubuntu CI about the issues with the machines and no response there either. That said, there are still a few remaining issues that are due to Unity and I'm currently working on them.

Thanks for the update! Indeed it's very good progress. Eventually when the release/publish is wanted to be done, we can use a whitelist of allowed-fails in case CI does not get to fix all the problems in time. Feature freeze is in two weeks and hopefully the release would be scheduled to be done before that with some days to spare at least.

I'm not sure a full whitelist can be used, at least for the Intel machine. That machine is having some resource issue where it begins to use a great amount of memory. Once this happens, random tests begin to fail because Unity actions cannot complete in time (such as opening the Hud/Dash) before the test times out. Eventually, the kernel begins killing processes to free memory which also affects tests running. Since this is non-deterministic, I can't say for sure which tests will fail.

Also, the Autopilot bug affects many test_panel tests, but this only occurs on multi-monitor when dragging a window to a different monitor. It is also non-deterministic which monitor the application opens up in, so it's hard to say which of these tests will fail. It depends on if a window needs to be dragged to a different monitor or not.

There are a few tests we could whitelist though, so if the time comes, I can provide a list of those tests.

Since we moved over to the CI train and the number of locally run AP test failures are acceptable, I'm marking this bug Fix Committed.

Yep, excellent work on bringing up the successful tests!