Monkey always fails (eventually) these failures should not be marked as such

Hi, plars, pfalcon, pfefferz

For this problem, I want to make the measurement displayed as the
attachment,
How do you think about that?

Thanks,
Yongqin Liu
On 20 October 2011 16:44, Launchpad Bug Tracker
<email address hidden>wrote:

> ** Branch linked: lp:~liuyq0307/lava-dashboard/add-measurement-for-json
>
> ** Branch linked: lp:~liuyq0307/linaro-android-frontend/add-measurement-
> for-testresult-table
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Android Test.
> https://bugs.launchpad.net/bugs/877859
>
> Title:
> Monkey always fails (eventually) these failures should not be marked
> as such
>
> Status in Lava Android Test Runner:
> Confirmed
>
> Bug description:
> Monkey will always "fail" in practice after a little bit (new builds
> 1-5 secs, more seasoned builds 1-4 hours, hardened builds 24-48
> hours). The monkey result should be how long it ran. This should be
> listed on the android-build page.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-android-test/+bug/877859/+subscriptions
>

Hello Yongqin,

On Thu, 20 Oct 2011 17:16:51 +0800
yong qin <email address hidden> wrote:

> Hi, plars, pfalcon, pfefferz
>
> For this problem, I want to make the measurement displayed as the
> attachment,
> How do you think about that?

Well, my concern (as it was before) is how much information we
want to present on build pages. For example, if we run 100 tests in
LAVA, how detailed results for them will look on a build page, would it
make rest of information on the page less accessible?

But otherwise, that's Zach's call.

>
> Thanks,
> Yongqin Liu
> On 20 October 2011 16:44, Launchpad Bug Tracker
> <email address hidden>wrote:
>
> > ** Branch linked:
> > lp:~liuyq0307/lava-dashboard/add-measurement-for-json
> >
> > ** Branch linked:
> > lp:~liuyq0307/linaro-android-frontend/add-measurement-
> > for-testresult-table
> >
> > --
> > You received this bug notification because you are a member of
> > Linaro Validation Team, which is subscribed to LAVA Android Test.
> > https://bugs.launchpad.net/bugs/877859
> >
> > Title:
> > Monkey always fails (eventually) these failures should not be
> > marked as such
> >
> > Status in Lava Android Test Runner:
> > Confirmed
> >
> > Bug description:
> > Monkey will always "fail" in practice after a little bit (new
> > builds 1-5 secs, more seasoned builds 1-4 hours, hardened builds
> > 24-48 hours). The monkey result should be how long it ran. This
> > should be listed on the android-build page.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/lava-android-test/+bug/877859/+subscriptions
> >

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

On Thu, 20 Oct 2011 13:50:48 +0300
Paul Sokolovsky <email address hidden> wrote:

> Hello Yongqin,
>
> On Thu, 20 Oct 2011 17:16:51 +0800
> yong qin <email address hidden> wrote:
>
> > Hi, plars, pfalcon, pfefferz
> >
> > For this problem, I want to make the measurement displayed as the
> > attachment,
> > How do you think about that?

Also, my attention caught your note that monkey will always crash -
sooner or later. I barely know anything about monkey (except for what
I heard from Zach before), so wondered what is so magic in it which
causes everything crash ;-). I tried to at least look at what it
crashes now, for example:

http://validation.linaro.org/lava-server/dashboard/streams/anonymous/android-daily/bundles/6b8cc3cd479a018083e6de6bf9205dcc24b833df/9419bb64-e306-4b05-b91c-1fdae38a3d84/attachments/16330/

# // java.lang.NullPointerException
# // at com.android.speechrecorder.SpeechRecorderActivity$4.run(SpeechRecorderActivity.java:192)
# //
# ** Monkey aborted due to error.
# Events injected: 14
# ## Network stats: elapsed time=5246ms (0ms mobile, 0ms wifi, 5246ms not connected)
# ** System appears to have crashed at event 14 of 100000 using seed 1
# MONKEY_RET_CODE=14

And I wonder if number of events processed before crash is actually a
bit more scientific criterion than just run time, and if it should be
captured and visibly reported in addition to run time.

Thanks,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

This looks good.

I'm actually okay with displaying all the test results on the build page. We may have to clean up the formatting eventually, but being able to see at a glance how things did, is extremely useful.

Frans to validate with CI loop.

lc4.11 hacking sessions work.

We're seeing the time Monkey ran for, but it still reports itself as a test failure, when we'd like this to always be a pass.

Example:
a-b.l.o:
https://android-build.linaro.org/builds/~linaro-android/panda/#build=364
lava:
http://validation.linaro.org/lava-server/dashboard/streams/anonymous/android-daily/bundles/38b54d9bd3f42b7ffdaf0811a6bfdac7c3a1ee07/9087ed6f-da10-4947-b210-43b3e4a2742e/

Can we change the way Monkey runs so that it always reports pass?

Hi, Frans,

The things you mentioned above is done in
https://code.launchpad.net/~liuyq0307/lava-android-test/monkey-always-scucess/+merge/81137,
but still not been merged and deployed.

Looks like this is working now:

https://android-build.linaro.org/builds/~linaro-android/panda/#build=375