Linaro PMWG QA

Intermediate results should not use pass/fail, it confuses LAVA

Bug #1156614 reported by Basil Eljuse
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Linaro PMWG QA
Fix Released
High
sanjay singh rawat
Linaro PMWG QA 2013.04

Bug Description

Despite pm-qa has fewer than 100 test scripts spread across multiple test suites, LAVA reports over than 250 test cases from this suite.

The reason for this mis-interpretation is down to the fact that LAVA has a results parsing method by which it uses a reg-ex to interpret each log outputs from the test suites and matches them with 'pass/fail' tokens.

In pm-qa suites the scripts output log files even for intermediate test steps with a pass/ fail verdict which prompts LAVA to pick up as separate test cases.

The solution is to use say OK/NOT OK tokens for all intermediate test steps in each script and have only a final pass/fail verdict from each script in pm-qa.

This would let LAVA results parser to ignore any intermediate steps to be reported as separate test cases while results parsing.

Amit Kucheria (amitk)
Changed in linaro-power-qa:
assignee: nobody → Hongbo Zhang (hongbo-zhang)
importance: Undecided → High
status: New → Confirmed
Amit Kucheria (amitk)
Changed in linaro-power-qa:
milestone: none → 2013.04
Amit Kucheria (amitk)
summary: - pm-qa test scripts need to use only pass/fail verdicts as a single final
- verdict rather than intermediate results which makes LAVA to interpret
- these as multiple separate test cases
+ Intermediate results should not use pass/fail, it confuses LAVA
Amit Kucheria (amitk)
Changed in linaro-power-qa:
assignee: Hongbo Zhang (hongbo-zhang) → sanjay singh rawat (sanjay-rawat)
Revision history for this message
sanjay singh rawat (sanjay-rawat) wrote :

- sent patch for the issue
- need to merge and relase

sanjay singh rawat (sanjay-rawat)
Changed in linaro-power-qa:
status: Confirmed → Fix Committed
sanjay singh rawat (sanjay-rawat)
Changed in linaro-power-qa:
status: Fix Committed → Fix Released
Revision history for this message
Basil Eljuse (basil-eljuse) wrote :

Could sanjay please confirm that for every test script there is a final 'PASS/FAIL' verdict?
The intermediate steps show OK or Err which is fine but as a final verdict each test should have a single pass/fail verdict. Please confirm.

Revision history for this message
Basil Eljuse (basil-eljuse) wrote :

Just to be clear. This is a concern primarily from LAVA results parsing point of view.

Lava will be looking for single output line which has the

<testcase id> <pass/fail verdict>

in some format so the results parsing https://lava-dispatcher.readthedocs.org/en/latest/lava_test_shell.html

An example results parse pattern from the documentation is

parse:
    pattern: "(?P<test_case_id>.*-*):\\s+(?P<result>(pass|fail))"

So for example

cpufreq_01.0/cpu0: checking 'scaling_available_frequencies' exists... Ok
cpufreq_01.1/cpu0: checking 'scaling_cur_freq' exists... Ok
cpufreq_01.2/cpu0: checking 'scaling_setspeed' exists... Ok
cpufreq_01.0/cpu1: checking 'scaling_available_frequencies' exists... Ok
cpufreq_01.1/cpu1: checking 'scaling_cur_freq' exists... Ok
cpufreq_01.2/cpu1: checking 'scaling_setspeed' exists... Ok
cpufreq_01.0/cpu2: checking 'scaling_available_frequencies' exists... Ok
cpufreq_01.1/cpu2: checking 'scaling_cur_freq' exists... Ok
cpufreq_01.2/cpu2: checking 'scaling_setspeed' exists... Ok
cpufreq_01.0/cpu3: checking 'scaling_available_frequencies' exists... Ok
cpufreq_01.1/cpu3: checking 'scaling_cur_freq' exists... Ok
cpufreq_01.2/cpu3: checking 'scaling_setspeed' exists... Ok
-------- total = 12
-------- pass = 12

If pass =12 implies the cpufreq_01 to be declared pass then there should be a final verdict line

'cpufreq_01 test is a pass'
or any similar verdict line

If atleast one of these subtests fail would mean the whole script to be declared a failure then

'cpufreq_01 test is a fail'
or any similar verdict line

so the lava parser could capture this.

Revision history for this message
Milosz Wasilewski (mwasilew) wrote :

With the current version of pm-qa test case LAVA will not pick any retults. The fix is pretty trivial:
 1. make a clear verdict in include/functions.sh (patch attached)
 2. fix the regexp in yaml file to
^testcase_(?P<test_case_id>\\w+)\\s+(?P<result>\\w+)$

Revision history for this message
sanjay singh rawat (sanjay-rawat) wrote :

Senthil had updated the pwrmgmt lava test script for this issue.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.