A high failure in hibernate log was not detected by log_check script

Bug #1201667 reported by Jeffrey Chang
8
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Checkbox
Fix Released
High
Daniel Manrique

Bug Description

checkbox 0.16.4+bzr1982+201307131056~precise1

I saw "High Failures: 1 WARNING: The following test cases were reported as high level failures by fwts."
in power-management/hibernate_30_cycles test comment.
But power-management/hibernate-30-cycles-log-check shows PASSED No errors detected.

Please see https://certification.canonical.com/hardware/201111-10185/submission/92711/test-results/pass/?page=2.
hibernate_30_cycles.log attached.

Related branches

Revision history for this message
Jeffrey Chang (modern911) wrote :
Changed in checkbox:
importance: Undecided → Medium
tags: added: ce-qa-concern
Daniel Manrique (roadmr)
Changed in checkbox:
status: New → Confirmed
Revision history for this message
Daniel Manrique (roadmr) wrote :

OK, I confirmed this by downloading the attached hibernate_30_cycles.log file as provided, then running:

/usr/share/checkbox/scripts/sleep_test_log_check -v s4 hibernate_30_cycles.log

the output was:

No errors detected

even though as you mention, there are some errors in the file.

Looking at the script/parser, the output from fwts appears to have changed, which is why the parser doesn't detect the error.

The test summary section now looks like this:

Test Failure Summary
================================================================================

Critical failures: NONE

High failures: 2
 s4: Found 1 differences in device configuation during S4 cycle.
 s4: Failed to restore hibernate image.

Medium failures: NONE

Low failures: NONE

Other failures: NONE

Changed in checkbox:
status: Confirmed → Triaged
Revision history for this message
Daniel Manrique (roadmr) wrote :

Work to do:

1- Fix the parser so it recognizes the "new" fwts file format
2- Have the parser somehow recognize if the log format is possibly unparsable and report this, so a format change doesn't slip by us and cause false negatives.

Daniel Manrique (roadmr)
Changed in checkbox:
milestone: none → 2013-sep-13
Daniel Manrique (roadmr)
Changed in checkbox:
milestone: 2013-sep-13 → 2013-sep-27
Ara Pulido (ara)
Changed in checkbox:
milestone: 2013-sep-27 → 2013-oct-11
Daniel Manrique (roadmr)
Changed in checkbox:
milestone: 2013-oct-11 → 2013-oct-25
Changed in checkbox:
milestone: 2013-oct-25 → 2013-nov-08
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Also, this high failure cannot be correctly parsed as well:

FAILED [HIGH] HibernateImageRestore: Test 1, Failed to restore hibernate image.

ubuntu@201304-13208: $ /usr/share/checkbox/scripts/sleep_test_log_check s4 ~/.checkbox/hibernate-single.log
No errors detected

hibernate-single.log attached

Daniel Manrique (roadmr)
tags: added: scripts
Changed in checkbox:
milestone: 2013-nov-08 → 2014-jan-31
importance: Medium → High
Daniel Manrique (roadmr)
Changed in checkbox:
assignee: nobody → Daniel Manrique (roadmr)
status: Triaged → In Progress
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

I see merged branches so I assume this is Fix Commited.

Changed in checkbox:
status: In Progress → Fix Committed
Zygmunt Krynicki (zyga)
Changed in checkbox:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.