Checkbox

fwts on armhf is missing many tests used for certification testing

Bug #1182630 reported by Jeff Lane 
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Checkbox
Fix Released
Low
Unassigned
Firmware Test Suite
Fix Released
Low
Colin Ian King

Bug Description

fwts on armhf is missing tests that we normally run in certification. For the general fwts test run, we run most of the "batch" tests. some of those (many, perhaps) are Intel or BIOS specific and don't apply to ARM. So we need to sort out WHAT tests ARE useful on arm and modify fwts_test to run one set of arm acceptable tests when arm is detected arch and another set when on intel/AMD systems.

This is the current version we're running on precise on ARM systems:
ubuntu@c18:/usr/share/checkbox/scripts$ fwts -v
fwts, Version V0.25.06precise1, Tue Sep 4 12:56:28 CST 2012
ubuntu@c18:/usr/share/checkbox/scripts$ apt-cache policy fwts
fwts:
  Installed: 0.25.06precise1
  Candidate: 0.25.06precise1
  Version table:
 *** 0.25.06precise1 0
        500 http://ports.ubuntu.com/ubuntu-ports/ precise-updates/universe armhf Packages
        100 /var/lib/dpkg/status
     0.24.21 0
        500 http://ports.ubuntu.com/ubuntu-ports/ precise/universe armhf Packages

And these are the tests avilable:
ubuntu@c18:/usr/share/checkbox/scripts$ fwts --show-tests
Batch tests:
 acpitables ACPI table settings sanity checks.
 apicinstance Check for single instance of APIC/MADT table.
 aspm PCIe ASPM check.
 checksum Check ACPI table checksum.
 hda_audio Check HDA Audio Pin Configs.
 klog Scan kernel log for errors and warnings.
 maxreadreq Checks firmware has set PCI Express MaxReadReq to a higher value on non-motherboard devices.
 method ACPI DSDT Method Semantic Tests.
 oops Scan kernel log for Oopses.
 version Gather kernel system information.

Utilities:
 acpidump Dump ACPI tables.
 uefidump Dump UEFI variables.

and THESE are the tests we normally run on Intel/AMD systems:
acpiinfo
acpitables
apicedge
apicinstance
bios_info
bios32
checksum
crs
dmesg_common
dmi_decode
ebda
fadt
fan
hda_audio
hpet_check
maxfreq
maxreadreq
microcode
mtrr
nx
os2gap
osilinux
smbios
version
virt
wmi
cstates
dmar

Revision history for this message
Jeff Lane  (bladernr) wrote :

Another missing test from the bug duped to this one:

$ sudo /usr/share/checkbox/scripts/fwts_test -w -l /tmp/fwts-wakealarm.log
No such test 'wakealarm', available tests:
acpidump acpitables aspm checksum crsdump hda_audio klog maxreadreq method oops
securebootcert uefidump uefirtmisc uefirttime uefirtvariable version

use: fwts --show-tests or fwts --show-tests-full for more information.

Since the system does have an rtc, we could test if it works and invoke rtcwake ourselves. We'd have to ask fwts folk whether this is equivalent and whether fwts can analyze the logs post-fact to get the same results we'd normally get.

Revision history for this message
Colin Ian King (colin-king) wrote :

I made a conscious decision to disable non-x86 specific BIOS/ACPI tests on some platforms (e.g. ARM, PowerPC, etc) which currently do not make sense of these platforms and will always fail (producing false falilures). When ACPI lands on ARM systems which we can test I will then enabled some of the ACPI specific tests on ARM, but not before as I want to ensure these work correctly.

I will add back the RTC wakealarm as I can see some non-x86 devices support this interface.

Revision history for this message
Colin Ian King (colin-king) wrote :

RTC wakealarm enabled for all architectures, fix committed 7961040de67af9b93e130f5e1aa232fce5b89d52, will appear in V13.06.00 of fwts

Revision history for this message
Colin Ian King (colin-king) wrote :

dmesg_common has been deprecated in newer versions of fwts (V0.27.03) and merged into the klog tests.

Revision history for this message
Colin Ian King (colin-king) wrote :

Jeff, is the cpufreq test missing from the list in the bug description?

Colin Ian King (colin-king)
Changed in fwts:
importance: Undecided → Low
assignee: nobody → Colin King (colin-king)
status: New → In Progress
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

@Jeff - is there actually anything to fix in Checkbox here? Or are we just waiting for Colin to update FWTS? If there will be no code changes in Checkbox then there shouldn't be a task here.

Changed in checkbox:
status: New → Incomplete
importance: Undecided → Low
Revision history for this message
Jeff Lane  (bladernr) wrote :

Well, not sure... truth is, it's possible that some of the tests we normally run simply aren't for ARM. And if so, then fwts_test will need to be fixed so that on x86 it runs the full list, and on ARM it runs only an appropriate subset.

Changed in checkbox:
status: Incomplete → Confirmed
Revision history for this message
Jeff Lane  (bladernr) wrote :

colin: I just tried out the latest build with the wakealarm test on armhf... it mostly works great now, and gives us something useful for the test flow. Thanks :) I need to add a trap in the wrapper to account for aborted tests as well, but that's trivial. Thank you for moving these along :)

Colin Ian King (colin-king)
Changed in fwts:
status: In Progress → Fix Released
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

Jeff - is this fixed in Checkbox now too, or what more do we need to do?

Changed in checkbox:
status: Confirmed → Incomplete
Revision history for this message
Jeff Lane  (bladernr) wrote :

I think we have addressed it for ARM at this point. Anything else that we encounter can be handled separately I think as a bug

Revision history for this message
Colin Ian King (colin-king) wrote :

If you need an extra fixes in fwts, file a new bug and I will get onto it.

Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

Thanks - currently there are no failures in fwts so I thought this wasn't a bug. Now Jeff has confirmed we can close it, and we'll deal with any further issues as new bugs.

Changed in checkbox:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.