checkbox should warn the user if a test case is not properly formatted

Hello!

We've had some discussions on the most desirable behavior for this.

The most obvious choice is for checkbox to warn at runtime of any erroneous jobs. The problem here is that if we warn the user mid-run about a bad job, there's little the user can do: for instance, he may not have knowledge of what the proper format is, or where to locate the job files. Worse, he'd have to abort the run in order to change the job, since checkbox doesn't reread jobs from disk if they change.

The other option is checking the jobs at build time. This simply warns the developers if a job has an incorrect format, and refuses to build a checkbox package with bad jobs. The developers *are* familiar with the format and can fix the problem so that the package always contains valid jobs. The problem with this option is that if someone other than checkbox developers wants to write tests, they will not be checked.

A possible compromise is including a job checking script with checkbox, that can be run both by developers and any other test writers. However I strongly believe that this would have to be separate from checkbox itself. The reason, as stated above, is that once the run is started, there's little we can do to fix a bad job, so for checkbox to warn the user of something that already failed and can't be fixed feels very unfriendly.

I'll get to work on something to address this need.

I agree that user has little to do with test case error while running checkbox.
But I believe this helps user to understand why a test case was not executed.
And then he/she could submit an issue for the case/tool developer.
I think this very important cause developer may miss required check during development.
And also the developer could confirm case is ok or not by running checkbox, no need for a separate script.

My point is user should be able to see some messages when anything goes wrong.
So they could help to report issue with useful info.

checkbox 0.16.4+bzr1978+201306151040~precise1

I intentionally remove verification section for misc/tester-info., and try checkbox-qt, but still no error message shown.

Daniel,

I think we may need to reconsider this approach (although the test is still valuable) - since oem-qa-checkbox variants also contain some jobs and they will not benefit from the testing based approach.

Looks like having the UI report malformatted jobs on-the-fly is the way to go here. Should not be too difficult to achieve, it's just replicating the verification done by the test but in the part that loads and sends the job to the UI.

My only question would be, is this something we want to happen all the time, to everyone? (i.e. Joe User who's just playing with checkbox). Maybe we could do it only if a certain oem-specific setting is activated, or only if we're in --debug mode (similar to what happens with the "jobs out of order" reporting. On the other hand, none of the checkbox default jobs should have a problem here (they're caught by the test), so it would in theory be safe to do this all the time.

Hi Daniel

Either way works for me. But I would prefer this been run all the time.
So problem in OEM branches could be noticed immediately.