Comment 1 for bug 1187216

Hello!

We've had some discussions on the most desirable behavior for this.

The most obvious choice is for checkbox to warn at runtime of any erroneous jobs. The problem here is that if we warn the user mid-run about a bad job, there's little the user can do: for instance, he may not have knowledge of what the proper format is, or where to locate the job files. Worse, he'd have to abort the run in order to change the job, since checkbox doesn't reread jobs from disk if they change.

The other option is checking the jobs at build time. This simply warns the developers if a job has an incorrect format, and refuses to build a checkbox package with bad jobs. The developers *are* familiar with the format and can fix the problem so that the package always contains valid jobs. The problem with this option is that if someone other than checkbox developers wants to write tests, they will not be checked.

A possible compromise is including a job checking script with checkbox, that can be run both by developers and any other test writers. However I strongly believe that this would have to be separate from checkbox itself. The reason, as stated above, is that once the run is started, there's little we can do to fix a bad job, so for checkbox to warn the user of something that already failed and can't be fixed feels very unfriendly.

I'll get to work on something to address this need.