subunit-filter --no-success doesn't filter out all successes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
subunit |
New
|
Undecided
|
Unassigned |
Bug Description
I don't know the internal details, but I'm definitely getting some weird results from this command. The output here is a test run from pqm where there was one failure, and I'm trying to figure out what that was:
$ zcat pqm-stdout.gz | subunit-filter --no-skip --no-success --no-passthrough | subunit2pyunit
bzrlib.
bzrlib.
...
There are 40 lines that end with 'ok'. (And otherwise 77 lines of output).
This is with subunit trunk:
132 Jonathan Lange 2010-08-05 [merge]
Move the bulk of subunit-ls into an importable Python module.
Which seems to be post the 0.0.6 release.
Also note that I checked with 'subunit-stats' to make sure:
$ zcat pqm-stdout.gz | subunit-filter --no-success --no-skip --no-passthrough | subunit-stats
Total tests: 41
Passed tests: 40
Failed tests: 1
Skipped tests: 0
Seen tags:
So --no-success leaves me with 40 *passed* tests. (These may be KnownFailure?)
Note that I *want* failures and errors, but *only* those. If I just use subunit-filter and do some manual filtering (for all the 'time' comments, etc that are just noise), I do find these lines:
test: bzrlib.
xfail: bzrlib.
I was thinking to change the bzr test suite to actually filter out more junk, because right now I get a 7MB file with 99.9% of that garbage talking about skipped tests. (And my mail host is configured to reject emails >10MB, so if I have a suite with actual failures, it can easily exceed the cap.)
However, even without that, there still seems to be something wrong if I'm getting 'ok' from subunit2pyunit, but I'm passing --no-success and --no-skip to the rest of the code.
I'm uploading the pqm output, but I'll note that I've gotten this behavior with *all* pqm output that I've analyzed.
It may be that we just need "--no-xfail" ?
Related branches
- Subunit Developers: Pending requested
-
Diff: 77 lines (+17/-5)3 files modifiedNEWS (+6/-1)
filters/subunit-filter (+7/-3)
python/subunit/test_results.py (+4/-1)
First WAG: you have a corrupted stream, and -filter is seeing the rest
of the stream as non-protocol content in the corrupted messages
context; then the next filter is seeing a clean stream but that
content isn't filtered.
compare
| subunit-filter | subunit-filter | subunit-filter | subunit-stats
| subunit-filter | subunit-filter | subunit-stats
| subunit-filter | subunit-stats
| subunit-stats
-Rob