subunit-filter --no-success doesn't filter out all successes

Bug #623642 reported by John A Meinel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
subunit
New
Undecided
Unassigned

Bug Description

I don't know the internal details, but I'm definitely getting some weird results from this command. The output here is a test run from pqm where there was one failure, and I'm trying to figure out what that was:

$ zcat pqm-stdout.gz | subunit-filter --no-skip --no-success --no-passthrough | subunit2pyunit
bzrlib.tests.blackbox.test_info.TestInfo.test_info_locking ... ok
bzrlib.tests.blackbox.test_revert.TestRevert.test_revert_dirname ... ok
...

There are 40 lines that end with 'ok'. (And otherwise 77 lines of output).

This is with subunit trunk:
  132 Jonathan Lange 2010-08-05 [merge]
      Move the bulk of subunit-ls into an importable Python module.

Which seems to be post the 0.0.6 release.

Also note that I checked with 'subunit-stats' to make sure:
$ zcat pqm-stdout.gz | subunit-filter --no-success --no-skip --no-passthrough | subunit-stats
Total tests: 41
Passed tests: 40
Failed tests: 1
Skipped tests: 0
Seen tags:

So --no-success leaves me with 40 *passed* tests. (These may be KnownFailure?)

Note that I *want* failures and errors, but *only* those. If I just use subunit-filter and do some manual filtering (for all the 'time' comments, etc that are just noise), I do find these lines:
test: bzrlib.tests.blackbox.test_info.TestInfo.test_info_locking
xfail: bzrlib.tests.blackbox.test_info.TestInfo.test_info_locking [ multipart

I was thinking to change the bzr test suite to actually filter out more junk, because right now I get a 7MB file with 99.9% of that garbage talking about skipped tests. (And my mail host is configured to reject emails >10MB, so if I have a suite with actual failures, it can easily exceed the cap.)

However, even without that, there still seems to be something wrong if I'm getting 'ok' from subunit2pyunit, but I'm passing --no-success and --no-skip to the rest of the code.

I'm uploading the pqm output, but I'll note that I've gotten this behavior with *all* pqm output that I've analyzed.

It may be that we just need "--no-xfail" ?

Related branches

Revision history for this message
John A Meinel (jameinel) wrote :
Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 623642] [NEW] subunit-filter --no-success doesn't filter out all successes

First WAG: you have a corrupted stream, and -filter is seeing the rest
of the stream as non-protocol content in the corrupted messages
context; then the next filter is seeing a clean stream but that
content isn't filtered.

compare
| subunit-filter | subunit-filter | subunit-filter | subunit-stats
| subunit-filter | subunit-filter | subunit-stats
| subunit-filter | subunit-stats
| subunit-stats

-Rob

Revision history for this message
Jonathan Lange (jml) wrote :

See also bug 526076.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.