Report skipped tests

I'm looking at the patch now. Could we make the formatter API backward-
compatible by putting the 'n_skipped' argument last, and defaulting it to
'None'? I realize that it makes the API read somewhat awkwardly, but I
think that having things "Just Work" under older Python versions, or where
'unittest2' is not present, trumps that. We could then keep the output text
compatible if 'n_skipped' was not passed (those same cases).

The attached patch makes the formatter API backward-compatible: omitting the 'n_skipped' argument causes the "old" format to be emitted.

Because isn't going to be straigtforward to test this across versions which do / don't support the 'skipped' feature in 'unittest2', we should bump the version number (i.e., to '4.1'), and ensure that the ZTK 1.0 and 1.1 configurations continue to use a branch created from before this patch is merged. We can then update the tests to match the new format (including the 'skipped' output).

On Tue, Oct 25, 2011 at 03:44:33PM -0000, Tres Seaver wrote:
> I'm looking at the patch now. Could we make the formatter API backward-
> compatible by putting the 'n_skipped' argument last, and defaulting it to
> 'None'? I realize that it makes the API read somewhat awkwardly, but I
> think that having things "Just Work" under older Python versions, or where
> 'unittest2' is not present, trumps that. We could then keep the output text
> compatible if 'n_skipped' was not passed (those same cases).

I'm not sure to perfectly understand what you mean but just to be clear:

* you can still run zope.testrunner with an older version of Python
  (without the new unittest module) or without unittest2. So, things
  should "Just Work" with those setups;

* but it breaks the internal zope.testrunner API: people who
  had a custom output formatter will need to adjust their code;

* it also breaks things if you were relying on the text output.

I had the feeling you were implying that it wasn't working 'normally'
with 'non-compatible' versions, but if you agree if the points above, so
everything is fine.

On Tue, Oct 25, 2011 at 05:19:34PM -0000, Tres Seaver wrote:
> The attached patch makes the formatter API backward-compatible:
> omitting the 'n_skipped' argument causes the "old" format to be emitted.
>
> Because isn't going to be straigtforward to test this across versions
> which do / don't support the 'skipped' feature in 'unittest2', we
> should bump the version number (i.e., to '4.1'), and ensure that the ZTK
> 1.0 and 1.1 configurations continue to use a branch created from before
> this patch is merged. We can then update the tests to match the new
> format (including the 'skipped' output).

To be clear: we keep the formatter API backward-compatible even if we
bump the version, ditch the old format (without the 'skipped' output)
along with the few branches in the code to support both and we only test
the new format?

> ** Patch added: "lp_880207-alternate.patch"
> https://bugs.launchpad.net/zope.testrunner/+bug/880207/+attachment/2572027/+files/lp_880207-alternate.patch

I looked at the patch, and there were two issues to fix:

* there were 2 calls to output.summary() in runner.py;
* there were 2 superfluous self.color('info') in the summary() method of
  the colored formatter, and a couple of extra whitespace too.

I attach the fixed version to this message.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/25/2011 09:01 PM, Jonathan Ballet wrote:
> On Tue, Oct 25, 2011 at 03:44:33PM -0000, Tres Seaver wrote:
>> I'm looking at the patch now. Could we make the formatter API
>> backward- compatible by putting the 'n_skipped' argument last,
>> and defaulting it to 'None'? I realize that it makes the API
>> read somewhat awkwardly, but I think that having things "Just
>> Work" under older Python versions, or where 'unittest2' is not
>> present, trumps that. We could then keep the output text
>> compatible if 'n_skipped' was not passed (those same cases).
>
> I'm not sure to perfectly understand what you mean but just to be
> clear:
>
> * you can still run zope.testrunner with an older version of
> Python (without the new unittest module) or without unittest2. So,
> things should "Just Work" with those setups;
>
> * but it breaks the internal zope.testrunner API: people who had a
> custom output formatter will need to adjust their code;
>
> * it also breaks things if you were relying on the text output.
>
> I had the feeling you were implying that it wasn't working
> 'normally' with 'non-compatible' versions, but if you agree if the
> points above, so everything is fine.

I was trying to ensure that formatters with the new API would work
even when called by older code: there may not be any such code, however.

Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 <email address hidden>
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6nZTIACgkQ+gerLs4ltQ7ApgCdG/2M43MT14KmFIASZjTBXNZD
VqoAniJAMQfy97NAF6cBvA5L+pfWkFZ/
=ywP7
-----END PGP SIGNATURE-----

I discovered on subtle bug, which occurs if you run tests with an 'old' Python version (without the new unittest module) but using the unittest2 module and this new skip feature:

unittest2 will happily report the skipped tests to zope.testrunner, which will try to add them into the TestResut object. However, since zope.testrunner only knows about 'unittest', and so, the old version from the stdlib which knows nothing about skipped tests, zope.testrunner fails to add them into the TestResult object because it lacks some methods there (notabley the TestResult.addSkip() method).

I'd be very interested to get zope.testrunner to report skips. So, what is the status here? What needs to be decided, what needs to be done to move forward? Is there anything I can do to help?

I haven't touched this patch since October 2011.
It should be first updated so that it applies cleanly on the latest
zope.testrunner on Github and then ensure it works in the following
cases:

* Python 2.6 without unittest2
* Python 2.6 with unittest2
* Python 2.7/3.1/3.1+ without unittest2
* Python 2.7/3.1/3.1+ with unittest2 (but unittest2 should not be used,
  since it has been merged in Python 2.7 and 3.1)

Support for Python < 2.6 has been dropped so it should be slightly
easier to test. zope.testrunner is also now using tox, which should also
make things easier.

I don't recall exactly what I was meaning in
https://bugs.launchpad.net/zope.testrunner/+bug/880207/comments/7 but it
seems there was some kind of problem running Python 2.6 (or 2.5, but
it's not a problem anymore now) with the unittest2 module.

If you give me a few days, I can give it a look and try to get this
merged upstream.

On Tue, May 07, 2013 at 07:05:07AM -0000, Wolfgang Schnerring wrote:
> I'd be very interested to get zope.testrunner to report skips. So, what
> is the status here? What needs to be decided, what needs to be done to
> move forward? Is there anything I can do to help?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/880207
>
> Title:
> Report skipped tests
>
> Status in zope.testrunner:
> Opinion
>
> Bug description:
> Since Python 2.7, Python 3.1 and when using the unittest2 package,
> developers can specify that some tests must be skipped and not
> launched by the test runner.
>
> I made a preliminary patch which adds support in zope.testrunner to
> report skipped tests. The code should work even if the underlying
> unittest library doesn't support the "skip" feature.
>
> It breaks the API of the formatters, because I need to specify the
> number of skipped tests in the summary line and I added a method to
> report when one test is being skipped.
>
> I didn't updated the tests of zope.testrunner itself, because I would
> like to have an early feedback on the implementation:
>
> * is it OK to break the API of the formatters?
> * is it OK to break the format of the defaut text outputs, since it now always add the number of skipped tests?
>
> If it seems good for the maintainer and they are willing to accept the
> patch, I'll update it with the fix for the zope.testrunner tests
> themselves.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/zope.testrunner/+bug/880207/+subscriptions

I moved the patch to Github: https://github.com/multani/zope.testrunner/tree/support-skip-feature

This has been merged in Git's master on https://github.com/zopefoundation/zope.testrunner/commit/87c4bf7abf7ea5462981bdeab3319a3e241c67d9
It should be available with the next release 4.4.

Fixed in zope.testrunner 4.0.0.