Flag for tests not applicable to current series

Bug #913812 reported by Max Brustkern
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
QA Regression Testing
In Progress
Medium
Unassigned

Bug Description

Is there any method for flagging tests cases that no longer apply to the current series? Removing the test doesn't make sense as long as it works in some supported series, but some sort of indication that it isn't expected to install or run in the current series could be useful for the automated test runners we're using on the QA team. If nothing else is suggested, I propose
# QRT-Deprecated
This currently appears to apply to test-opie.py (after Natty,) test-zope3.py (after Hardy,) test-python2.4 (after Hardy,) test-python2.5 (after Hardy,) test-python3.1 (after Natty,) and test-ruby1.9 (after Lucid.) I can provide a patch if the idea is accepted.

Tags: qa-test-code
Changed in qa-regression-testing:
status: New → Opinion
summary: - opie not installable in Oneiric, Precise
+ Flag for tests not applicable to current series
description: updated
tags: added: qa-test-code
description: updated
description: updated
Revision history for this message
Steve Beattie (sbeattie) wrote :

Hi Max,

As an aside, the Opinion status is for when there's a dispute as to whether the issue is actually a bug. It is treated as a closed bug in launchpad, making it not show up in the default bug listing for a project.

In discussing it with the security team, something like this would probably be reasonable. I assume the QRT-Deprecated flag would take a release version in which the test became deprecated? We may have a weird issue where for some reason a package gets removed for a release or more and then comes back into the archive; netbeans is an example of a package doing this, though we don't have a test script for it.

Thanks!

Changed in qa-regression-testing:
importance: Undecided → Medium
status: Opinion → In Progress
Revision history for this message
Max Brustkern (nuclearbob) wrote :

In the branch I submitted I did use the first release in which the package was no longer available. I'm not sure of the best way to handle cases like netbeans without just moving to a list of releases in which the package is deprecated since we can't use a programmatic method without ensuring dependencies are in place.

Revision history for this message
pawciobiel (pawciobiel) wrote :

IMHO adding more logic either directly in test code or to runners as flags/switches/options makes test cases code unmaintainable.
I think it would be better if there was `qa-regression-testing` branch for every release.
Pros:
 * readable code
 * easier to maintain
 * faster to lunch and run
 * history separated but kept
 * easier to fix test cases

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.