want to run tests in a particular plugin without installing it

  affects /products/bzr
  importance wishlist

Oops, I accidently typed products/bzr-svn (bad habit) here, while I
actually meant /products/bzr.
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

Interesting though. Though isn't it sufficient to do:

BZR_PLUGIN_PATH="parent/dir" bzr selftest plugin_name
?

On Thu, 2007-02-01 at 17:34 +0000, John A Meinel wrote:
> Interesting though. Though isn't it sufficient to do:
>
> BZR_PLUGIN_PATH="parent/dir" bzr selftest plugin_name
> ?
That would mean that all the plugins in the parent directory would be loaded. In my case, that means loading 12 different variations of the plugin I would like to test, all of which try to register stuff.

Cheers,

Jelmer
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

is a bzr bug

What about creating a directory containing links to the plugin directories you want to test instead ?

As in:

mkdir test-plugins
cd test-plugins
ln -s path_to_plugin_to_be_tested plugin_name
BZR_PLUGIN_PATH=test-plugins bzr selftest plugin_name

On Mon, 2007-02-05 at 07:52 +0000, vila wrote:
> What about creating a directory containing links to the plugin
> directories you want to test instead ?
>
> As in:
>
> mkdir test-plugins
> cd test-plugins
> ln -s path_to_plugin_to_be_tested plugin_name
> BZR_PLUGIN_PATH=test-plugins bzr selftest plugin_name
That's what I do right now, but it's suboptimal - it requires me to have
one extra directory with symlink around for each branch of the plugin.

Cheers,

Jelmer

Several plugins appear to work around this by adding "test-<pluginname>" commands, which is suboptimal IMHO.

I think this would make a lot of sense. Packagers often want to run the tests without installing bzr or the relevant plugin, for example. Perhaps having a specific command for testing plugins would make sense. It could load the plugin from the specified location and run that plugin's tests only (by default)

Discussed on IRC:
we could add a test-plugin command that will implement jam suggestion above (load the plugin and run selftest -s bp.plugin) with options to load *only* the plugin or *also* the plugin and accept the plugin path as a parameter and various other schemes for plugin authors that needs to test various versions of the same plugin.

Hi,

Having a better solution for this would be great, but with the help of
jam and vila I just knocked this up:

    if __name__ == '__main__':
       import os
       import subprocess
       import sys
       dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "plugins")
       retcode = subprocess.call("bzr selftest -s bzrlib.plugins.<plugin name>",
               shell=True, env={"BZR_PLUGIN_PATH": dir})
       sys.exit(retcode)

Put that at the top of __init__.py (before any "import bzrlib.plugins.<plugin name>"
type imports), with a symlink in the branch that points

   ./plugins/<plugin name> -> ./

so that it is loaded under the correct name. The "python __init__.py" will run the
tests.

This obviously doesn't work on windows, so it's not good for wide adoption.

Thanks,

James

On Thu, 2009-04-16 at 20:30 +0000, James Westby wrote:
> Hi,
>
> Having a better solution for this would be great, but with the help of
> jam and vila I just knocked this up:
>
> if __name__ == '__main__':

ug.

from bzrlib import option, commands
commands.Command.hooks.install_named_hook('extend_command',
    test_coverage, 'test_plugin')

def test_plugin(cmd):
    if cmd.name() != 'selftest':
        return
    def handle_test_plugin(option, name, value, parser):
        cmd.additional_selftest_args['starting_with'] = \
            'bzrlib.plugins.%s' % value
    cmd.takes_options = list(cmd.takes_options)
    cmd.takes_options.append(
        option.Option('test-plugin',
           help='Run tests for a single named plugin',
           custom_callback=handle_test_plugin))

On Thu, 2009-04-16 at 22:28 +0000, Robert Collins wrote:
> On Thu, 2009-04-16 at 20:30 +0000, James Westby wrote:
> > Hi,
> >
> > Having a better solution for this would be great, but with the help of
> > jam and vila I just knocked this up:
> >
> > if __name__ == '__main__':
>
> ug.
>
> from bzrlib import option, commands
> commands.Command.hooks.install_named_hook('extend_command',
> test_coverage, 'test_plugin')
>
>
> def test_plugin(cmd):
> if cmd.name() != 'selftest':
> return
> def handle_test_plugin(option, name, value, parser):
> cmd.additional_selftest_args['starting_with'] = \
> 'bzrlib.plugins.%s' % value
> cmd.takes_options = list(cmd.takes_options)
> cmd.takes_options.append(
> option.Option('test-plugin',
> help='Run tests for a single named plugin',
> custom_callback=handle_test_plugin))

Thanks,

However, as far as I can see this doesn't cover all the things
that I need to make this useful.

Allow me to be more explicit about the requirements that I have.

I have written the following hook:

import os
import shutil
import subprocess
import tempfile

from bzrlib import (
        branch,
        errors,
        export,
        trace,
        )

def run_test_suite_hook(params):
    config = params.branch.get_config()
    command = config._get_location_config()._get_user_option("pre_change_branch_tip_test_command")
    if command is None:
        return
    trace.note("Running the testsuite: %s" % command)
    revid = params.new_revid
    revtree = params.branch.repository.revision_tree(revid)
    tempdir = tempfile.mkdtemp()
    try:
        export_dir = os.path.join(tempdir, "export")
        export.export(revtree, export_dir, format="dir")
        retcode = subprocess.call(command, cwd=export_dir, shell=True)
        if retcode != 0:
            raise errors.TipChangeRejected("The testsuite failed")
    finally:
        shutil.rmtree(tempdir)

branch.Branch.hooks.install_named_hook('pre_change_branch_tip',
        run_test_suite_hook, "Check the testsuite passes before the "
        "change is made.")

which looks for a configuration option giving the command to run to
validate the commit, for a poor-man's PQM for my local work.

I'd like to use this for my bzr plugins, however the usual way
I test those is "bzr selftest -s bp.<plugin name>". However,
that requires the tree I am committing is the one that bzr will
load, which isn't always going to be the case. In particular,
the above code exports to a temporary directory to run the tests,
so it won't be something loaded by bzr.

So, I need a way to run the tests for a plugin, but also pass
an arbitrary filesystem location to use the tests from, and have
the plugin appear as the correct name to bzr.

My code does this by setting BZR_PLUGIN_PATH based on __file__,
so that it will always be testing the code that you are running
it from, as well as a symlink to get it to appear as the correct
name, regardless of the directory name.

Running the particular tests of a plugins isn't hard thanks to
"-s", but loading a plugin from an arbitrary location with a specific
name still is (As J...

On Thu, 2009-04-16 at 23:36 +0000, James Westby wrote:

> Thanks,
>
> However, as far as I can see this doesn't cover all the things
> that I need to make this useful.
>
> Allow me to be more explicit about the requirements that I have.
>
> I have written the following hook:
>
> import os
> import shutil
> import subprocess
> import tempfile
>
> from bzrlib import (
> branch,
> errors,
> export,
> trace,
> )
>
> def run_test_suite_hook(params):
> config = params.branch.get_config()
> command = config._get_location_config()._get_user_option("pre_change_branch_tip_test_command")
> if command is None:
> return
> trace.note("Running the testsuite: %s" % command)
> revid = params.new_revid
> revtree = params.branch.repository.revision_tree(revid)

nb: branch.basis_tree() is more pithy

> tempdir = tempfile.mkdtemp()
> try:
> export_dir = os.path.join(tempdir, "export")
> export.export(revtree, export_dir, format="dir")
> retcode = subprocess.call(command, cwd=export_dir, shell=True)
> if retcode != 0:
> raise errors.TipChangeRejected("The testsuite failed")
> finally:
> shutil.rmtree(tempdir)
>
>
> branch.Branch.hooks.install_named_hook('pre_change_branch_tip',
> run_test_suite_hook, "Check the testsuite passes before the "
> "change is made.")

> which looks for a configuration option giving the command to run to
> validate the commit, for a poor-man's PQM for my local work.

Better than that; its useful for servers that have [relatively] cheap
precommit test actions to enforce.

> I'd like to use this for my bzr plugins, however the usual way
> I test those is "bzr selftest -s bp.<plugin name>". However,
> that requires the tree I am committing is the one that bzr will
> load, which isn't always going to be the case. In particular,
> the above code exports to a temporary directory to run the tests,
> so it won't be something loaded by bzr.
>
> So, I need a way to run the tests for a plugin, but also pass
> an arbitrary filesystem location to use the tests from, and have
> the plugin appear as the correct name to bzr.

I would keep the layers separate. Reinvoking bzr is more than a little
ugly - and it a circular definition which means it won't work anyway.

so when you export, export to a subdir of the temporary directory; then
set the env when you invoke the sub process.

> My code does this by setting BZR_PLUGIN_PATH based on __file__,
> so that it will always be testing the code that you are running
> it from, as well as a symlink to get it to appear as the correct
> name, regardless of the directory name.

This is what is broken - you aren't testing the commit, you're testing
the original tree which isn't the same thing.

> Running the particular tests of a plugins isn't hard thanks to
> "-s", but loading a plugin from an arbitrary location with a specific
> name still is (As Jelmer stated as well, BZR_PLUGIN_PATH isn't
> ideal because you often have many branches of the same code in
> adjacent directories).

We could add improvements in this area, yes.

> Perhaps this bug ...

On Fri, 2009-04-17 at 00:01 +0000, Robert Collins wrote:
> so when you export, export to a subdir of the temporary directory; then
> set the env when you invoke the sub process.

Well, not really. That would work fine if the test plugin were for
testing bzr plugins, but it is supposed to be more generic than that.

A similar issue comes up for running the test suite of a plugin while
building the debian package of it, I have tried to crack this problem
before for that and failed to get something that works in all the
ways that it needs to.

> > My code does this by setting BZR_PLUGIN_PATH based on __file__,
> > so that it will always be testing the code that you are running
> > it from, as well as a symlink to get it to appear as the correct
> > name, regardless of the directory name.
>
> This is what is broken - you aren't testing the commit, you're testing
> the original tree which isn't the same thing.

I don't understand what you mean by this.

Thanks,

James

On Fri, 2009-04-17 at 00:18 +0000, James Westby wrote:
> On Fri, 2009-04-17 at 00:01 +0000, Robert Collins wrote:
> > so when you export, export to a subdir of the temporary directory; then
> > set the env when you invoke the sub process.
>
> Well, not really. That would work fine if the test plugin were for
> testing bzr plugins, but it is supposed to be more generic than that.

It is configured by config options though, so surely you can set that
per trunk.

> A similar issue comes up for running the test suite of a plugin while
> building the debian package of it, I have tried to crack this problem
> before for that and failed to get something that works in all the
> ways that it needs to.

bzr plugin-info may help.

-Rob

On Fri, 2009-04-17 at 00:31 +0000, Robert Collins wrote:
> It is configured by config options though, so surely you can set that
> per trunk.

Yes, I could, but as I said, this problem occurs in other situations as
well, so it would be nice to have a fix that works everywhere.

> > A similar issue comes up for running the test suite of a plugin while
> > building the debian package of it, I have tried to crack this problem
> > before for that and failed to get something that works in all the
> > ways that it needs to.
>
> bzr plugin-info may help.

Thanks, I'll take a look.

James

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

>>> My code does this by setting BZR_PLUGIN_PATH based on __file__,
>>> so that it will always be testing the code that you are running
>>> it from, as well as a symlink to get it to appear as the correct
>>> name, regardless of the directory name.
>> This is what is broken - you aren't testing the commit, you're testing
>> the original tree which isn't the same thing.
>
> I don't understand what you mean by this.
>
> Thanks,
>
> James
>

branch.basis_tree() is a committed tree, which is the tree *just before*
the one you are adding.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknn38sACgkQJdeBCYSNAAMKnwCfZp0WTULoZzWvrzBSi+D43cig
c7AAoKgkOr6pHwsKxDdKCuL8Af4xPwNy
=ff3s
-----END PGP SIGNATURE-----

On Fri, 2009-04-17 at 01:47 +0000, John A Meinel wrote:
>
> branch.basis_tree() is a committed tree, which is the tree *just
> before*
> the one you are adding.

Oh, thats a mistake in my suggestion; sorry.

Anyhow, I meant that having a symlink to a fixed path will only help if
you export to that path, but you export to a random path.

Seems to me much easier to export to a random place, using the plugin
metadata plugin-info can get to name the subdir.

-Rob