Tokenizing ast nodes

After the ast has been worked on by the refactor package, it needs to be converted back to source code so that it can be saved. Unfortuately the python compiler module looses the comments and whitespace information from the source code when it creates the AST.

To get round this problem, the addtokens module matches scanner tokens (which include comments and whitespace) to the ast nodes that represent them. This allows the source to be output in its original form after the refactoring has taken place.

Tokenhandler base classes

To manage the tokens, each ast node derives from a TokenHandler class. Each tokenhandler implements a 'getTokens()' method which returns its tokens and those of its children. Thus calling getTokens() on a module returns all the scanner tokens for the source to that module.

Because the compiler ast classes are already instantiated when the parser package gets to work on them, this inheritence relationship is created on the fly by modifying the __bases__ attribute of the ast classes.