to.etc.lexer
Class TextTokenizerBase

java.lang.Object
  extended by to.etc.lexer.TextReaderBase
      extended by to.etc.lexer.ReaderScannerBase
          extended by to.etc.lexer.TextTokenizerBase

public abstract class TextTokenizerBase
extends ReaderScannerBase

A tokenizing lexer which passes information into LexerToken instances, allowing tokens to be pushed back where needed. This does no token management (responsibility of caller) because that performs way better when using token instances.

Author:
Frits Jalvingh Created on Oct 28, 2009

Field Summary
 
Fields inherited from class to.etc.lexer.ReaderScannerBase
T_BASE_LAST, T_COMMENT, T_EOF, T_IDENT, T_IPADDR, T_NUMBER, T_STRING
 
Constructor Summary
TextTokenizerBase(java.lang.Object source, java.io.Reader r)
           
 
Method Summary
 int nextToken(LexerToken t)
          Checks if the token queue contains tokens, and if so returns the topmost one.
protected abstract  int nextTokenPrim()
          Handles actual token generation using the reader's globals.
 LexerToken popToken()
           
 void pushToken(LexerToken t)
          Push the specified token into the token queue.
 void pushTokenCopy(LexerToken t)
           
 int tokenStackSize()
           
 
Methods inherited from class to.etc.lexer.ReaderScannerBase
error, getSourceLocation, getTokenColumn, getTokenLine, isAllowNewlineInString, isIdChar, isIdStart, scanIdentifier, scanNumber, scanSimpleString, setAllowNewlineInString, skipWs, skipWsNoNL, startToken
 
Methods inherited from class to.etc.lexer.TextReaderBase
accept, accept, append, append, clearCopy, copy, copy, getCopied, getCurrentColumn, getCurrentLine, getSource, LA, LA, sb
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

TextTokenizerBase

public TextTokenizerBase(java.lang.Object source,
                         java.io.Reader r)
Method Detail

nextTokenPrim

protected abstract int nextTokenPrim()
                              throws java.lang.Exception
Handles actual token generation using the reader's globals.

Parameters:
t -
Returns:
Throws:
java.lang.Exception

pushToken

public void pushToken(LexerToken t)
Push the specified token into the token queue. The token becomes OWNED by this - it should not be re-used.

Parameters:
t -

pushTokenCopy

public void pushTokenCopy(LexerToken t)

popToken

public LexerToken popToken()

tokenStackSize

public int tokenStackSize()

nextToken

public int nextToken(LexerToken t)
              throws java.lang.Exception
Checks if the token queue contains tokens, and if so returns the topmost one. If not this scans input for the next token.

Parameters:
t -
Returns:
Throws:
java.lang.Exception