to.etc.lexer
Class TextTokenizerBase
java.lang.Object
to.etc.lexer.TextReaderBase
to.etc.lexer.ReaderScannerBase
to.etc.lexer.TextTokenizerBase
public abstract class TextTokenizerBase
- extends ReaderScannerBase
A tokenizing lexer which passes information into LexerToken instances, allowing tokens to
be pushed back where needed. This does no token management (responsibility of caller) because
that performs way better when using token instances.
- Author:
- Frits Jalvingh
Created on Oct 28, 2009
Methods inherited from class to.etc.lexer.ReaderScannerBase |
error, getSourceLocation, getTokenColumn, getTokenLine, isAllowNewlineInString, isIdChar, isIdStart, scanIdentifier, scanNumber, scanSimpleString, setAllowNewlineInString, skipWs, skipWsNoNL, startToken |
Methods inherited from class to.etc.lexer.TextReaderBase |
accept, accept, append, append, clearCopy, copy, copy, getCopied, getCurrentColumn, getCurrentLine, getSource, LA, LA, sb |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
TextTokenizerBase
public TextTokenizerBase(java.lang.Object source,
java.io.Reader r)
nextTokenPrim
protected abstract int nextTokenPrim()
throws java.lang.Exception
- Handles actual token generation using the reader's globals.
- Parameters:
t
-
- Returns:
-
- Throws:
java.lang.Exception
pushToken
public void pushToken(LexerToken t)
- Push the specified token into the token queue. The token becomes OWNED by this - it should not be re-used.
- Parameters:
t
-
pushTokenCopy
public void pushTokenCopy(LexerToken t)
popToken
public LexerToken popToken()
tokenStackSize
public int tokenStackSize()
nextToken
public int nextToken(LexerToken t)
throws java.lang.Exception
- Checks if the token queue contains tokens, and if so returns the topmost one. If not this
scans input for the next token.
- Parameters:
t
-
- Returns:
-
- Throws:
java.lang.Exception