This is a tokenizer for the Lua language. I put it up as a library so others may benefit from my weird love of creating low-level code. It is written to be run in standard (CLI) Lua 5.1 as well as WoW Lua.

The Library is not yet fully featured, but I consider the "API" stable and the tokenizer itself as well. There is a sample parser that outputs HTML in the test folder and some sample WoW Lua parser will follow or will be accessible in the Addon GreaseGoblin.

API

LuaTokenizer:Tokenize(...)

`array = LuaTokenizer:Tokenize(string[, transform_callback])`

Tokenizes a string string, calls transform_callback for each token and returns an array containing all return values of the callback. If no callback is given, a default callback is used, that returns all arguments as an array.

transform_callback(...)

mixed = transform_callback(token, value, line-start, line-end, char-start, char-end, ...)

The callback is called for every token generated by Tokenize, and is expected to return a value that represents the token for your application. This can be any value, the tokenizer does not constrain this in any way. Note that returning nil will not create an entry in the array returned by Tokenize.

Arguments:

value contains the source string that represents the token. If token is not a symbolic name, this value is the same as token
token the token name. This can be either a symbolic name for "ERROR", "NEWLINE", "MLSTRING", "COMMENT", "KEYWORD", "ID", "HEXNUM", "NUMBER" or "STRING". In this case value will contain the actual source string of that token.
line-start is the line where the token starts [NYI]
line-end is the line where the token ends
character-start is the character (counting from the beginning of the string) where the token starts [NYI]
character-end is the character (counting from the beginning of the string) where the token ends [NYT]
... Additional parameters: For "STRING" this will contain the string delimiter (" or '), and for "MLSTRING" this will contain the = signs, if the opening sequence is only [[ this is an empty string (not nil!).