Get 0.05 TON 💎Earlier Versions
| Name | Size | Uploaded | Game Version | Downloads | |
| 0.2 release | 6.02 KB | Jul 23, 2016 | 7.0.3 | 88 |  Download |
| 0.1 release | 5.91 KB | Jul 23, 2016 | 7.0.3 | 37 | Download |
| r33 alpha | 7.21 KB | Jul 23, 2016 | 7.0.3 | 31 | Download |
| r32 alpha | 7.21 KB | Jul 23, 2016 | 7.0.3 | 32 | Download |
| r29 alpha | 7.19 KB | Jul 3, 2013 | 5.3.0 | 37 | Download |
| r27 alpha | 7.03 KB | Jun 26, 2013 | 5.3.0 | 43 | Download |
| r24 alpha | 7.41 KB | Jun 25, 2013 | 5.3.0 | 33 | Download |
| r13 alpha | 3.12 KB | Jun 19, 2013 | 5.3.0 | 29 | Download |
Description
This is a tokenizer for the Lua language. I put it up as a library so others may benefit from my weird love of creating low-level code. It is written to be run in standard (CLI) Lua 5.1 as well as WoW Lua.
The Library is not yet fully featured, but I consider the "API" stable and the tokenizer itself as well. There is a sample parser that outputs HTML in the test folder and some sample WoW Lua parser will follow or will be accessible in the Addon GreaseGoblin.
API
LuaTokenizer:Tokenize(…)
`array = LuaTokenizer:Tokenize(string[, transform_callback])`
Tokenizes a string string, calls transform_callback for each token and returns an array containing all return values of the callback. If no callback is given, a default callback is used, that returns all arguments as an array.
transform_callback(…)
mixed = transform_callback(token, value, line-start, line-end, char-start, char-end, ...)
The callback is called for every token generated by Tokenize, and is expected to return a value that represents the token for your application. This can be any value, the tokenizer does not constrain this in any way. Note that returning nil will not create an entry in the array returned by Tokenize.
Arguments:
valuecontains the source string that represents the token. Iftokenis not a symbolic name, this value is the same astokentokenthe token name. This can be either a symbolic name for"ERROR","NEWLINE","MLSTRING","COMMENT","KEYWORD","ID","HEXNUM","NUMBER"or"STRING". In this casevaluewill contain the actual source string of that token.line-startis the line where the token starts [NYI]line-endis the line where the token endscharacter-startis the character (counting from the beginning of the string) where the token starts [NYI]character-endis the character (counting from the beginning of the string) where the token ends [NYT]...Additional parameters: For "STRING" this will contain the string delimiter ("or'), and for "MLSTRING" this will contain the=signs, if the opening sequence is only[[this is an empty string (not nil!).
Add a comment