WoW LuaTokenizer addon Dragonflight/Wrath of the Lich King Classic 2022
wow addon LuaTokenizer


Game Version: 7.0.3
Total Downloads: 330
Updated: Jul 23, 2016
Created: Jun 19, 2013
download LuaTokenizerDownload Earlier Versions

Earlier Versions

Name Size Uploaded Game Version Downloads
0.2 release 6.02 KB Jul 23, 2016 7.0.3 88 download LuaTokenizer 0.2 releaseDownload
0.1 release 5.91 KB Jul 23, 2016 7.0.3 37 download LuaTokenizer 0.1 releaseDownload
r33 alpha 7.21 KB Jul 23, 2016 7.0.3 31 download LuaTokenizer r33 alphaDownload
r32 alpha 7.21 KB Jul 23, 2016 7.0.3 32 download LuaTokenizer r32 alphaDownload
r29 alpha 7.19 KB Jul 3, 2013 5.3.0 37 download LuaTokenizer r29 alphaDownload
r27 alpha 7.03 KB Jun 26, 2013 5.3.0 43 download LuaTokenizer r27 alphaDownload
r24 alpha 7.41 KB Jun 25, 2013 5.3.0 33 download LuaTokenizer r24 alphaDownload
r13 alpha 3.12 KB Jun 19, 2013 5.3.0 29 download LuaTokenizer r13 alphaDownload


Share this:

This is a tokenizer for the Lua language. I put it up as a library so others may benefit from my weird love of creating low-level code. It is written to be run in standard (CLI) Lua 5.1 as well as WoW Lua.

The Library is not yet fully featured, but I consider the "API" stable and the tokenizer itself as well. There is a sample parser that outputs HTML in the test folder and some sample WoW Lua parser will follow or will be accessible in the Addon GreaseGoblin.



`array = LuaTokenizer:Tokenize(string[, transform_callback])`

Tokenizes a string string, calls transform_callback for each token and returns an array containing all return values of the callback. If no callback is given, a default callback is used, that returns all arguments as an array.


mixed = transform_callback(token, value, line-start, line-end, char-start, char-end, ...)

The callback is called for every token generated by Tokenize, and is expected to return a value that represents the token for your application. This can be any value, the tokenizer does not constrain this in any way. Note that returning nil will not create an entry in the array returned by Tokenize.


  • value contains the source string that represents the token. If token is not a symbolic name, this value is the same as token
  • token the token name. This can be either a symbolic name for "ERROR", "NEWLINE", "MLSTRING", "COMMENT", "KEYWORD", "ID", "HEXNUM", "NUMBER" or "STRING". In this case value will contain the actual source string of that token.
  • line-start is the line where the token starts [NYI]
  • line-end is the line where the token ends
  • character-start is the character (counting from the beginning of the string) where the token starts [NYI]
  • character-end is the character (counting from the beginning of the string) where the token ends [NYT]
  • ... Additional parameters: For "STRING" this will contain the string delimiter (" or '), and for "MLSTRING" this will contain the = signs, if the opening sequence is only [[ this is an empty string (not nil!).


Add a comment