An experimental port of OpenAI's Tokenizer to lua
$ luarocks install tiktoken_core
The Lua module written in Rust that provides Tiktoken support for Lua.
root
