ztok — a fast multithreaded tokenizer in Zig that loads tiktoken / HF / SentencePiece and is 2–5× faster

I built ztok, a tokenizer library focused on being fast and format-agnostic for local pipelines. - Loads what you already have -.tiktoken, HF tokenizer.json, SentencePiece.model, TokenMonster, Mistral Tekken. Auto-detected. - Bit-identical to tiktoken / HF / SentencePiece on the equivalence gate, so it's a drop-in. - Faster on the same vocab + same bytes (cl100k vs tiktoken, EPYC 24c/48t): ~2× single-thread, 3.8-5.5× batched (~291-425 MB/s vs ~78). Also faster than HF tokenizers andSentencePiece on their own vocabs. - 8 language bindings over one C ABI - Python, Node, Ruby, Go, Rust.