Gigatoken: BPE Tokenizer in Rust Encodes Text at 24.53 GB/s, Up to 989x Faster Than HuggingFace Tokenizers

Stanford PhD student Marcel Rød released Gigatoken, a byte-pair encoding (BPE) tokenizer written in Rust with Python bindings, under an MIT license. On a 144-core AMD EPYC 9565 dual-socket system, Gigatoken processed the 11.9 GB owt_train.txt corpus at 24.53 GB/s, compared to HuggingFace tokenizers at 24.8 MB/s (989x slower) and OpenAI's tiktoken at 36.0 MB/s (681x slower). On an Apple M4 Max with 16 cores, it achieved 8.79 GB/s, outperforming HuggingFace by 1,268x and tiktoken by 140x. The performance gains come from a hand-written pretokenizer that uses SWAR (SIMD Within A Register) techniques to check 8 bytes at once with branchless arithmetic, achieving 1,049 MiB/s for GPT-2 pretokenization — a 22.3x improvement over regex-based approaches. Additionally, pretoken caching avoids recomputation of previously seen words. Compatibility mode with existing HuggingFace or tiktoken tokenizers achieves 200–300x speedup due to Python overhead. Independent benchmarks on KrabArena with a 4-vCPU VM confirmed a median throughput of 277.8 MB/s, outperforming tiktoken by 26.2x and tokenizers by 83.4x. The library supports 23 tokenizer families including GPT-2, Llama 3–4, Qwen 2–3.6, DeepSeek V3/R1/V4, GLM 4/5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma, and Mistral, but SentencePiece tokenizers see only 7–22x speedup and WordPiece is unsupported.