Allen Institute for AI

Gigatoken: BPE Tokenizer in Rust Encodes Text at 24.53 GB/s, Up to 989x Faster Than HuggingFace Tokenizers

Gigatoken, a BPE tokenizer written in Rust by Stanford PhD student Marcel Rød, achieves text encoding speeds of 24.53 GB/s on a 144-core AMD EPYC system, outperforming HuggingFace tokenizers by 989x and OpenAI's tiktoken by 681x. The library supports 23 tokenizer families and attributes its speed to a hand-optimized pretokenizer using SWAR SIMD techniques and pretoken caching.

OpenAI Hugging Face Google/DeepMind Meta Alibaba/Qwen DeepSeek Moonshot AI NVIDIA Microsoft Allen Institute for AI Answer.AI Mistral