Gigatoken: GB/s Language-Model Tokenization Benchmarks
Gigatoken: GB/s Language-Model Tokenization Benchmarks
Source: https://github.com/marcelroed/gigatoken/#benchmarks
📌 Gigatoken is an open-source Rust tokenizer (with Python bindings) by Stanford PhD student Marcel Rød that claims multi-GB/s encode throughput—often hundreds to ~1000× faster than Hugging Face Tokenizers and ~50–100×+ vs tiktoken on common LLM vocabularies. Gains come from SIMD, state-machine pretokens, caching, and low-contention parallelism, making large-corpus re-tokenization practical on a single machine.
⚡ What it is
MIT-licensed Rust + Python project (marcelroed/gigatoken). Drop-in alternative to Hugging Face Tokenizers with optional tiktoken-style compatibility for GPT-2, Llama, and similar families.
🚀 Headline speedups
Claimed ~500–1000× vs HF Tokenizers and ~100× vs tiktoken; published ranges often ~280–1,353× vs HF and ~50–681× vs tiktoken. Dual EPYC 9565: GPT-2 encode ~24.5 GB/s vs ~24.8 MB/s HF on 11.9 GB owt_train.txt (~989×).
🔧 How it gets faster
SIMD, hand-written state machines instead of heavy regex pretokens, aggressive pretoken caching, low-contention parallelism, and fewer Python–Rust round-trips—on top of libraries that are already multithreaded Rust.
📦 Why it matters
Tokenization is on the critical path for training and bulk packing. Multi-GB/TB re-encoding that took hours can drop to minutes, making full-crawl tokenization on one machine realistic rather than aspirational.
✅ Independent checks
Third-party runs on smaller OpenWebText slices reproduce a large lead over HF and tiktoken; absolute GB/s scales with cores, memory bandwidth, and dataset size. Compatibility mode keeps HF/tiktoken-style APIs/outputs at a modest cost.
Key facts
| Fact | Value |
|---|---|
| Author / license | Marcel Rød (Stanford PhD); MIT; marcelroed/gigatoken |
| EPYC GPT-2 encode | ~24.5 GB/s vs ~24.8 MB/s HF on 11.9 GB owt_train.txt (~989×) |
| Speedup ranges | ~280–1,353× vs HF; ~50–681× vs tiktoken (setup-dependent) |
| Benchmark platforms | Dual EPYC 9565 (144 cores), Apple M4 Max, Ryzen 7 9800X3D |
| Independent 4-vCPU run | ~278 MB/s vs ~10.6 MB/s tiktoken and ~3.3 MB/s HF on 174 MB OWT slice |
| Benchmark corpus | 11.9 GB owt_train.txt (large multi-document encode, not short-string micros) |
Details
Gigatoken targets the encode path that every training and bulk-inference pipeline must run before GPUs can train or pack long contexts. Hugging Face Tokenizers and OpenAI tiktoken already use multithreaded Rust, so further gains are non-trivial; the project attributes multi-GB/s throughput to SIMD, state-machine pretokens (instead of heavy regex), pretoken caching, low-contention parallelism, and reduced Python overhead.
Official benchmarks encode a fixed 11.9 GB OpenWebText-derived file across three CPUs—dual-socket AMD EPYC 9565, Apple M4 Max, and AMD Ryzen 7 9800X3D—comparing Gigatoken to HF Tokenizers and tiktoken for multiple vocabularies. The standout EPYC figure is ~24.5 GB/s for GPT-2 encode versus ~24.8 MB/s for HF (~989×), with multi-GB/s also reported on M4 Max and the 9800X3D.
Independent reproduction on a 174 MB OWT slice (4-vCPU VM) still showed a large lead (~278 MB/s vs ~10.6 MB/s tiktoken and ~3.3 MB/s HF). Absolute rates scale with core count, memory bandwidth, and dataset size. A compatibility mode preserves exact HF/tiktoken-style APIs and outputs at modest cost—often still a few hundred× faster than HF rather than ~1000× for the native API.