Gigatoken: World’s Fastest Open-Source LLM Tokenizer
Gigatoken: World’s Fastest Open-Source LLM Tokenizer
Source: https://x.com/i/status/2079642166553608473
📌 Gigatoken is an MIT-licensed, open-source LLM tokenizer by Stanford PhD student Marcel Rød that tokenizes raw text at multi-GB/s on ordinary CPUs. It targets drop-in replacement of Hugging Face tokenizers and OpenAI’s tiktoken by redesigning pretokenization and BPE for SIMD, caching, and low-contention parallelism.
⚡ What it is
Open-source BPE-style tokenizer (MIT) that converts raw text into model tokens at multi-GB/s on CPUs. Drop-in for Hugging Face tokenizers and tiktoken in Python, with its own API and a no-install bench path via `uvx gigatoken bench`.
🚪 Why it matters
Tokenization is the front door of LLM pipelines—training prep, batching, inference, and cost accounting. Large gains over already-multithreaded Rust baselines can cut hours or days off corpus prep and make high-throughput local tooling practical.
🔧 How it gets speed
Bottlenecks were regex pretokenization and repeated BPE on common pretokens—not “Rust vs Python.” Hand-written SIMD state machines, hierarchical pretoken caches, minimized Python round-trips, and multi-arch paths (AVX-512/AVX2/NEON) redesign the hot path.
📊 Performance claims
Author cites ~500–1000× vs HF tokenizers, ~100× vs tiktoken, and ~24 GB/s on dual-socket AMD EPYC (Common Crawl–scale job under seven hours on one server). Independent checks still show large speedups that scale with cores and dataset size.
🧪 Origins & status
Built by Marcel Rød (Stanford PhD; CS336 CA), inspired by simdjson-style systems craft and CS336 work. Early/beta open-source release on GitHub with paper-style focus on SIMD and cache hierarchies for BPE.
Key facts
| Fact | Value |
|---|---|
| Author | Marcel Rød (Stanford PhD student; CS336 CA) |
| License / status | Open source (MIT); early/beta release on GitHub |
| Peak throughput | ~24 GB/s on dual-socket AMD EPYC; Common Crawl–scale \<7 hours on one machine |
| Claimed speedups | ~500–1000× vs Hugging Face tokenizers; ~100× vs OpenAI tiktoken |
| Core techniques | SIMD pretokenization state machines, hierarchical caches, low-contention parallelism, multi-arch (AVX-512/AVX2/NEON) |
| Interface | Drop-in for HF tokenizers / tiktoken; own API; `uvx gigatoken bench \<tokenizer> \<dataset>` |
Details
Gigatoken attacks the same job as Hugging Face’s tokenizers and OpenAI’s tiktoken—byte-pair encoding and related schemes that turn text into model tokens—but redesigns the hot path for modern CPUs. Author Marcel Rød frames it as a Python drop-in replacement plus a native API, with a zero-install benchmark path via `uvx gigatoken bench`.
The work argues that further gains over multithreaded Rust baselines were blocked less by language choice than by pretokenization and redundant BPE. Regex-based word splits and repeated work on common pretokens dominate cost; replacing them with hand-written state machines, SIMD parsing (simdjson-style), hierarchical caching, and careful parallelism unlocks multi-GB/s throughput.
On a dual-socket AMD EPYC host, Rød reports ~24 GB/s and claims Common Crawl–scale tokenization in under seven hours on one server, with author speedups of roughly 500–1000× vs HF and ~100× vs tiktoken. Independent re-benchmarks on smaller or different hosts still show large multipliers (e.g., tens of times faster), though absolute numbers depend on cores and dataset size.
Development grew out of Stanford CS336 (“Language Modeling from Scratch”) and systems-first craft. The project is open source on GitHub under MIT (per coverage), multi-arch (AVX-512/AVX2/NEON), and still early/beta in some writeups, with a paper-style emphasis on SIMD and cache hierarchies for BPE.