Gigatoken: GB/s-scale LLM tokenization via SIMD and caching
Gigatoken: GB/s-scale LLM tokenization via SIMD and caching
Source: https://x.com/i/status/2079642157489721719
📌 Gigatoken is an open-source, high-performance tokenizer for language models that uses SIMD-friendly pretokenization and aggressive caching to reach hundreds of MB/s to tens of GB/s. Released July 2026 by Stanford PhD student Marcel Rød, it aims to be a drop-in replacement for Hugging Face tokenizers and OpenAI tiktoken.
⚡ What it is
Open-source BPE tokenizer designed as a drop-in for Hugging Face and tiktoken. Targets the text-to-token bottleneck in data prep and inference, with a native API plus compatibility layers and quick try via uvx.
🔧 How it gets fast
Replaces regex pretokenization with a hand-written, SIMD-capable state machine and a carefully engineered cache hierarchy so previously seen pretokens are looked up instead of recomputed—inspired by simdjson and Handmade software ethos.
📈 Claimed performance
Roughly 500–1000× faster than Hugging Face and ~100× vs tiktoken on typical setups. Peak published numbers include ~24 GB/s on dual-socket AMD EPYC and Common Crawl in under ~7 hours on one machine.
🎯 Why it matters
Tokenization sits at the front of dataset construction, training packing, and serving. When it lags I/O or model compute, large corpora get expensive to prepare and long prompts add latency before the first token.
✅ Independent checks
Small-scale reproduction (e.g. KrabArena) has shown large speedups such as tens of times over tiktoken and Hugging Face on modest multi-vCPU VMs, supporting the direction of the claims though absolute throughput varies by CPU and data.
Key facts
| Fact | Value |
|---|---|
| Author | Marcel Rød (Stanford PhD; advisors include Tatsu Hashimoto and Jure Leskovec) |
| Released | July 2026 |
| Speedups (claimed) | ~500–1000× vs Hugging Face; ~100× vs OpenAI tiktoken |
| Peak throughput | ~24 GB/s on dual-socket AMD EPYC; Common Crawl \<~7 hours on one machine |
| Core techniques | SIMD pretokenizer state machine; aggressive pretoken-result caching |
| Compatibility | Drop-in for Hugging Face / tiktoken in Python; native API; bench via uvx gigatoken |
Details
Gigatoken addresses a long-standing gap: even multithreaded Rust tokenizer baselines often run far slower than modern CPUs can process comparable workloads, while tools like simdjson routinely hit several GB/s. Rød’s design treats pretokenization as a hand-tuned, CPU-specific path rather than regex, then layers caching so repeated pretokens avoid recomputation—unlike some libraries where caching can hurt performance.
The project is framed around “SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs.” It documents both its own API and Hugging Face / tiktoken compatibility so existing Python pipelines can swap implementations with minimal change.
Absolute throughput depends on hardware and corpus size, but independent checks have reproduced order-of-magnitude wins on smaller slices, reinforcing that tokenizer redesign—not just more threads—can unlock GB/s-scale text encoding for LLM data pipelines.