GigaToken tokenizes at 24 GB/s by replacing regex with SIMD

The headline is "~1000x faster language model tokenization" and it sounds like one of those benchmark claims that turns out to be measuring something nobody does in production. This one is not that. The numbers are real, the speedup is consistent across hardware, and the technique behind it is something that arguably should have been done years ago. It just required somebody to care about tokenization enough to rewrite it from scratch.

GigaToken is a tokenizer library from Marcel Rod that dropped on the Hacker News front page on July 22, 2026, picking up 388 points and 77 comments. It claims to tokenize text at gigabytes per second, compared to the megabytes per second that HuggingFace's tokenizers and OpenAI's tiktoken both produce. Both of those are already written in Rust with multithreaded C under the hood. This is not a case of a fast tool beating a slow tool. It is a fast tool beating another fast tool by a factor of a thousand.

The numbers

The benchmark is simple: tokenize the OpenWebText training sample (11.9 GB) using the same tokenizer on different libraries and measure throughput. The test was run on three machines: a dual-socket AMD EPYC 9565 (144 cores), an Apple M4 Max (16 cores), and an AMD Ryzen 7 9800X3D (16 cores).

Why such variance between tokenizers? GPT-2 is a small simple tokenizer with 50k vocabulary, so the BPE merge table is easier to cache. Llama 3.1 has a 128k vocabulary and more complex pretokenization. But even the "worst" speedup on this chart, Gemma 4 at 14x on the EPYC, is still a 14x improvement over the HuggingFace tokenizer on the same hardware. The best case for GPT-2 is 1,268x on the M4 Max.

To put the throughput in perspective: at 24 GB/s on the EPYC, you could tokenize all 130 trillion tokens of Common Crawl in about 6.5 hours. That is the entire internet. The same job through HuggingFace would take months.

How it works

The bottleneck in every modern BPE tokenizer is not the byte-pair merging itself. It is the pretokenization step: the regex pattern that splits raw text into word-like chunks before BPE merges are applied. Nearly every tokenizer library uses the same regex engine (the regex or fancy-regex Rust crate) for this step. These regex engines are well-optimized but they are general-purpose. They do not know that the pretokenization pattern is always the same few rules: split on whitespace, handle punctuation, split camelCase, keep numbers together.

GigaToken replaces the regex engine entirely with hand-written SIMD implementations of the exact same pretokenization rules. Instead of running a general regex VM on each byte, it loads 8 or 16 or 32 bytes at a time into vector registers, does the splitting comparisons in parallel across all lanes, and produces the chunk boundaries in one pass. This is the same idea behind simdjson and simdutf: take a regex that everyone runs thousands of times over millions of bytes, and replace it with a vectorized function that does the same thing in hardware.

The second big win is caching. Natural text has a massively long-tailed word distribution. In a 12 GB corpus, a small number of words appear millions of times and most of the rest appear once. GigaToken builds a cache of pretoken-to-token-ID mappings using concurrent data structures that avoid the thread communication overhead of shared-state caches. If a word has been seen before, its token sequence is looked up instead of recomputed. The README calls this "a very hard problem" because the cache grows quickly and the distribution is skewed, which is exactly the kind of problem where a custom data structure beats a general one.

The third source of speed is minimizing Python overhead. When using the Gigatoken API (not the compatibility wrappers), the Rust code reads files directly and handles all iteration internally. The Python layer just starts the job and receives the result. The compatibility mode that mimics the HuggingFace API is still much faster than the original, but leaves a 10 to 20% performance penalty on the table because it must round-trip through Python data structures.

The compatibility story

The part that makes GigaToken practical rather than just impressive is that it ships a compatibility layer. You can take an existing HuggingFace tokenizer object, wrap it in gt.Tokenizer(hf_tokenizer).as_hf(), and use the result as a drop-in replacement. Token IDs match exactly. The README says "a substantial amount of effort" went into making outputs identical, which is the kind of sentence that tells you someone actually did the boring validation work, not just the exciting optimization.

The compatibility mode is slower than the native Gigatoken API because it still has to honor HuggingFace's data flow, but it is still hundreds of times faster than the original in most cases. For a project that already has a tokenization pipeline built around HuggingFace, this is the path of least resistance to a 10x speedup with a two-line change.

What is missing

Not everything works yet. WordPiece (used by BERT-style models) is not supported. SentencePiece-based tokenization (Google models) is present but not as optimized as the BPE path. Windows has not been tested much. File sink output to directly write token IDs to disk is not implemented. The author explicitly flags these as known issues in the README rather than hiding them.

The more fundamental caveat is one the README acknowledges in its FAQ: "Did you just way over-optimize for a specific CPU and tokenizer?" The answer given is "I way over-optimized for every combination of these," accompanied by a claim that results are consistent across CPUs. The benchmarks back this up to a degree. The speedup ratios do vary a lot by tokenizer (9.6x for Gemma 3, 989x for GPT-2 on EPYC), which means some tokenizers are bottlenecked by BPE merge table size rather than pretokenization, and GigaToken helps less there. If a tokenizer already spends most of its time in the merge step, making pretokenization 1000x faster will not help much. The Gemma tokenizers sit in this bucket: HuggingFace already runs them at 330 to 357 MB/s, which is fast enough to hint that their pretokenization regex is simple, so there is less to gain.

Why I think this matters

Tokenization is the part of the LLM pipeline that everyone takes for granted because it is not the expensive part. Training a model costs thousands of GPU-hours. Tokenization costs CPU-hours, running before the GPU ever sees a token. When tokenization is 1000x faster, it stops being a bottleneck entirely. You can tokenize the training corpus in an afternoon instead of a week. You can re-tokenize when you change the vocabulary without blocking the whole research team. You can tokenize streaming data feeds in real time with a single CPU core instead of needing a rack.

The broader pattern here is one I keep seeing in systems work and find hard to stop thinking about. The slow part is rarely where you think it is. Everyone assumed tokenization was fast enough because the Rust tokenizers run at tens of megabytes per second, which sounds fast. It turns out the same task could be running at gigabytes per second all along, and nobody noticed because the regex engine was a black box that nobody thought to open. The same was true of JSON parsing before simdjson. It was true of UTF-8 validation before simdutf. Each time, the answer was the same: stop using a general-purpose regex or state machine for a fixed, known pattern, and replace it with vectorized code that does exactly that one thing.

I do not think GigaToken will change how most developers work. Most people are not tokenizing 12 GB of text at a time. But for the teams that are, the ones building data pipelines for pretraining runs, the ones curating datasets, the ones running continuous data refresh for model updates, this is a 10 to 1000x improvement to a step that was quietly eating hours of CPU time. That is the kind of optimization that changes how you architect a pipeline.

The other thing I like about this project is what the author says about AI in the README. There is an AI Use Disclosure section that says the majority of the codebase was hand-written, with AI used only in the final stages for API polish, compatibility porting, and the last 4x of performance from branch elimination and cache tuning. That is an honest accounting of what AI is actually useful for in systems programming: the last 10% of the polish work, not the core algorithm design. The SIMD implementations, the cache hierarchy design, the concurrent data structures: those came from a person who understood the problem. The AI helped with the bolting on at the end.

It is also worth noting that this hit the front page the same day as Mitchell Hashimoto's "Everyone Should Know SIMD" post about how SIMD is not as hard as people think and the common shape of a SIMD loop is something any developer can learn. GigaToken is what happens when you take that idea seriously and apply it to a problem that was quietly bottlenecking an entire field.

If you want to try it without installing anything, the README has a one-liner: uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt -validate. It will tokenize 100 MB of text, compare the output against HuggingFace for correctness, and print the speedup. On my read of the benchmarks, you will see something between 10x and 1300x depending on your CPU. Even at the low end, it is hard to argue with free performance that just drops in.