DeepSpec and the speculative decoding arms race

DeepSeek released DeepSpec this week, an MIT-licensed framework for training and evaluating the "draft models" that power speculative decoding. It went straight to the top of Hacker News with 732 upvotes and 306 comments. The repo includes three algorithms: DSpark (DeepSeek's own), DFlash, and Eagle3. It also includes a complete pipeline from data prep through evaluation, which is more than most research releases bother with.

If you have heard the term "speculative decoding" but never quite clicked on what it means or why people care, this release is a good excuse to walk through the whole thing.

What speculative decoding actually does

Autoregressive language models generate one token at a time. Each token depends on the previous one. The GPU does a big matrix multiplication, produces a probability distribution over the vocabulary, picks a token, and then does the whole thing again. The memory bandwidth bottleneck means the GPU spends most of its time waiting for data to move around, not actually computing. This is why an A100 can do 312 TFLOPS of FP16 math but only generates maybe 30-60 tokens per second on a 70B model. The math hardware is starved.

Speculative decoding sidesteps this by running two models. There is a target model (the big one you actually want outputs from) and a draft model (a much smaller one that runs fast because it fits in cache). The draft model generates several tokens ahead speculatively. Then the target model verifies them all in a single forward pass. If the draft model guessed correctly, you get multiple tokens for the cost of one target-model invocation. If it guessed wrong, you throw away the bad tokens and continue from the last correct one.

The trick is that verification is cheap. The target model was going to process all those tokens anyway. By running the draft model's guesses through the target model's attention layers in parallel, you can check many tokens at once. When the acceptance rate is high, you can get 2-3x speedups with no change in output quality. The math guarantees the output distribution is identical to the target model alone.

Why this matters more now

Two years ago, speculative decoding was a neat research trick. Now it is close to mandatory for anyone running open models in production. The reason is simple: model sizes have grown faster than memory bandwidth. A Qwen3-4B or Gemma-4-12B model barely fits in a single GPU's memory. You are running at the edge of what the hardware can do. Speculative decoding is the most effective way to get more tokens per second out of the same hardware, because it attacks the actual bottleneck rather than trying to squeeze more FLOPS out of silicon that is already compute-rich and bandwidth-poor.

Google introduced the core idea back in 2022 in a paper called "Fast Inference from Transformers via Speculative Decoding." The idea has been floating around since then, but this year it went mainstream. Gemma 4 shipped with multi-token prediction (MTP) heads. Qwen 3.6 shipped with working MTP in llama.cpp before anyone else. Nvidia's Nemotron 3 Super included it too. The frontier model providers have started baking drafting directly into their architectures.

Three ways to build a draft model

DeepSpec includes three algorithms. They differ in how they build and train the draft model.

DSpark is DeepSeek's own approach. The draft model shares the target model's internal state and refers to its parameters, rather than being a completely separate network. This is more memory-efficient than running two independent models, because you do not duplicate the full embedding layers and attention weights. The DSpark paper (included as a PDF in the repo) describes the training recipe and architecture. DeepSeek says DSpark achieves acceptance rates of 60-85% depending on the benchmark, which means 2.5-4x effective throughput gains on their hardware.

DFlash takes a different tack. Originally from a paper at arxiv.org/abs/2602.06036, DFlash focuses on flash-attention-style efficiency in the draft heads, minimizing the overhead of the drafting step itself. The idea is that if drafting is not cheap enough, the overhead eats into your speedup even when acceptance rates are high. A slow draft model that is right 80% of the time can be worse than a fast one that is right 60% of the time, because you spend less time waiting for the draft.

Eagle3 is the third option, adapted from the SpecForge project (Apache-2.0). Eagle3 uses a separate draft model with its own weights but keeps it small enough to run in a single forward pass much faster than the target. The trade-off is more memory usage but potentially better acceptance on diverse outputs, because the draft model is not constrained by the target model's internal representations.

The 38 TB problem

DeepSpec's README includes a detail that stops you in your tracks: the default data preparation pipeline for Qwen3-4B requires roughly 38 TB of storage for the target cache. You read that right. The data preparation step regenerates the target model's answers for a training corpus and builds a cache of the internal states. For a 4B parameter target model, that cache is massive.

This is the part of speculative decoding that most papers gloss over. The inference-speed gains are real, but the training cost is enormous. You are not just training a small model. You are training a small model against the outputs and internal states of a large model, which means you need to run the large model over a massive dataset first. For anyone thinking about training their own draft model, plan for serious compute and storage before you start.

The training itself is more reasonable. DeepSpec's default configs assume a single node with 8 GPUs. You can scale down by reducing CUDA_VISIBLE_DEVICES. Evaluation runs against nine benchmarks: GSM8K, MATH500, AIME25, HumanEval, MBPP, LiveCodeBench, MT-Bench, Alpaca, and Arena-Hard-v2. That is a solid set for measuring whether your draft model actually helps in practice, not just on paper.

The packaging debate

The HN thread had an interesting argument about how draft models should be packaged. Google's Gemma 4 ships MTP heads as a separate file that has to be "welded in" by the inference engine (their word, not mine). Qwen 3.6 and Step shipped MTP heads grafted directly into the main model, sharing internal state. One commenter pointed out that keeping MTP heads separate makes more sense for modularity: you can adjust the draft model independently, and for their own internal serving Google bundles them anyway via LiteRT-LM. Others argued that grafting reduces layer duplication and is simpler for end users who just want to download one file and go.

I come down on the side of bundling. Most people who use a draft model will never train one. The whole point of speculative decoding is that it should be invisible to the user. GGUF files are technically unnecessary compared to loading SafeTensors and configuring a Jinja prompt template by hand, but nobody does that because it is miserable. Bundling the draft model is the same thing. If you want to swap draft models, you should be able to, but the default should be zero-config.

Why DeepSeek keeps publishing what others will not

The HN discussion circled a question that comes up every time DeepSeek releases something: why do they publish detailed papers and code when American labs increasingly do not? The conversation had a few theories.

One is structural. DeepSeek came out of a quantitative hedge fund. Their parent company views AI as infrastructure for trading alpha, not as a product to monetize through API revenue. Publishing does not hurt their business model the way it might hurt OpenAI or Anthropic, whose revenues depend on keeping their models hard to replicate. DeepSeek's parent has since raised around $7B in external funding, so that dynamic might change over time.

Another is market dynamics. In China, open-source distribution is often the only way to reach enterprise clients who will not pay SaaS premiums. The business logic aligns with openness in a way that VC-funded American labs' models do not. A commenter made a comparison I liked: Chinese companies are treating models as shared infrastructure, closer to Linux than to a proprietary database. The money will be in customization niches and support, not per-token margins.

The cynics noted that DeepSeek's publications tend to land right after big announcements from American labs, and the impact on AI stock prices is not exactly accidental. A hedge fund that publishes research showing frontier models are commoditizing? There is a trading thesis embedded in the code release. But whatever the motivation, the code is out there and it works. I will take useful open source from cynical actors over closed source from principled ones.

What this means if you run local models

If you are running open models on your own hardware, speculative decoding is the single most impactful optimization you are probably not using yet. vLLM, llama.cpp, and most major inference engines support it out of the box. You need a draft model that matches your target model, which is where DeepSpec becomes useful: it gives you a recipe for training one, and the pre-trained checkpoints will probably show up on HuggingFace soon if they have not already.

The catch is that speculative decoding helps most when your target model is large enough to be memory-bandwidth-bound. If you are already running a tiny model that fits comfortably in cache and generates 100+ tokens per second, drafting will not do much because the overhead of running the verification pass may exceed the savings. For anything 7B and up on a single consumer GPU, speculative decoding is worth testing.

One gotcha for AMD users: OpenVINO acceleration does not apply to speculative decoding. The draft model runs on the same hardware as the target, and the optimization is about reducing memory round-trips, not about instruction-level acceleration. If you were hoping OpenVINO would speed up your draft model, it will not. PyTorch with ROCm is the path for AMD GPUs, same as it ever was.

Where this goes next

Speculative decoding is moving from a research optimization to a default feature. The fact that Google, DeepSeek, Qwen, and Nvidia all shipped MTP-capable models this year tells you where the industry is heading. Within a year, I expect most production inference endpoints to use some form of drafting by default, transparently.

The open question is how good the draft models can get. Current acceptance rates of 60-85% are decent, but the speedup ceiling is bounded by how many tokens you can draft before the target model's verification cost grows. If someone figures out a draft architecture that hits 90%+ acceptance consistently, that is a big deal. It means 5x+ throughput gains on existing hardware, which directly translates to lower inference costs at scale.

DeepSpec is not going to be the last word on this. But it is the first open framework that gives you the full pipeline: data prep, training, and evaluation. That matters more than any single algorithm improvement, because it lowers the barrier for anyone who wants to experiment. More people trying things means faster progress on the draft model quality problem. I am glad this exists.