Speculative decoding on its own silicon

A large language model writes one token at a time. For one person, each token is a full forward pass through hundreds of billions of parameters, and at that batch size the pass is bound by reading weights from memory, not by arithmetic: the GPU’s compute sits mostly idle while the weights stream past. Speculative decoding puts the idle arithmetic to work. A small draft model proposes the next few tokens, and the large model, the target, verifies these speculative tokens in one pass, keeping those that match what it would have written itself, up to the first that doesn’t. Every token the user sees is still the target’s own; the answer simply arrives in fewer passes. It is now standard at the frontier: OpenAI, Anthropic and Google all serve their closed frontier models with speculative decoding, using a drafter of some kind.

Where the draft runs gets less attention, even though it determines how much of the gain survives. In common GPU-based designs, the drafter shares the target’s accelerator and often depends on the target’s internal state. We took a different approach: Polyglot-DC1-Draft uses one model architecture co-designed with Exstill silicon, with target-specific weights distilled and aligned for five well-established open models. The target stays on the GPU; the draft runs on its own chip. This post explains why that separation matters, what purpose-built distillation adds at the same model size, and how the gains hold as more users are served.

How speculative decoding works

Each round has four steps. The draft model proposes K speculative tokens, one at a time, and sends them to the GPU. The target verifies all K in one forward pass. For one person, that pass costs about what writing a single token costs, because it is limited by reading the target’s weights rather than arithmetic. Verification accepts a prefix of the proposed tokens and stops at the first rejection. The pass also yields one token from the target: at a miss, its own token replaces the rejected proposal; when all K proposals match, it generates the token after them. Each pass therefore yields the accepted draft tokens plus one target token.

The result is fewer target forward passes without changing the target’s behavior. At temperature 0, verification uses greedy comparison; at non-zero temperatures it uses speculative sampling (Leviathan et al., 2023; Chen et al., 2023), accepting and rejecting proposals so that the output has exactly the target’s own sampling distribution. Greedy outputs are unchanged; sampled outputs preserve the distribution, though not necessarily an identical sequence. The measure of a draft is therefore its mean accepted length: the number of tokens each verification pass yields, including the target’s own token. A draft averaging 4.4 accepted tokens per pass lets the target advance by 4.4 tokens per pass, before accounting for drafting cost and the extra work of verification.

How speculative decoding is done on GPUs today

The first speculative decoding papers (Leviathan et al., 2023; Chen et al., 2023) used standalone draft models: small language models, often from the target’s own family, run before each verification pass. But on a GPU, even a small standalone drafter pays the per-token overhead described in part 1. On a B300, Qwen2.5-0.5B decodes at 793 tokens a second for one person, or 1.3 ms per token. Six drafted tokens would take about 7.6 ms, four times longer than gpt-oss-120b’s whole 1.9 ms step. In this setup, a standalone GPU drafter costs more than it saves. That is why many production GPU drafters took another shape: they attached themselves to the target.

EAGLE-3 (Li et al., 2025) drafts from the target model’s own hidden state. Its drafter is a single decoder layer with a language-model head that reads features fused from several of the target’s layers, so it reuses computation the target has already done and a draft step costs a fraction of a millisecond; trained to draft several steps ahead, it reports speedups of up to 6.5x at one user. EAGLE-3.1, from May 2026, keeps the design and adds a normalization scheme (Eldenk et al., 2026): a post-norm on the drafter’s hidden state and an RMSNorm on each target state before fusion, so the drafter’s state stops growing with the depth of the chain, which raises acceptance and slows its decay at depth and at long context. NVIDIA’s EAGLE-3 head for gpt-oss-120b achieves an accepted length of 3.28 on SPEED-Bench coding at a speculation length of K = 7; LightSeek’s EAGLE-3.1 head for Kimi K3, built with multi-head latent attention to match its target, achieves an accepted length of 3.10, the bonus token included, at a speculation length of K = 3, and on its own serving stack it decodes 2.15x faster at one user and 1.51x at sixteen.

DFlash (Chen et al., 2026) replaces the token-by-token drafter with a block diffusion model that proposes a whole block in a single forward pass, conditioned on context features from the target; it reports up to 2.5x the speedup of EAGLE-3. DSpark (Cheng et al., 2026) pairs a parallel drafting backbone with a light sequential module, and schedules the verification length per request from the acceptance it expects and the engine’s throughput profile; in DeepSeek-V4’s serving system it raises per-user speed by 60 to 85% over the MTP-1 baseline at matched throughput. Both are designed for the GPU’s economics: fewer, wider passes, and verification sized to the batch.

In the GPU implementations compared here, two features recur. The drafter is tied to the target, through its hidden states or serving engine, so supporting a new target typically calls for a new drafter. And draft generation runs on the target’s GPUs, sharing their compute and scheduling resources with verification. The first limits reuse across models. The second adds overhead precisely where speculative decoding is supposed to save time, particularly as more requests share the GPU. Both motivate a different design.

Polyglot-DC1-Draft: one architecture, five open models

Polyglot-DC1-Draft is that different design: a standalone 1.7B draft model built on the shared Polyglot-DC1 architecture described in part 5. We co-designed the architecture with Exstill silicon, then distilled and aligned its weights separately for each target. Proprietary techniques let each draft operate in its target’s tokenizer and vocabulary without reading the target’s internal state. The architecture stays the same across five well-established open models:

Table 1. One draft architecture, five open models

Mean accepted length per verification pass at K = 6, measured with vLLM v0.32.0.

Across these targets, the draft architecture and serving integration remain fixed. Switching targets requires different distilled weights and the corresponding tokenizer, not a new chip. A sixth target is a training run, not a new hardware design.

Why draft on the Exstill chip

Drafting sits before every verification pass, so its latency directly affects the speedup. On our pre-silicon data-center design, each stream decodes at 19,921 tokens a second, about 50 µs per token. Drafting 14 tokens therefore takes 0.7 ms, compared with 2.3 ms for gpt-oss-120b to verify them on a DGX B300. The chip can also prepare the next draft while the GPU verifies the current one, overlapping much of the drafting work with verification.

Because the draft does not need the target’s hidden states, communication is small: only token IDs cross the link, tens of bytes a round. Drafting stays off the GPU. To measure the cost of sharing it instead, we ran a draft alongside gpt-oss-20b on the same RTX 5090: target decoding slowed by 2.3 to 2.8 times as the two contended for memory and scheduling resources.

And the chip holds twelve streams at once, so one card drafts for twelve people, for any of the five targets, by loading the weights for that target. Each stream’s slot is sized for a 16K-token context, but the draft is not bound by it: proprietary techniques, the subject of a filed provisional, let it keep drafting at contexts many times longer, for agents and long conversations, with its memory per stream fixed, so twelve long sessions fit the card as well as twelve short ones.

Off the shelf, or co-designed and distilled for the target

Moving drafting off the GPU addresses where the work runs. The next question is how well the draft predicts its target. An obvious alternative to a purpose-built drafter is a small open model from the target’s own family. For Qwen3-Next-80B-A3B-Thinking, Qwen3-1.7B is a particularly strong candidate: it shares the tokenizer and was distilled by Qwen from Qwen3-32B and Qwen3-235B-A22B, first off-policy and then on-policy, on top of 36 trillion tokens of pretraining (Yang et al., 2025). No off-the-shelf model is better placed to draft for it.

Polyglot-DC1-Draft for Qwen3-Next has the same parameter count and shares nothing else with it. Its architecture is Polyglot-DC1’s, found by exploring the model and the chip as one design space, and its weights are distilled from Qwen3-Next’s own outputs and aligned to its choices. Figure 2 puts the two side by side, both drafting on the Exstill chip for Qwen3-Next, scored on the same answers in the same way, on SPEED-Bench and on the nine datasets DSpark reports on.

Across the nine datasets, ours yields 4.18 accepted tokens per pass against 3.32 at K = 6, a 26% improvement. With a tree of 64, the advantage is 34% (7.42 against 5.52). End to end, that translates to 4.7x speedup against 3.4x: 39% faster decoding for the same target on the same chip, with ours ahead on every row. The largest gaps are on mathematics and code: GSM8K reaches 7.0x against 5.2x, and MBPP 5.4x against 3.1x.

Same size, same tokenizer, same target: what separates the two is the model, its architecture and what it was trained to do. A draft is not judged by its own quality but by how often its speculative tokens survive the target’s verification, which is a property of the pair. Qwen3-1.7B was distilled to be a good model in its own right; a drafter is distilled to agree with one target, token by token, which the literature frames as aligning the draft’s distribution to the target’s under the verification rule (Zhou et al., 2024). We have in-house techniques for distilling and aligning drafts that do this effectively; they are not described here.

Figure 2 isolates the benefit of the model: both drafts are costed on the same chip at the same per-token speed. Co-design also changes what silicon can be built around the model. In our pre-silicon estimates, a chip built around Qwen3-1.7B as released uses 2,995 µJ per token, compared with 523 µJ for the Exstill data-center design running Polyglot-DC1, or 5.7x as much energy. The approximately 50 µs per-token draft speed reported earlier comes from that co-designed model–silicon pairing. Both hardware results are pre-silicon estimates.

Five targets, one architecture, one chip

Qwen3-Next is only one of the five targets. Each Polyglot-DC1-Draft uses the same architecture and alignment approach, with weights distilled from its own target. All five run on the same silicon: changing the target means loading the corresponding weights. Yet the targets differ enormously, from 80B parameters to 2.8T, and from 3B active to 104B, with different tokenizers and reasoning behavior.

Figure 3 replays one SPEED-Bench coding prompt for each of them. Both lanes write the same text, the target’s own output at its maximum reasoning effort, its reasoning in grey and then its answer: on the left at the target’s own speed on NVIDIA GPUs, on the right with its draft on the Exstill chip. The words don’t change, only how soon they arrive. Pick a target, or slow the replay down: with Exstill, the fastest answers arrive in under a second.

Every target gets faster, by 3.5x to 6.3x for one person on SPEED-Bench coding. Qwen3-Next gains the most, from 275 tokens a second to 1,738, so its whole answer of about 1,400 tokens arrives in 0.85 s instead of 5.18 s. gpt-oss-120b, already the fastest of the five alone at 526 tokens a second, reaches 2,365, 4.5x, and finishes in 0.63 s instead of 2.70 s. The trillion-parameter models gain too: DeepSeek-V4-Pro-0813, at 1.6T, goes from 100 tokens a second to 411, 4.1x, and answers in 3.29 s instead of 13.31 s; Kimi K3, the largest at 2.8T, answers in 3.89 s instead of 13.40 s. One draft architecture, distilled and aligned for each target, makes every one of them several times faster, from 80B parameters to 2.8T, without changing a word it writes.

Why the lead holds as users are added

The sandbox shows a single user, but production serving rarely stops there. As concurrency grows, verification has more tokens to process, and the GPU’s memory and compute balance changes. That can reduce speculative decoding’s advantage. It also makes the location of draft execution more consequential.

The reason is what a decode step on a GPU spends its time on. For one user, a step is bound by memory: the GPU streams the model’s weights from HBM and does only a few operations for each byte it reads, so verifying K + 1 tokens in the pass instead of one uses arithmetic that would otherwise sit idle, and costs little more than a plain step. Each user added puts more tokens through the same pass, and with speculation each user brings K + 1 of them. The operations per byte climb toward what the GPU’s compute can sustain, and past that point every verified token costs time of its own. Two more costs grow with the tokens in a pass. All five targets here are mixture-of-experts models: each token is routed to a few of many experts, so the more tokens a pass verifies, the more expert weights it has to read. And every verified token attends over its user’s whole context, so attention work grows with users, positions and context length.

When drafting also runs on the GPU, its own compute and scheduling costs add to each step. In Figure 4, the GPU-based methods shown decline from between 1.6x and 3.0x at one user to between about 1x and 2x at sixteen.

Ours fades too, from a higher start and by less: gpt-oss-120b from 4.5x to 3.1x, DeepSeek-V4-Pro-0813 from 4.1x to 2.8x and Kimi K3 from 3.6x to 3.0x, ahead of every GPU drafter shown at every number of users. The chip removes the drafter’s share of the cost: the draft runs on its own silicon, twelve streams to a card and in two waves at sixteen users, and drafts the next round while the GPU verifies this one. The verification stays on the GPU, and in our simulation the experts are most of its growth: a pass costs about 1.2 to 1.5 plain steps at one user and 1.4 to 2.3 at eight or sixteen. Kimi K3’s curve turns up past eight users because by then a plain step already reads many of its experts, so the extra tokens add relatively less.

This is also where the draft’s quality matters most. For one user, verifying a long run of proposals can cost little more than a plain step, so even weak proposals are relatively cheap. As the GPU fills, each verified token adds work, and what matters is how many of those tokens survive verification. The best speculation length therefore depends on batch size (Su et al., 2023). Over the nine datasets of Figure 2, at K = 6, Qwen3-Next keeps 4.18 tokens per pass from our draft and 3.32 from Qwen3-1.7B: 0.60 of the seven tokens a pass can yield, against 0.47. Drafting two tokens further, to K = 8, adds 0.46 kept tokens per pass with ours and 0.22 with Qwen3-1.7B. In other words, those extra speculative positions are about twice as likely to be useful with ours, even as verification grows more expensive. That is why the chip can continue speculating deeply as users are added: the simulation re-chooses draft length and shape at every concurrency level, and for gpt-oss-120b it still verifies a tree of 12 tokens at sixteen users, down from a chain of 14 at one.

Two assumptions carry Figure 4. First, ours is simulated: measured acceptance, with the target’s verification costed as its plain step, plus the experts the extra tokens wake, plus the extra tokens at the engine’s prompt-processing rate. Second, the GPU drafters are as published, each on its own serving engine (TensorRT-LLM, vLLM or SGLang), workload and draft length.

The draft belongs on its own silicon

Speculative decoding’s benefit comes down to two things: how many proposed tokens survive verification, and how much it costs to draft and check them. Polyglot-DC1-Draft addresses both. One architecture, co-designed with Exstill silicon and distilled and aligned for each target, accelerates five open models from 80B parameters to 2.8T by 3.5x to 6.3x for one person on coding workloads in our pre-silicon projections, while preserving the target’s output distribution. Drafting runs on its own chip, leaving the GPU to verify, and the lead holds from one user to sixteen. Because the architecture remains fixed, the next target needs new weights, not a new chip.

The target stays on the GPU. The draft gets silicon of its own.