Inference notes drafted accepted rejected
← All notes

Inference engines · how it works, what it gains, where it costs you

Speculative Decoding

A small model guesses the next few tokens. The big model checks all of them in a single pass and keeps the ones it would have written itself. You get 2–3× faster generation, plus the part people don't believe until they see the proof: mathematically identical output.

Generating a token from a large language model is slow for a reason that has nothing to do with arithmetic. To produce one token, the GPU must read every weight in the model out of memory. For a 70B model in 16-bit that's roughly 140 GB moved across the memory bus, and the actual multiplication that happens with those weights is trivial by comparison. The hardware spends its time waiting on memory, not computing.

Which leads to a strange and exploitable fact: checking five tokens costs almost exactly what generating one costs. The weights get read once either way. If you already have five candidate tokens in hand, you can score all five in a single forward pass, for very nearly the price of scoring one.

You don't have candidate tokens, of course. That's what the small model is for.

Fig 1Why verification is nearly free
READING WEIGHTS FROM MEMORY ARITHMETIC one at a time 4 passes 140 GB 140 GB 140 GB 140 GB verify 4 at once 1 pass 140 GB THE SAME FOUR TOKENS, WITHOUT RE-READING 420 GB The arithmetic block grows with the number of tokens. The memory read, the part that dominates, does not.
Decoding is memory-bandwidth-bound, not compute-bound. Sizes are for a 70B model at 16-bit precision. This single asymmetry is the entire basis of the technique: if verification were as expensive as generation, there would be nothing to win.

The algorithmDraft, verify, keep what survives

The scheme was published independently by Leviathan et al. and Chen et al. in 2023. You run two models: a draft model that's small and fast, and the target model whose output you actually want. One round goes like this.

  1. The draft model generates k tokens the normal slow way. It's small, so this is cheap: call it a tenth the cost of a target pass.
  2. The target model runs one forward pass over the prompt plus all k draft tokens, which gives you the target's probability for every drafted position at once.
  3. Walk the draft left to right. Accept token i with probability min(1, p(xᵢ) / q(xᵢ)), target probability over draft probability. If the target likes the token at least as much as the draft did, it's kept outright.
  4. At the first rejection, throw away that token and every token after it, and resample one token from the residual distribution norm(max(0, p − q)). If nothing is rejected, sample one bonus token from the target directly.

So a round yields somewhere between 1 and k+1 tokens, for the price of one target pass plus k cheap draft passes. Here is a real round.

Fig 2One verification round, token by token
prompt Speculative decoding works because LLM inference is DRAFT MODEL PROPOSES 5 TOKENS " limited" " by" " the" " speed" " of" q = 0.62   p = 0.71 q = 0.88   p = 0.92 q = 0.55   p = 0.41 q = 0.47   p = 0.09 never scored p/q ≥ 1 p/q ≥ 1 accept w.p. 0.75 accept w.p. 0.19 ACCEPT ACCEPT ACCEPT  r=0.31 REJECT  r=0.66 DISCARDED RESAMPLE FROM norm(max(0, p − q)) KEPT: 4 TOKENS FROM ONE TARGET PASS " limited" " by" " the" " memory" ← resampled, not drafted Cost: 5 draft passes + 1 target pass ≈ 1.5 target passes. Baseline cost for 4 tokens: 4 target passes.
Five drafted, three accepted, one corrected, one discarded. The draft wanted "limited by the speed of"; the target model assigned "speed" only 0.09 against the draft's 0.47 and overruled it with "memory". Note what the rejection bought you: not just speed, but the target's answer. The round produced four tokens at roughly 1.5 target passes instead of four.

The guaranteeThe output is not an approximation

This is where people expect a catch, so it's worth being precise: the accept-reject rule above is constructed so that the tokens coming out are distributed exactly as if you had sampled from the target model directly. Not close to it. Identically. The proof is a page long and it's the reason the technique matters: the draft model influences speed and nothing else.

The intuition is in the residual. When the draft proposes a token the target dislikes, you don't just drop it. Dropping it alone would bias the distribution toward tokens the draft happens to favour. You resample from max(0, p − q), the probability mass the target wanted that the draft failed to offer. The rejection step pays back exactly the bias the acceptance step introduced.

A bad draft model makes speculative decoding slow. It cannot make it wrong.

Two caveats on that guarantee, both real. It holds in exact arithmetic; in practice floating-point differences mean a speculative run won't reproduce a non-speculative run token-for-token, even at temperature 0. And several deployed variants deliberately break it: "lenient" or "typical" acceptance relaxes the test to accept more tokens, buying speed with a distribution that is no longer the target's. If a vendor offers you an aggressive speculative mode, that is the question to ask about it.

Projected gainsTwo numbers predict your speedup

What you get depends on the acceptance rate α (how often the draft guesses what the target would have said) and the draft length k. Expected tokens per round is a geometric series:

Expected tokens per verification round E[tokens] = (1 − αk+1) / (1 − α)
Speedup, where c = draft cost / target cost speedup = E[tokens] / (1 + k·c)

The denominator is why longer drafts stop helping. Every extra drafted token costs you whether or not it survives, and the chance it survives decays as αk. There's an interior optimum, and it moves with α.

Fig 3Speedup against draft length, at three acceptance rates
BELOW 1.0×: SLOWER THAN NOT SPECULATING 1.0× 1.5× 2.0× 2.5× 3.0× 3.5× 1 4 8 12 16 DRAFT LENGTH k: TOKENS PROPOSED PER ROUND α = 0.9  →  peak 3.43× at k = 10 α = 0.8  →  peak 2.47× at k = 6 α = 0.6  →  peak 1.67× at k = 3
Computed from the formula above at c = 0.1. A draft model that agrees 90% of the time earns you 3.4×; one that agrees 60% of the time earns 1.7×, and only if you keep the draft short. Push k to 16 at α = 0.6 and you land at 0.96×. You have made the system slower by speculating.

The practical reading: α is a property of your workload and your model pair, and you must measure it rather than assume it. It's high on formulaic text, boilerplate code, and anything that copies from the prompt. It's low on dense reasoning, unusual formatting, and domains your draft model never saw. Measure it on your own traffic, then read k off the curve.

The real riskIt trades compute for latency, and compute isn't always free

Everything above assumes the premise from Fig 1, that verifying extra tokens is nearly free because you're memory-bound. At batch size 1, serving one user, that's true. It stops being true as the batch grows.

Once you're decoding for many sequences at once, the weights you read from memory are already being reused across the whole batch. The GPU is no longer waiting on memory; it's doing real arithmetic, and arithmetic scales with the number of tokens you push through it. Now those k+1 verified positions cost k+1 times the compute, per sequence, and every rejected token is compute you spent on output you threw away.

Fig 4Throughput against batch size
0 0.5× 1.0× 1.5× 2.0× 2.5× SPECULATION WINS SPECULATION LOSES CROSSOVER ≈ BATCH 41 2.40× settles at 0.62×, 38% worse than not speculating 1 4 16 64 256 CONCURRENT SEQUENCES IN THE BATCH (LOG)
A model, not a benchmark. α = 0.8, k = 4, draft 10× cheaper, compute-bound above roughly 64 token-rows per pass. The shape is what matters and it is not controversial: speculative decoding is a latency optimisation that spends throughput. Serving stacks handle this by disabling it above a batch threshold or shortening k dynamically as load rises.

This is the single most important thing to understand before enabling it. Speculative decoding buys latency with FLOPs. If you serve one user on a dedicated GPU, FLOPs are sitting idle and the trade is free money. If you run a busy inference service at high batch, those FLOPs have somewhere better to be, and speculation takes capacity from the queue to make individual requests marginally faster.

The rest of the risksSmaller, but they'll find you

Acceptance rate drifts

α is measured, not configured. It falls when traffic shifts to a new domain, a new language, or a new output format, and your speedup falls with it, silently. Monitor it as a production metric, not a one-time benchmark.

The draft model eats your KV cache

It occupies HBM that would otherwise hold key-value cache, which lowers your maximum batch size. On a memory-tight deployment this cost can exceed the benefit before you measure anything.

Reproducibility against a non-speculative baseline

Output distributions match; individual sequences won't, even at temperature 0. If you have golden-output tests pinned to exact tokens, they will break, and the breakage is not a bug.

Tail latency gets lumpier

Rounds return between 1 and k+1 tokens, so inter-token latency becomes bursty rather than steady. Streaming UIs that felt smooth can start to feel uneven even as the average improves.

If you are turning this onCheapest paths first

Before you provision a draft model, try the version that needs no second model at all. Prompt lookup decoding, also called n-gram speculation, drafts by searching the prompt for a matching prefix and copying what followed. Zero extra memory, zero extra compute, and on copy-heavy work like editing a file, summarising a document, or answering from retrieved context, the acceptance rate is enormous because the answer genuinely is mostly in the input. It's usually a config flag in vLLM or TensorRT-LLM.

If that doesn't fit, the ladder is roughly: prompt lookup, then a same-family small model as the draft (a 1B drafting for a 70B tends to work well, since shared tokenizer and training data lift α), then trained approaches like Medusa or EAGLE that graft extra prediction heads onto the target model and avoid a separate draft model entirely.

Then measure three things on your own traffic: acceptance rate, your typical batch size, and end-to-end latency with and without. The first tells you the ceiling, the second tells you whether the ceiling is available to you, and the third is the only number that settles it.

The technique is unusual in being genuinely free on the axis people worry about most. It does not degrade your model. What it does is spend compute to buy time: an excellent trade when you have compute to spare, and a bad one when you don't.