Speculative Decoding

ELI5 - The Vibe Check

Speculative decoding is like having a fast, slightly reckless intern draft several sentences at once while the careful senior engineer just checks and approves them instead of writing every word from scratch. A small, cheap model guesses the next few tokens, and the big model verifies them all in a single pass. When the guesses are good, you get the big model's quality at a fraction of its usual speed.

Real Talk

Speculative decoding accelerates autoregressive LLM inference by using a small draft model to propose multiple future tokens, which the larger target model then verifies in parallel in one forward pass instead of generating one token at a time. Accepted tokens are kept and rejected ones are regenerated by the target model, preserving the target model's exact output distribution while cutting wall-clock latency, often by two to three times.

When You'll Hear This

"We turned on speculative decoding and cut p50 latency almost in half." / "The draft model's acceptance rate dropped on code, so the speedup shrank."