Prefill and Decode

ELI5 - The Vibe Check

LLM inference actually happens in two very different phases. Prefill is the model reading your entire prompt at once, which is heavy on raw compute but happens in parallel, so it's fast per token. Decode is generating the response one token at a time, where each new word depends on the last, so it's slow per token but light on compute per step. It's the difference between speed-reading a whole page versus writing your reply one word at a time.

Real Talk

Prefill is the inference phase where the entire input prompt is processed in a single parallelized forward pass to populate the KV cache, making it compute-bound and highly parallel. Decode is the subsequent phase where output tokens are generated one at a time, each requiring a full forward pass that depends on the previous token, making it memory-bandwidth bound rather than compute bound. Serving engines optimize the two phases separately, and techniques like chunked prefill and disaggregated prefill/decode serving exist specifically to keep GPUs efficiently utilized across both.

When You'll Hear This

"Long prompts blow up prefill time even before the model writes a single output token." / "We disaggregated prefill and decode onto separate GPU pools to stop them competing for compute."