[{"data":1,"prerenderedAt":78},["ShallowReactive",2],{"term-p\u002Fprefill-and-decode":3,"related-p\u002Fprefill-and-decode":59},{"id":4,"title":5,"acronym":6,"body":7,"category":40,"description":41,"difficulty":42,"extension":43,"letter":16,"meta":44,"navigation":45,"path":46,"related":47,"seo":53,"sitemap":54,"stem":57,"subcategory":6,"__hash__":58},"terms\u002Fterms\u002Fp\u002Fprefill-and-decode.md","Prefill and Decode",null,{"type":8,"value":9,"toc":33},"minimark",[10,15,19,23,26,30],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"LLM inference actually happens in two very different phases. Prefill is the model reading your entire prompt at once, which is heavy on raw compute but happens in parallel, so it's fast per token. Decode is generating the response one token at a time, where each new word depends on the last, so it's slow per token but light on compute per step. It's the difference between speed-reading a whole page versus writing your reply one word at a time.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"Prefill is the inference phase where the entire input prompt is processed in a single parallelized forward pass to populate the KV cache, making it compute-bound and highly parallel. Decode is the subsequent phase where output tokens are generated one at a time, each requiring a full forward pass that depends on the previous token, making it memory-bandwidth bound rather than compute bound. Serving engines optimize the two phases separately, and techniques like chunked prefill and disaggregated prefill\u002Fdecode serving exist specifically to keep GPUs efficiently utilized across both.",[11,27,29],{"id":28},"when-youll-hear-this","When You'll Hear This",[16,31,32],{},"\"Long prompts blow up prefill time even before the model writes a single output token.\" \u002F \"We disaggregated prefill and decode onto separate GPU pools to stop them competing for compute.\"",{"title":34,"searchDepth":35,"depth":35,"links":36},"",2,[37,38,39],{"id":13,"depth":35,"text":14},{"id":21,"depth":35,"text":22},{"id":28,"depth":35,"text":29},"ai","LLM inference actually happens in two very different phases.","advanced","md",{},true,"\u002Fterms\u002Fp\u002Fprefill-and-decode",[48,49,50,51,52],"KV Cache","Inference","Time to First Token","Continuous Batching","Context Window",{"title":5,"description":41},{"changefreq":55,"priority":56},"weekly",0.7,"terms\u002Fp\u002Fprefill-and-decode","8kEq_1HNO1x6k0gOgFQu58reMbgjuMvQuz0QcV7Ll50",[60,65,68,71,74],{"title":52,"path":61,"acronym":6,"category":62,"difficulty":63,"description":64},"\u002Fterms\u002Fc\u002Fcontext-window","vibecoding","intermediate","A context window is how much text an AI can 'see' at once — its working memory.",{"title":51,"path":66,"acronym":6,"category":40,"difficulty":42,"description":67},"\u002Fterms\u002Fc\u002Fcontinuous-batching","Old-school batching made every request in a group wait for the slowest one to finish, like a bus that won't leave until every passenger's errand is done.",{"title":49,"path":69,"acronym":6,"category":40,"difficulty":63,"description":70},"\u002Fterms\u002Fi\u002Finference","Inference is when the AI actually runs and generates output — as opposed to training, which is when it's learning.",{"title":48,"path":72,"acronym":6,"category":40,"difficulty":42,"description":73},"\u002Fterms\u002Fk\u002Fkv-cache","KV cache is how LLMs remember previous tokens without recomputing them.",{"title":50,"path":75,"acronym":76,"category":40,"difficulty":63,"description":77},"\u002Fterms\u002Ft\u002Ftime-to-first-token","TTFT","Time to First Token is how long a user stares at a blank screen before the AI starts typing back.",1792528109753]