[{"data":1,"prerenderedAt":141},["ShallowReactive",2],{"term-c\u002Fcontinuous-batching":3,"related-c\u002Fcontinuous-batching":121},{"id":4,"title":5,"acronym":6,"body":7,"category":101,"description":102,"difficulty":103,"extension":104,"letter":105,"meta":106,"navigation":107,"path":108,"related":109,"seo":115,"sitemap":116,"stem":119,"subcategory":6,"__hash__":120},"terms\u002Fterms\u002Fc\u002Fcontinuous-batching.md","Continuous Batching",null,{"type":8,"value":9,"toc":95},"minimark",[10,15,19,23,26,30,84,88,91],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"Old-school batching made every request in a group wait for the slowest one to finish, like a bus that won't leave until every passenger's errand is done. Continuous batching lets new requests hop on and finished ones hop off at every single step, so the GPU is never sitting there waiting on one straggler. It is the trick that let serving engines like vLLM squeeze way more requests out of the same hardware.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"Continuous batching (also called in-flight or iteration-level batching) dynamically adds and removes sequences from a GPU batch at every decoding step rather than waiting for a fixed batch to fully complete before starting the next one. This keeps GPU utilization high under variable-length generations and is a core technique in serving engines such as vLLM and TensorRT-LLM.",[11,27,29],{"id":28},"show-me-the-code","Show Me The Code",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Static batching: batch waits for the longest sequence to finish\n# Continuous batching: new requests join mid-generation\nengine = LLMEngine(\n    model=\"meta-llama\u002FLlama-3-8B\",\n    max_num_seqs=256,       # requests that can be in-flight together\n    enable_chunked_prefill=True\n)\n","python","",[38,39,40,48,54,60,66,72,78],"code",{"__ignoreMap":36},[41,42,45],"span",{"class":43,"line":44},"line",1,[41,46,47],{},"# Static batching: batch waits for the longest sequence to finish\n",[41,49,51],{"class":43,"line":50},2,[41,52,53],{},"# Continuous batching: new requests join mid-generation\n",[41,55,57],{"class":43,"line":56},3,[41,58,59],{},"engine = LLMEngine(\n",[41,61,63],{"class":43,"line":62},4,[41,64,65],{},"    model=\"meta-llama\u002FLlama-3-8B\",\n",[41,67,69],{"class":43,"line":68},5,[41,70,71],{},"    max_num_seqs=256,       # requests that can be in-flight together\n",[41,73,75],{"class":43,"line":74},6,[41,76,77],{},"    enable_chunked_prefill=True\n",[41,79,81],{"class":43,"line":80},7,[41,82,83],{},")\n",[11,85,87],{"id":86},"when-youll-hear-this","When You'll Hear This",[16,89,90],{},"\"Switching to continuous batching tripled our throughput without new GPUs.\" \u002F \"vLLM's continuous batching is why it beats the naive HuggingFace serving loop.\"",[92,93,94],"style",{},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":36,"searchDepth":50,"depth":50,"links":96},[97,98,99,100],{"id":13,"depth":50,"text":14},{"id":21,"depth":50,"text":22},{"id":28,"depth":50,"text":29},{"id":86,"depth":50,"text":87},"ai","Old-school batching made every request in a group wait for the slowest one to finish, like a bus that won't leave until every passenger's errand is done.","advanced","md","c",{},true,"\u002Fterms\u002Fc\u002Fcontinuous-batching",[110,111,112,113,114],"Model Serving","Inference","KV Cache","Throughput","GPU",{"title":5,"description":102},{"changefreq":117,"priority":118},"weekly",0.7,"terms\u002Fc\u002Fcontinuous-batching","ka4oQr-iQWSjMmcUXQnO_g-KscywnhqDjZFkgn_k9vg",[122,127,131,134,137],{"title":114,"path":123,"acronym":124,"category":101,"difficulty":125,"description":126},"\u002Fterms\u002Fg\u002Fgpu","Graphics Processing Unit","beginner","A GPU was originally built for rendering graphics in games, but turns out it's also perfect for AI.",{"title":111,"path":128,"acronym":6,"category":101,"difficulty":129,"description":130},"\u002Fterms\u002Fi\u002Finference","intermediate","Inference is when the AI actually runs and generates output — as opposed to training, which is when it's learning.",{"title":112,"path":132,"acronym":6,"category":101,"difficulty":103,"description":133},"\u002Fterms\u002Fk\u002Fkv-cache","KV cache is how LLMs remember previous tokens without recomputing them.",{"title":110,"path":135,"acronym":6,"category":101,"difficulty":129,"description":136},"\u002Fterms\u002Fm\u002Fmodel-serving","Model serving is the infrastructure that takes a trained AI model and makes it available as a fast, reliable API.",{"title":113,"path":138,"acronym":6,"category":139,"difficulty":129,"description":140},"\u002Fterms\u002Ft\u002Fthroughput","cloud","Throughput is how much actual work gets done per second — how many requests your server handles, how much data it actually transfers.",1792528074539]