Continuous Batching

ELI5 - The Vibe Check

Old-school batching made every request in a group wait for the slowest one to finish, like a bus that won't leave until every passenger's errand is done. Continuous batching lets new requests hop on and finished ones hop off at every single step, so the GPU is never sitting there waiting on one straggler. It is the trick that let serving engines like vLLM squeeze way more requests out of the same hardware.

Real Talk

Continuous batching (also called in-flight or iteration-level batching) dynamically adds and removes sequences from a GPU batch at every decoding step rather than waiting for a fixed batch to fully complete before starting the next one. This keeps GPU utilization high under variable-length generations and is a core technique in serving engines such as vLLM and TensorRT-LLM.

Show Me The Code

# Static batching: batch waits for the longest sequence to finish
# Continuous batching: new requests join mid-generation
engine = LLMEngine(
    model="meta-llama/Llama-3-8B",
    max_num_seqs=256,       # requests that can be in-flight together
    enable_chunked_prefill=True
)

When You'll Hear This

"Switching to continuous batching tripled our throughput without new GPUs." / "vLLM's continuous batching is why it beats the naive HuggingFace serving loop."