Latency Budget
ELI5 - The Vibe Check
A latency budget is deciding upfront how many milliseconds each part of your AI pipeline is allowed to spend before the whole thing feels slow to a user. If your product needs to respond in 2 seconds total, you might allot 200ms to retrieval, 100ms to reranking, and the rest to generation. Blow past your slice and something downstream has to make up the difference, or the user just waits longer.
Real Talk
A latency budget is an allocation of total acceptable response time across the stages of a pipeline, such as retrieval, reranking, prompt construction, and generation, used to set performance targets for each component and identify which stage to optimize when the overall system is too slow. It is especially relevant for RAG and agentic pipelines where multiple sequential LLM and retrieval calls compound into user-facing latency.
When You'll Hear This
"Retrieval is eating half our latency budget, that's where we optimize first." / "We set a 3-second latency budget for the whole RAG pipeline, streaming or not."
Related Terms
Inference
Inference is when the AI actually runs and generates output — as opposed to training, which is when it's learning.
RAG (Retrieval Augmented Generation)
RAG is how you give an AI access to your private documents without retraining it.
Reranking
Reranking is the second opinion after your search engine hands you 50 maybe-relevant results.
Streaming
Streaming is when the AI sends you its response word by word as it generates, instead of making you wait for the whole thing at once.
Time to First Token (TTFT)
Time to First Token is how long a user stares at a blank screen before the AI starts typing back.