Model Router
ELI5 - The Vibe Check
Not every question needs your smartest, most expensive model. A model router looks at each incoming request and decides which model should actually handle it: a cheap fast model for 'what's the weather,' a big expensive one for 'debug this race condition.' It's a dispatcher for AI models instead of restaurant orders, sending each ticket to the cook who can actually make it well without wasting a specialist on toast.
Real Talk
A model router is a component that classifies incoming requests and dynamically dispatches each one to the most appropriate model from a pool, balancing cost, latency, and capability. Routing logic can be rule-based (task type, prompt length), a lightweight trained classifier, or LLM-based triage, and it typically pairs with fallback logic so a failed or overloaded model doesn't take the whole system down.
When You'll Hear This
"The router sends simple classification tasks to Haiku and reasoning-heavy ones to Opus." / "Our model router cut average cost per request by two-thirds."
Related Terms
Cost Per Token
Cost per token is how much each token (input or output) costs with a given AI provider. Flagship models cost more per token than cheap ones.
Inference
Inference is when the AI actually runs and generates output — as opposed to training, which is when it's learning.
Latency Budget
A latency budget is deciding upfront how many milliseconds each part of your AI pipeline is allowed to spend before the whole thing feels slow to a user.
LLM (Large Language Model)
An LLM is a humongous AI that read basically the entire internet and learned to predict what words come next, really really well.
Model Fallback
Model fallback is your AI app's plan B, C, and D for when the primary model has a bad day: it's rate-limited, it's down, or it just times out.