LLM Evals

ELI5 - The Vibe Check

LLM evals are the test suite for a model or a prompt, except instead of asserting that 2 plus 2 equals 4, you're often asking 'is this answer actually good?' which is a much fuzzier question. Evals can be simple exact-match checks, or they can be another LLM acting as a judge, or a human rating outputs. Ship a prompt change without evals and you're just vibing in production.

Real Talk

LLM evals are structured test suites that measure a model's or prompt's output quality against a dataset of inputs with defined success criteria, ranging from exact-match and rubric-based scoring to LLM-as-judge and human evaluation. Eval frameworks like promptfoo, Braintrust, and OpenAI Evals let teams catch regressions when swapping models, changing prompts, or updating RAG pipelines before those changes reach production.

Show Me The Code

# promptfoo-style eval config
tests:
  - vars: { question: "What is our refund window?" }
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "answer is polite and on-topic"

When You'll Hear This

"We don't ship a prompt change without running it against the eval suite first." / "The new model regressed on our evals for multi-step math."