Eval Harness

ELI5 - The Vibe Check

An eval harness is the automated testing rig for AI behavior instead of code correctness. You feed it a batch of prompts, capture what the model or agent does with each one, score every response against a rubric, and get a dashboard of pass rates instead of a single green checkmark. It's how you catch a model getting worse before your users do.

Real Talk

An eval harness is the infrastructure for running evaluations of an LLM or agent at scale: it manages the test dataset, invokes the model or agent under test, scores outputs via exact match, LLM-as-judge, or human review, and aggregates results into pass rates tracked across model versions and prompt changes. It's the closest thing agentic AI development has to a CI test suite.

Show Me The Code

results = []
for case in eval_dataset:
    output = agent.run(case["prompt"])
    score = scorer(output, case["expected"])
    results.append({"id": case["id"], "score": score})
print(f"Pass rate: {sum(r['score'] for r in results) / len(results):.1%}")

When You'll Hear This

"Run it through the eval harness before you touch the system prompt." / "The eval harness caught a 5% regression the manual testing missed."