[{"data":1,"prerenderedAt":131},["ShallowReactive",2],{"term-e\u002Feval-harness":3,"related-e\u002Feval-harness":114},{"id":4,"title":5,"acronym":6,"body":7,"category":95,"description":96,"difficulty":97,"extension":98,"letter":99,"meta":100,"navigation":101,"path":102,"related":103,"seo":108,"sitemap":109,"stem":112,"subcategory":6,"__hash__":113},"terms\u002Fterms\u002Fe\u002Feval-harness.md","Eval Harness",null,{"type":8,"value":9,"toc":89},"minimark",[10,15,19,23,26,30,78,82,85],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 - The Vibe Check",[16,17,18],"p",{},"An eval harness is the automated testing rig for AI behavior instead of code correctness. You feed it a batch of prompts, capture what the model or agent does with each one, score every response against a rubric, and get a dashboard of pass rates instead of a single green checkmark. It's how you catch a model getting worse before your users do.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"An eval harness is the infrastructure for running evaluations of an LLM or agent at scale: it manages the test dataset, invokes the model or agent under test, scores outputs via exact match, LLM-as-judge, or human review, and aggregates results into pass rates tracked across model versions and prompt changes. It's the closest thing agentic AI development has to a CI test suite.",[11,27,29],{"id":28},"show-me-the-code","Show Me The Code",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","results = []\nfor case in eval_dataset:\n    output = agent.run(case[\"prompt\"])\n    score = scorer(output, case[\"expected\"])\n    results.append({\"id\": case[\"id\"], \"score\": score})\nprint(f\"Pass rate: {sum(r['score'] for r in results) \u002F len(results):.1%}\")\n","python","",[38,39,40,48,54,60,66,72],"code",{"__ignoreMap":36},[41,42,45],"span",{"class":43,"line":44},"line",1,[41,46,47],{},"results = []\n",[41,49,51],{"class":43,"line":50},2,[41,52,53],{},"for case in eval_dataset:\n",[41,55,57],{"class":43,"line":56},3,[41,58,59],{},"    output = agent.run(case[\"prompt\"])\n",[41,61,63],{"class":43,"line":62},4,[41,64,65],{},"    score = scorer(output, case[\"expected\"])\n",[41,67,69],{"class":43,"line":68},5,[41,70,71],{},"    results.append({\"id\": case[\"id\"], \"score\": score})\n",[41,73,75],{"class":43,"line":74},6,[41,76,77],{},"print(f\"Pass rate: {sum(r['score'] for r in results) \u002F len(results):.1%}\")\n",[11,79,81],{"id":80},"when-youll-hear-this","When You'll Hear This",[16,83,84],{},"\"Run it through the eval harness before you touch the system prompt.\" \u002F \"The eval harness caught a 5% regression the manual testing missed.\"",[86,87,88],"style",{},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":36,"searchDepth":50,"depth":50,"links":90},[91,92,93,94],{"id":13,"depth":50,"text":14},{"id":21,"depth":50,"text":22},{"id":28,"depth":50,"text":29},{"id":80,"depth":50,"text":81},"ai","An eval harness is the automated testing rig for AI behavior instead of code correctness.","intermediate","md","e",{},true,"\u002Fterms\u002Fe\u002Feval-harness",[104,105,106,107],"LLM-as-Judge","Evaluation","CI","Regression Test",{"title":5,"description":96},{"changefreq":110,"priority":111},"weekly",0.7,"terms\u002Fe\u002Feval-harness","9j1vDXH6utTic4eoneHoUcCACCVyQQcJG9xdqqB059Y",[115,121,124,127],{"title":106,"path":116,"acronym":117,"category":118,"difficulty":119,"description":120},"\u002Fterms\u002Fc\u002Fci","Continuous Integration","cicd","beginner","CI is like a robot assistant that instantly checks your homework every time you hand it in.",{"title":105,"path":122,"acronym":6,"category":95,"difficulty":97,"description":123},"\u002Fterms\u002Fe\u002Fevaluation","Evaluation in AI is figuring out if your model actually works — not just on test data, but in the real world.",{"title":104,"path":125,"acronym":6,"category":95,"difficulty":97,"description":126},"\u002Fterms\u002Fl\u002Fllm-as-judge","Instead of paying a human to grade a thousand AI-generated essays, you hire a second AI as the strict teacher who reads the first AI's homework and hands b...",{"title":107,"path":128,"acronym":6,"category":129,"difficulty":119,"description":130},"\u002Fterms\u002Fr\u002Fregression-test","testing","A regression test is a test you write AFTER fixing a bug, to make sure that bug never comes back.",1792528082821]