LLM-as-Judge

ELI5 - The Vibe Check

Instead of paying a human to grade a thousand AI-generated essays, you hire a second AI as the strict teacher who reads the first AI's homework and hands back a score. It's fast, it's cheap, and it's shockingly consistent when the rubric is good and shockingly gameable when it isn't.

Real Talk

LLM-as-judge is an evaluation technique where a language model, often a stronger model or one prompted with a strict rubric, scores or compares the outputs of another model, replacing or supplementing human raters at scale. It's widely used for evals, preference data collection, and automated code review, though it inherits its own systematic biases, such as favoring longer or more confidently worded answers, which need to be calibrated against human judgment.

Show Me The Code

judge_prompt = f"""
Rate this response from 1 to 5 on accuracy and helpfulness.
Question: {question}
Response: {response}
Return only JSON: {{"score": <int>, "reason": "<string>"}}
"""
verdict = judge_model.generate(judge_prompt)

When You'll Hear This

"We use LLM-as-judge to grade the PR summaries before they ever reach a human." / "The judge model kept favoring the longer answer, so we had to fix its rubric."