Building and Calibrating an LLM-as-Judge

A Judge Is a Second Model Call, Not a Magic Score

LLM-as-judge is one of the three scoring methods available to an eval (alongside exact match and a code-graded check), reserved for open-ended quality -- faithfulness, instruction-following, tone -- that no code rule can express. Concretely, it's a second model call: you hand a grading model the task and the solution, guided by a rubric, and it returns a score. Building one well means being deliberate about exactly what you ask it to return and in what order, then earning trust in its output before relying on it in production.

What the Judge Prompt Should Return

Ask for structured JSON with four fields, not a bare number:

eval_prompt = f"""
Act as a careful reviewer grading the solution below against the task it was meant to solve.
Task: {{task}}
Solution: {{solution}}
Respond with JSON containing:
  "strengths":  1 to 3 specific things the solution does well
  "weaknesses": 1 to 3 specific things it gets wrong or misses
  "reasoning":  a short justification, no more than ~50 words
  "score":      an integer from 1 to 10
"""

• 1-3 strengths -- specific things the solution got right
• 1-3 weaknesses -- specific things it got wrong or missed
• reasoning -- capped at roughly 50 words, forcing the judge to be concrete rather than vague
• score -- the final numeric verdict

Ordering Is Not Cosmetic: Reasoning Before Score

The order these fields are requested in is itself a deliberate design choice, not incidental formatting. Asking the judge for strengths, weaknesses, and reasoning before score anchors the numeric score to the specific reasoning the judge just committed to in writing. Without that ordering, judge models measurably drift toward a *safe middle number* -- a score around 6 out of 10 -- almost regardless of the output's actual quality, because there's nothing forcing the score to be justified by anything concrete. Reasoning-first construction is what breaks that drift: once the judge has already written down two specific weaknesses, giving the same output an 9/10 would visibly contradict its own stated reasoning, so the score tends to actually track the substance.

Calibration: Earning the Right to Trust the Judge

A judge prompt that returns clean JSON is not the same thing as a judge whose scores mean anything. Before trusting a judge's scores in production, measure its agreement rate against a set of human-labeled examples: take cases a human has already scored, run the judge on the same cases, and check how often the judge's verdict matches the human's. A judge whose calls only line up with the human's about half the time is producing a number that looks authoritative but tells you nothing real.

If agreement comes back low, the fix is not to discard the judge -- it's to iterate on the rubric: tighten what each score band actually means, add explicit examples of a good answer and a bad answer directly in the judge prompt, and re-measure agreement. Repeat until agreement is high enough to trust. Skipping this step and deploying an uncalibrated judge on the assumption that "it's an LLM, it'll figure out quality" is the single most common way an eval program ends up defending a number nobody actually believes.

Common exam traps

• Building a judge prompt that returns only a bare numeric score. Without strengths/weaknesses/reasoning fields, the judge has nothing concrete anchoring its number, and scores drift toward an uninformative safe middle value.
• Requesting the score before the reasoning. The order matters -- reasoning-first is what anchors the score; score-first re-introduces the same drift-to-the-middle problem the structured fields were meant to solve.
• Deploying a judge straight from its prompt design without calibrating against human-labeled examples first. An uncalibrated judge can look rigorous (clean JSON, confident numbers) while actually tracking nothing -- calibration against human agreement is what turns it from a guess into evidence.