LLM as a Judge: How to Evaluate LLM Outputs Reliably

LLM as a judge means using one language model to grade the output of another (or of itself), using a written rubric. It is the most common way to evaluate LLM features at scale: people cannot read ten thousand answers after every prompt change, and exact-match checks fail on free text where many wordings are correct.

Think of it as a teaching assistant marking exam papers with the teacher’s marking scheme. It is fast and mostly right, but the teacher still checks a sample, because a marker with a bad scheme, or a habit of being generous, gets every paper wrong in the same way.

When to use a judge

Method	Good for	Weak at
Exact match or code checks	Classification, extraction, numbers, valid JSON	Free text with many correct wordings
People	Final quality calls, building the gold set, edge cases	Volume, speed, cost
LLM as a judge	Large sets of free-text answers: correctness against a source, tone, completeness	Anything it cannot check from the input; subtle domain errors

Use the cheapest method that works: code checks first, a judge for what code cannot check, and people to calibrate the judge and look at the hard cases.

Write the judge prompt

A judge is only as good as its rubric. Give it a fixed scale with a meaning for every score, the material it must check against, and an output format your code can read:

import json

RUBRIC = """Score the ANSWER against the SOURCE on a 1-5 scale:
5 = fully correct and supported by the source
4 = correct, small omission
3 = partly correct or partly unsupported
2 = mostly wrong or unsupported
1 = wrong, or contradicts the source
Return JSON only: {"score": <1-5>, "reason": "<one sentence>"}"""

def judge_prompt(question, source, answer):
    """Build the judge's input: the rubric first, then the item, clearly labelled."""
    return f"{RUBRIC}\n\nQUESTION: {question}\nSOURCE: {source}\nANSWER: {answer}"

def parse_verdict(text):
    """Accept only a valid score; anything else is recorded as a failed judgement, never guessed."""
    try:
        v = json.loads(text.strip().removeprefix("```json").removesuffix("```"))
        score = int(v["score"])
        if 1 <= score <= 5:
            return score, v.get("reason", "")
    except (ValueError, KeyError, TypeError):
        pass
    return None, "unparseable"

print(judge_prompt("How long do refunds take?",
                   "Refunds are paid within 14 days of the return arriving.",
                   "Refunds take up to 7 days.").splitlines()[-1])

raw_outputs = [
    '{"score": 5, "reason": "Matches the 14-day rule."}',
    '```json\n{"score": 2, "reason": "Says 7 days; source says 14."}\n```',
    'The answer looks fine to me!',
    '{"score": 9, "reason": "Great"}',
]
for raw in raw_outputs:
    print(parse_verdict(raw))

ANSWER: Refunds take up to 7 days.
(5, 'Matches the 14-day rule.')
(2, 'Says 7 days; source says 14.')
(None, 'unparseable')
(None, 'unparseable')

Two habits matter here:

• The source is in the prompt. “Is this answer correct?” without the source asks the judge to use its own memory, which brings back the hallucination problem you are trying to measure. See context windows and hallucinations.
• Bad replies are counted, not guessed. A chatty reply or a score of 9 is a failed judgement. Track how often it happens: a rising rate means the judge prompt or model needs attention. Structured output features help (see prompting and structured output).

Asking for the reason before the score, or for a short checklist (“Does the answer state the number of days? Does it match the source?”), usually makes scores more consistent than asking for a bare number.

Check the judge against people

Before trusting a judge, have people score a sample, often 50 to 200 items, and compare. The example uses 10 so the numbers are easy to follow:

# Compare the judge with people on the same 10 answers before trusting it.
human = [5, 2, 4, 1, 3, 5, 2, 4, 5, 1]
judge = [5, 2, 5, 1, 4, 5, 3, 4, 5, 2]

exact = sum(h == j for h, j in zip(human, judge)) / len(human)
within_one = sum(abs(h - j) <= 1 for h, j in zip(human, judge)) / len(human)

def kappa(a, b):
    """Cohen's kappa for pass (score >= 4) vs fail: agreement beyond what chance would give."""
    pa = [x >= 4 for x in a]; pb = [x >= 4 for x in b]
    n = len(a)
    observed = sum(x == y for x, y in zip(pa, pb)) / n
    p_yes = (sum(pa) / n) * (sum(pb) / n)
    p_no = (1 - sum(pa) / n) * (1 - sum(pb) / n)
    expected = p_yes + p_no
    return (observed - expected) / (1 - expected)

print(f"exact agreement:  {exact:.0%}")
print(f"within one point: {within_one:.0%}")
print(f"pass/fail kappa:  {kappa(human, judge):.2f}")
print("judge mean:", sum(judge) / len(judge), " human mean:", sum(human) / len(human))

exact agreement:  60%
within one point: 100%
pass/fail kappa:  0.80
judge mean: 3.6  human mean: 3.2

How to read it:

• Exact agreement of 60% sounds poor, but every score is within one point, and the pass/fail decision matches on 9 of 10 items. For most product decisions, pass/fail is what matters.
• Cohen’s kappa of 0.80 means strong agreement beyond chance. Plain percentage agreement can look high just because most items pass; kappa corrects for that.
• The judge’s average is higher than the people’s. It is slightly generous: leniency bias. Fine for comparing two prompt versions with the same judge, but not for reporting an absolute quality number.

Biases to test for

Bias	What happens	How to reduce it
Position bias	In “which answer is better?”, the judge prefers the first (or second) one shown	Ask twice with the order swapped; count only consistent wins
Length bias	Longer answers score higher even when they add nothing	Say in the rubric that length is not quality; compare at similar lengths
Leniency	Scores drift high, as above	Anchor each score with an example; calibrate against people
Self-preference	A model rates its own outputs higher	Use a different model, or a different model family, as the judge

Position bias is easy to measure. Run each pairwise comparison in both orders and see whether the winner changes:

# Pairwise judging: ask twice with the answers swapped. A fair judge picks the same answer both times.
verdicts = {  # what the judge said for each pair, in both orders ("first" or "second" wins)
    "q1": ("first", "second"),   # A then B -> A wins; B then A -> A wins (consistent)
    "q2": ("first", "first"),    # picks whatever comes first: position bias
    "q3": ("second", "first"),   # consistent: B wins both times
    "q4": ("first", "first"),    # position bias again
}
consistent, biased = [], []
for q, (ab, ba) in verdicts.items():
    winner_ab = "A" if ab == "first" else "B"
    winner_ba = "B" if ba == "first" else "A"
    (consistent if winner_ab == winner_ba else biased).append(q)
print("consistent:", consistent)
print("position-biased:", biased, f"({len(biased) / len(verdicts):.0%})")

consistent: ['q1', 'q3']
position-biased: ['q2', 'q4'] (50%)

Half of these verdicts depended on order, not quality. Treat those pairs as ties, and if the rate stays high, improve the rubric or change the judge model.

Running judges in practice

1. Keep a fixed test set of questions with sources and, ideally, reference answers, including questions the system should refuse. Reuse it on every change.
2. Pin the judge: same model version, same prompt, temperature 0. Changing the judge changes the ruler.
3. Score one thing per prompt (correctness, then tone) instead of one prompt for everything.
4. Recalibrate against a fresh human sample when you change the judge or the product changes a lot.
5. Read the low scores. The reasons point to the real problems: missing context, retrieval misses, wrong format.

For RAG systems, combine a judge for answer faithfulness with retrieval metrics such as recall@k, as in how to evaluate a RAG system. For agents, judge the final result and check the tool calls, as in evaluating and securing agents.

Common mistakes

1. No source in the judge prompt, so the judge grades from memory.
2. Never comparing with people, so nobody knows if the judge is right.
3. Trusting absolute scores from a lenient judge instead of comparing versions with the same judge.
4. One pairwise order only, which hides position bias.
5. Silently dropping unparseable replies, which skews the results towards easy items.

What interviewers look for

When asked “how would you evaluate an LLM feature?” or “what is LLM as a judge?”:

• The layered answer: code checks, an LLM judge with a rubric, and human review to calibrate it.
• How you would validate the judge: a human-labelled sample, agreement and kappa.
• The main biases (position, length, leniency, self-preference) and one fix for each.
• How it fits a release process: a fixed test set run on every prompt or model change.

Practise with the LLM interview questions and the AI engineer interview questions.

Key takeaway

LLM as a judge grades free-text outputs at scale with a written rubric. Put the source in the prompt, demand a strict output format, count failed judgements, and check the judge against human ratings before trusting it. Test for position, length, leniency and self-preference bias, and compare versions with the same pinned judge rather than trusting absolute scores. This completes the LLM Fundamentals course; continue with the RAG course.