LLM-as-judge: using one model to grade another model
LLM-as-judge means using a language model to evaluate another model's answers. Learn when it helps, which biases it has and how to validate it against people.
What is a model judge?
An eval needs a grader. Anthropic distinguishes three types: code, people and models. Code is fast and reliable for objective things, such as whether a JSON is valid or whether a date appears. People are flexible but slow and expensive. For nuanced criteria, such as “the explanation is clear to someone who does not know the product”, a model can do the work at scale.
The judge’s prompt describes the criterion, includes examples and asks for a verdict with a critique. What matters is not which model you pick, but how closely its verdicts match those of an expert. Anthropic puts it this way: judges should be closely calibrated with human experts, and people are still useful for occasional review.
The approach became popular with the paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023). The authors found that strong judges such as GPT-4 agree with human preferences in over 80% of cases, the same level of agreement that exists between people (on MT-Bench, excluding ties, 85% for GPT-4 with people versus 81% between people). They also documented its limits, covered below.
Why binary and not a scale?
This is the most counterintuitive recommendation. Hamel Husain warns against metric dashboards with 1-to-5 scores: the difference between adjacent points is subjective, “people don’t know what to do with a 3 or 4”, and those numbers rarely correlate with what the domain expert cares about.
A pass-or-fail verdict, with a written critique explaining why, is more useful for deciding. Husain suggests starting with binary labels to understand what “bad” looks like. Anthropic’s documentation also accepts 1-to-5 scales, but asks for detailed rubrics and for the judge to give a concrete output (for example, only “correct” or “incorrect”), because purely qualitative evaluations are hard to review at scale.
What biases does a model judge have?
Zheng et al. describe three:
| Bias | What it means | How to mitigate it |
|---|---|---|
| Position | When comparing two answers, it favors one position; in the paper, most judges preferred the first | Run the comparison twice with the order reversed and declare a win only if the same answer wins both times |
| Verbosity | Tends to prefer longer answers even when others are clearer or more accurate | State in the criterion that length does not count and check some cases by hand |
| Self-preference | May favor answers it generated itself; the authors could not confirm it with their data | Use a different model as judge or validate against people |
The paper also notes that judges can grade math and reasoning questions wrongly, even basic ones, because the answers they see mislead them. For those cases it proposes giving the judge a reference answer, which cut the failure rate from 70% to 15% in their experiment. In every case, measure the judge against human labels in your own case.
How do you validate the judge?
Treat the judge as a classifier. The procedure Husain describes:
- An expert labels a set of answers as pass or fail.
- Split those examples into training, development and test sets. Set the test set aside until the end.
- Tune the prompt with the development examples and freeze it before the final test run.
- Measure the true positive rate (real failures the judge catches) and the true negative rate (passes it correctly approves).
Do not stop at the agreement percentage: if only 5% of answers fail, a judge that always passes has 95% agreement and catches none of the failures. Husain also warns that after many rounds of tuning, the judge can do well on the development set and poorly on new examples, even if you never copied those examples into the prompt.
How do you write a good judge prompt?
- Explicit criterion: what pass means, with one example on each side.
- Detailed critiques: Husain asks that the critique for each example be detailed enough to use as an example in the prompt; being too terse is a common mistake.
- One judge per dimension: Anthropic recommends grading each dimension (tone, accuracy, clarity) with an isolated judge instead of a single one for everything.
- An escape hatch: Anthropic suggests allowing an “Unknown” answer so the judge does not invent a verdict without data.
- Reason before grading: Anthropic’s documentation says that having the judge reason first improves evaluation on tasks requiring complex judgment.
- Same context as the expert: if the person looked at the user’s history, the judge must see it too.
When should you NOT use one?
- When the criterion can be checked with code. A model judge adds cost and nondeterminism where they were not needed.
- When you have no human labels to validate it. An unvalidated judge is an opinion dressed up as a metric.
- When the judge is the same model that generates, without having measured whether it favors itself.
Example
Illustrative example: a bank’s assistant explains fees. An expert reviews 80 answers and marks which ones pass: 10 fail and 70 pass. The judge receives the criterion “explains the fee with its amount and condition, without using undefined financial jargon”, two passing examples and two failing ones with written critiques. In testing, the judge catches 9 of the 10 bad answers (a true positive rate of 90%) but wrongly fails 4 of the 70 good ones (a true negative rate of about 94%). The team adjusts the prompt with those 4 cases, freezes the version and uses it for the weekly eval.
For the framework of what evals are, read what are evals. To discover the criteria through error analysis, evals as UX research. And for the case set that feeds the judge, golden datasets.
Common mistakes
- Using 1-to-5 scales without defining what each number means.
- Reporting only overall agreement, without true positive and true negative rates.
- Letting the judge grade without written criteria or examples.
- Ignoring position bias: always comparing A against B in the same order.
- Not freezing the judge version. If you change the judge’s prompt, historical results stop being comparable.