What does “LLM-as-Judge” mean?
Using a language model to score or compare outputs from other models during evaluation.
How LLM-as-Judge Works
Two or more answers to grade
Judge model gets a rubric
Scores or picks the better one
Aggregate across many cases.
Practical Example
To compare two summaries, a judge model is given the source text and a rubric on accuracy and clarity, then picks the better summary. A sample of its verdicts is checked by people.
Why It Matters
It scales evaluation far beyond what people can review. Judges have biases, such as preferring longer answers, so they need calibration against human ratings.
Common Questions About LLM-as-Judge
Q: Can I trust it?
Reasonably for many tasks when checked against human judgments, but not blindly.
Q: Known biases?
Preferences for longer answers, for the first option shown, and for its own style.
Q: How to reduce bias?
Swap answer order, use clear rubrics, and compare against a human-labeled sample.