AI ART Discover · Learn · Create
L AI Glossary Term • Term 116 of 224
Plain English Definition

What does “LLM-as-Judge” mean?

Core Definition

Using a language model to score or compare outputs from other models during evaluation.

Operational Mechanism

How LLM-as-Judge Works

1

Two or more answers to grade

2

Judge model gets a rubric

3

Scores or picks the better one

4

Aggregate across many cases.

Real-World Scenario

Practical Example

To compare two summaries, a judge model is given the source text and a rubric on accuracy and clarity, then picks the better summary. A sample of its verdicts is checked by people.

Core Significance

Why It Matters

It scales evaluation far beyond what people can review. Judges have biases, such as preferring longer answers, so they need calibration against human ratings.

Expert Q&A

Common Questions About LLM-as-Judge

Q: Can I trust it?

Reasonably for many tasks when checked against human judgments, but not blindly.

Q: Known biases?

Preferences for longer answers, for the first option shown, and for its own style.

Q: How to reduce bias?

Swap answer order, use clear rubrics, and compare against a human-labeled sample.