LLM judge
An LLM judge is a language model that grades another model's output against a set of criteria.
What it means
An LLM judge is a language model used as a grader. Benchmark teams give it a task, the answer under test, and a rubric, then ask it to decide whether each criterion is met. Judges are cheaper and faster than human graders, which is why many recent deep research benchmarks use them. They can also score hundreds of tasks without a panel of experts reading every report.
The judge is a source of error. It can favor answers in its own style, miss a wrong figure, or mark a claim as met when the source does not say it. Good benchmark runs name the judge, keep it the same across tools, and report how it was checked. When you compare two scores, confirm that both used the same judge, because a change of judge can move a ranking even when the reports stay the same.
Continue this reading path
Cite this entry
https://bestdeepresearchai.com/glossary/llm-judge/