Section 7 · Learn · Evaluation
LLM-as-judge and blind judging
Using a language model to grade relevance or faithfulness against a published rubric — and keeping it blind to the answers you want.
Using a language model to grade relevance or faithfulness against a published rubric — and keeping it blind to the answers you want.
What it is
A model grades each query-passage pair against a written rubric, standing in for a human annotator at a fraction of the cost.
Why it is needed
Grading 10,000 candidates by hand is not feasible. A model does it in hours for a couple of dollars. The question is whether the grades mean anything.
How it works
The rubric is published. The judge sees only the question and the passage text. A stratified sample is re-graded by a human and the agreement rate published beside the results.
In this project
Drafting and judging use different models, and the judge is blind by construction: the judging prompt interpolates only the relevance statement, the question and the passages. Whether a passage was one the query was drafted from is recorded in the output but never sent to the judge. Model references are pinned to dated snapshots, and every record carries name@snapshot rather than a bare name.
Why blindness is the whole design
If the judge can see which passages a query was written from, it agrees with them, and the agreement rate then measures conformity rather than correctness. The number goes up, which is exactly what makes it dangerous — a high agreement rate looks like a well-constructed set.
What happened here
The human verification sample is currently unusable and the site says so rather than publishing it. Of 32 verified judgments, 30 were graded 3 and 2 were graded 2 — 94% a single value — while the model graded 91% of the same pool as 0. A sample that does not discriminate cannot measure a judge in either direction, so the agreement rate is withheld with the reason stated, pending re-grading.
Trade-offs
Advantages
- Scales to thousands of judgments
- Consistent application of a written rubric
- Cheap enough to re-run
Limitations
- Inherits the model's biases
- Non-deterministic unless temperature is pinned
- Silently changes if the model reference floats
Common mistakes
- Letting the judge see the drafter's labels
- Using a -latest alias, so re-running months later silently changes the judge
- Publishing an agreement rate from a degenerate sample
When to use it
Reach for it when
- Grading at a scale humans cannot reach
Avoid it when
- As the sole authority with no human verification at all
How it interacts with the rest of the system
Golden sets and graded relevanceModel pinning and reproducibility