Section 7 · Learn · Evaluation
Golden sets and graded relevance
The labelled query set a retrieval system is scored against, and the grades that define what counts as a correct result.
The labelled query set a retrieval system is scored against, and the grades that define what counts as a correct result.
What it is
A golden set pairs queries with judgments: for each query, which passages are relevant and how relevant, usually on a 0-3 scale.
Why it is needed
Every retrieval metric is a function of these grades. If the grades are wrong, the metrics are wrong in a way no amount of engineering downstream can detect.
How it works
Queries are drafted across categories, candidates are pooled with a lexical retriever, and each candidate is graded. A stratified sample is then re-graded by a human and the agreement rate published.
At a glance
| Grade | Meaning |
|---|---|
| 3 | fully answers the question on its own |
| 2 | contains a substantial part of the answer |
| 1 | related context, does not contain the answer |
| 0 | not relevant |
In this project
Targets 200 queries across five categories: single-chunk factual, multi-chunk synthesis, cross-document, tenant-scoped, and genuinely unanswerable. Candidates are pooled 50 per query by BM25, with the passages a query was drafted from always included so a known positive is always in the pool. The set is model-drafted and only partially human-verified, and is described that way rather than as hand-labelled.
Trade-offs
Advantages
- Makes retrieval quality measurable at all
- Graded relevance supports NDCG, not just hit-or-miss
Limitations
- Expensive to build
- Grades encode one person's judgement
- Pooling can miss relevant passages no retriever surfaced
Common mistakes
- Describing a model-drafted set as hand-labelled
- Publishing an agreement rate from a sample that does not discriminate
When to use it
Reach for it when
- Any retrieval system you intend to make claims about
How it interacts with the rest of the system
LLM-as-judge and blind judgingRecall@k and retrieval metricsNDCG and rank-aware scoringAbstention and knowing when not to answer