Section 5
Golden set
Every retrieval number on this site is computed against this query set. Publishing it is the only way a reader can judge whether those numbers mean anything.
A golden set is a list of questions, the passages that actually answer them, and a relevance grade for every candidate a retriever might return. Recall, NDCG and MRR are all defined against those grades — a flawed golden set does not produce a slightly wrong score, it produces a confidently wrong one.
5.1 State of the set
- Queries
- 12
- Relevance judgments
- 256
- Human-verified
- 32
11 answerable · 1 deliberately not
one per candidate chunk considered
13% of judgments, across 4 of 12 queries
The set was drafted by gemini-3.6-flash and graded by gemini-3.1-flash-lite, both pinned to dated snapshots. Nothing here is hand-written by the author except the verification grades, which is precisely the part that matters.
5.2 Where the human and the model disagree
Verification presents a stratified 25% sample by category, showing the eight highest-graded candidates for each sampled query, and the author re-grades them against the same rubric the model was given. Of 32 judgments checked so far, the human and the model agreed on 1 — 3%.
| human − model | judgments | meaning |
|---|---|---|
| -1 | 3 | human graded 1 step lower |
| 0 | 1 | agreed |
| +1 | 6 | human graded 1 step higher — the model called it less relevant than it was |
| +2 | 6 | human graded 2 steps higher — the model called it less relevant than it was |
| +3 | 16 | human graded 3 steps higher — the model called it less relevant than it was |
Every disagreement runs one way. In 28 of 32 checked judgments the human graded the chunk higher than the model did, and in 3 lower. The pattern is strongest on chunks the model had scored as irrelevant.
What this does not establish
- Which grade is correct. One annotator disagreeing with one model is a disagreement, not a correction. Neither side is ground truth here.
- That the pattern holds. 32 judgments across 4 of 12 queries is a small sample, and verification is still in progress.
- An independent second opinion. Verification is not blind — the verification tool prints the model's grade and its stated reason before asking for the human's. Anchoring is possible in either direction, and a blind pass would be a stronger instrument.
- A published error rate. No retrieval score has been computed from this set, so nothing on this site is currently affected by it. That is the point of checking first.
What it does establish is that this set is not yet fit to compute a published metric from, and that a run shipped today would have inherited the disagreement silently.
5.3 Contradictions inside the set
The disagreement above counts how often the human and the model differ. It does not say which is right. Checking the set against itself does say something: these are statements the golden set makes that cannot all be true at once, whoever made them.
The clearest is the unanswerable query. It is marked as having no answer in the corpus, and eight of its chunks carry a human grade of 3 — fully answers the question on its own. If a chunk fully answers it, the query is answerable. One of those two statements is wrong, and the model's stated reasoning on those same chunks (“lacks a specific timeframe”) reads as the correct one.
That matters for how the disagreement figure above should be read. It was tempting to conclude the judge under-grades. On at least this query the judge looks right and the verification looks wrong, so the 28 disagreements are not all the same kind of thing.
- unanswerable query has relevant chunks
- unanswerable-0008 chunk(s) graded >= 2 (['HUMAN']). A chunk that answers the question contradicts answerable=false — one of the two is wrong.
- declared source graded irrelevant
- cross-document-0001 of 3 source chunk(s) the query itself names as its answer are graded below 2.
- declared source graded irrelevant
- multi-chunk-0003 of 4 source chunk(s) the query itself names as its answer are graded below 2.
- declared source graded irrelevant
- tenant-scoped-0002 of 3 source chunk(s) the query itself names as its answer are graded below 2.
- queries with no judgments
- single-chunk-001, single-chunk-002, single-chunk-003, single-chunk-004, single-chunk-005, single-chunk-006, single-chunk-0077 of 12 queries have no relevance judgments and cannot contribute to any metric.
Produced by consistency.py · commit 74cbff4 · 2026-08-26
No retrieval metric is published anywhere on this site, and this is the second reason. The first is the sample size; this is the set disagreeing with itself. Both are fixable, and neither is fixed by publishing a number and adding a footnote.
5.4 The rubric
The same four grades are given to the model and to the human verifier.
- 3
- fully answers the question on its own4 of 256 judgments
- 2
- contains a substantial part of the answer3 of 256 judgments
- 1
- related context, but does not contain the answer15 of 256 judgments
- 0
- not relevant234 of 256 judgments
234 of 256 candidates were graded not relevant. A retrieval candidate pool is mostly misses by construction; that is what makes the few positives worth checking.
5.5 Query categories
Categories are not decoration — the set is sampled and the agreement rate reported per category, so a category that is thin is visible rather than averaged away.
- 8
- single-chunk
- 1
- cross-document
- 1
- multi-chunk
- 1
- tenant-scoped
- 1
- unanswerable
5.6 Every query
The full set, with each query's candidate grades and the verification result where one exists.