Section 7 · Learn · Evaluation
Recall@k and retrieval metrics
Whether the passages needed to answer a query made it into the top k results.
Whether the passages needed to answer a query made it into the top k results.
What it is
Recall@k is the proportion of queries for which at least one sufficiently relevant passage appears in the top k.
Why it is needed
It is the ceiling on everything downstream. A passage the retriever never returns cannot be reranked, cited or quoted.
How it works
For each query, look at the top k, ask whether any of them is graded 2 or higher, and average that across the query set.
Formula
Recall@k = |{ q in Q : max grade(c) >= 2 for c in top_k(q) }| / |Q|
MRR@k = (1/|Q|) * sum over q of 1 / rank of first relevant
Context precision = |{ c in top_k : grade(c) >= 2 }| / k- Q
- the answerable subset of the query set
- grade(c)
- graded relevance 0-3 from the golden set
- >= 2
- the relevance boundary: contains a substantial part of the answer
In this project
Measured at k=5 and k=10. Reported with a bootstrapped 95% confidence interval, and always labelled with the corpus it was measured on, because retrieval metrics are properties of a corpus-and-query-set pair rather than of an architecture.
Trade-offs
Advantages
- Directly measures the thing that caps the pipeline
- Easy to interpret
Limitations
- Binary at the threshold: a 3 and a 2 count the same
- Says nothing about ordering within the top k
- Depends entirely on grade quality
Common mistakes
- Measuring as an unrestricted role when the real system is access-controlled
- Publishing a point estimate with no interval
- Comparing across corpora
When to use it
Reach for it when
- Whenever retrieval changes
Avoid it when
- As the only metric — it ignores ranking and grounding
How it interacts with the rest of the system
Golden sets and graded relevanceNDCG and rank-aware scoringBootstrap confidence intervalsANN post-filtering under access control