Section 7 · Learn · Evaluation
Abstention and knowing when not to answer
Declining to answer when the retrieved evidence does not support one — and measuring whether the system does it for the right reason.
Declining to answer when the retrieved evidence does not support one — and measuring whether the system does it for the right reason.
What it is
An abstention gate inspects the retrieved context and the draft answer and suppresses the answer when support is insufficient.
Why it is needed
A confident wrong answer is worse than no answer in any domain where the output is acted on. Most benchmarks do not measure this at all.
How it works
The query set includes queries that are plausible, on-topic and genuinely unanswerable from the corpus. The metric is the proportion of those correctly declined.
In this project
30 of the 200 planned queries are unanswerable by construction. The metric is deliberately reported separately from headline retrieval numbers, and this is why: if approximate search under access control returns an empty candidate list, the system abstains because it received nothing, not because it judged the evidence insufficient. That inflates the abstention score for a reason that has nothing to do with the behaviour being measured.
Trade-offs
Advantages
- Directly measures a failure mode users care about
- Cheap to add to a query set
Limitations
- Requires genuinely unanswerable queries, which are hard to write
- Trivially gamed by abstaining more often
Common mistakes
- Reporting abstention without reporting answer coverage
- Letting an empty retrieval result count as a correct abstention
When to use it
Reach for it when
- The output is acted on rather than browsed
How it interacts with the rest of the system
ANN post-filtering under access controlRecall@k and retrieval metricsCitation resolution and quote verification