Section 7 · Learn · Evaluation
Bootstrap confidence intervals
Estimating how much a measured score would move if you had drawn a different query set of the same size.
Estimating how much a measured score would move if you had drawn a different query set of the same size.
What it is
Resample the query set with replacement many times, recompute the metric on each resample, and take the middle 95% of the resulting distribution.
Why it is needed
A score over 200 queries is one draw from a distribution. Reporting it as a fact invites a reviewer to point out that a different 200 queries would have given a different number.
How it works
No assumption of normality is needed. The resampling distribution stands in for the sampling distribution directly.
Formula
for b = 1..B:
Q*_b = sample |Q| queries from Q, with replacement
theta*_b = metric(Q*_b)
CI95 = [ percentile(theta*, 2.5), percentile(theta*, 97.5) ]
B = 1000- with replacement
- a query can appear twice in a resample; that is the point
- B = 1000
- resamples; more narrows the estimate of the interval, not the interval
In this project
Every quality metric carries a bootstrapped 95% interval over 1,000 resamples. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks, and rightly.
Trade-offs
Advantages
- No distributional assumptions
- Works for any metric you can compute
- Simple to implement correctly
Limitations
- Cannot fix a query set that is too small or unrepresentative
- Costly if the metric is expensive to recompute
Common mistakes
- Resampling judgments instead of queries — the query is the sampling unit
- Reporting an interval as if it covered corpus differences too
When to use it
Reach for it when
- Every published quality metric
Avoid it when
- Deterministic quantities of a single run, such as index build time
How it interacts with the rest of the system
Recall@k and retrieval metricsNDCG and rank-aware scoringGolden sets and graded relevance