NDCG and rank-aware scoring

A metric that rewards putting the most relevant passages highest, not merely including them somewhere.

What it is

Normalised Discounted Cumulative Gain sums the relevance of each result, discounted by how far down the list it sits, then divides by the best possible ordering.

Why it is needed

Recall treats rank 1 and rank 10 identically. For a generator given only the top few passages, that difference decides whether the answer is grounded.

How it works

Gain grows exponentially with grade, so a 3 counts seven times a 1. The discount is logarithmic in rank. Dividing by the ideal ordering puts the result on a 0-1 scale.

Formula

DCG@k = sum over i=1..k of  ( 2^grade(c_i) - 1 ) / log2(i + 1)

NDCG@k = DCG@k / IDCG@k

Gain by grade:   0 -> 0    1 -> 1    2 -> 3    3 -> 7
Discount by rank: 1 -> 1.00  2 -> 0.63  3 -> 0.50  10 -> 0.29

c_i: the passage at rank i
IDCG@k: DCG of the best ordering possible for this query

In this project

Reported at k=10 alongside Recall@10 and MRR@10. Using graded relevance rather than binary is why the golden set grades 0-3 instead of relevant/not.

Trade-offs

Advantages

Rank-aware
Uses the full grade scale
Bounded 0-1 and comparable across queries

Limitations

The exponential gain is a convention, not a derivation
Needs graded judgments
Harder to explain than recall

Common mistakes

Comparing NDCG across different grading scales
Forgetting that IDCG depends on how many relevant passages exist

When to use it

Reach for it when

Ordering matters, which for RAG it always does

Avoid it when

Only binary judgments are available

How it interacts with the rest of the system

Golden sets and graded relevanceRecall@k and retrieval metricsCross-encoder reranking

← All topics