Section 7 · Learn · Retrieval
BM25 and lexical search
A ranking function that scores documents by term overlap with the query, weighted by how rare each term is and normalised for document length.
A ranking function that scores documents by term overlap with the query, weighted by how rare each term is and normalised for document length.
What it is
BM25 scores a passage by how often the query's terms appear in it, discounting terms that appear everywhere and correcting for the fact that longer passages contain more of everything by chance.
Why it is needed
Exact terms matter. Identifiers, statute numbers, defined contract terms and product names must match literally, and an embedding model will happily rank a paraphrase above the exact string you asked for.
How it works
Each term contributes idf x saturated term frequency. The saturation means the tenth occurrence of a word adds far less than the second, and the length normalisation prevents long documents winning on volume alone.
Formula
score(q,d) = sum over terms t in q of idf(t) * ( tf(t,d) * (k1+1) ) / ( tf(t,d) + k1 * (1 - b + b * |d|/avgdl) ) idf(t) = log( 1 + (N - df(t) + 0.5) / (df(t) + 0.5) )
- tf(t,d)
- how often term t appears in passage d
- df(t)
- how many passages contain t at all
- |d| / avgdl
- this passage's length against the average
- k1 = 1.5
- term-frequency saturation; higher means repetition keeps counting
- b = 0.75
- length normalisation strength; 0 disables it entirely
In this project
Implemented by hand rather than pulled from a library, because it is used to pool judgment candidates for the golden set and that scoring has to be auditable. Postings are stored in array('i') rather than lists of tuples: 51,310 chunks produce roughly 15 million postings, and the tuple form costs gigabytes of object overhead for no benefit. Index build takes about 5 seconds.
Trade-offs
Advantages
- No training, no model, no GPU
- Exact identifiers and rare terms rank correctly
- Scores are explainable term by term
Limitations
- No notion of meaning: a paraphrase scores zero
- Vocabulary mismatch between query and document is fatal
- Stopword and tokenisation choices quietly change results
Common mistakes
- Comparing BM25 scores across queries — they are not calibrated and have no fixed range
- Assuming a high score means relevance rather than term overlap
When to use it
Reach for it when
- Identifiers, codes, names and defined terms
- As one half of a hybrid retriever
Avoid it when
- Alone, when users ask in their own words
How it interacts with the rest of the system
Reciprocal Rank FusionHybrid retrievalEmbeddings and dense retrieval