Learn

What it is

An abstention gate inspects the retrieved context and the draft answer and suppresses the answer when support is insufficient.

Why it is needed

A confident wrong answer is worse than no answer in any domain where the output is acted on. Most benchmarks do not measure this at all.

How it works

The query set includes queries that are plausible, on-topic and genuinely unanswerable from the corpus. The metric is the proportion of those correctly declined.

In this project

30 of the 200 planned queries are unanswerable by construction. The metric is deliberately reported separately from headline retrieval numbers, and this is why: if approximate search under access control returns an empty candidate list, the system abstains because it received nothing, not because it judged the evidence insufficient. That inflates the abstention score for a reason that has nothing to do with the behaviour being measured.

Trade-offs

Common mistakes

• Reporting abstention without reporting answer coverage
• Letting an empty retrieval result count as a correct abstention

When to use it

How it interacts with the rest of the system

ANN post-filtering under access controlRecall@k and retrieval metricsCitation resolution and quote verification

What it is

The index has no knowledge of who is asking. It returns its ef_search nearest vectors by distance. Only then does the row-level security policy discard the rows this role may not see. The discarded slots stay empty.

Why it is needed

This is the most valuable finding in the project and it is almost invisible. The query succeeds. It returns fewer rows, or none. An empty result is indistinguishable from 'no evidence exists', which is exactly the conclusion an abstention gate will draw.

How it works

Filtering happens as a Filter node above the index scan in the query plan. The planner has no model of ef_search, so its row estimate for that node is fiction, and no error is raised at any point.

At a glance

In this project

Measured on the real corpus — 51,310 chunks, 47 tenants, real embeddings, PostgreSQL RLS, connected as a non-superuser role. At ef_search=40 recall falls from 0.850 unrestricted to 0.300 for a single-tenant role, and 6 of 12 queries return nothing at all while exact search under the identical policy returns a full result set. Widening the search recovers some of it and then plateaus at 0.667 against a 0.975 ceiling, with 4 queries still empty at ef_search=800.

A correction worth stating

An earlier measurement of this project on a SYNTHETIC corpus found that raising ef_search changed nothing at all. That does not replicate on real data. The synthetic corpus used generated tenant clusters with inter-tenant cosine of 0.014 — near-perfect separation — so a restricted role's neighbours were entirely other tenants at any search width. Real documents share vocabulary, boilerplate and structure, so tenant regions overlap and a wider search does reach permitted rows. The separated case was the worst case, not the typical one.

Why the lexical path does not have this problem

Full-text search applies the policy before ranking: the plan is a sort over an already-filtered scan, so ts_rank only ever orders permitted rows and no recall is lost. The two retrieval paths are both correct under the policy, but they are not affected identically — which matters when comparing them.

Trade-offs

Common mistakes

• Benchmarking as an unrestricted role and publishing the number as the system's recall
• Concluding a metric is fine because no error appeared
• Letting an empty candidate list inflate the abstention metric

When to use it

How it interacts with the rest of the system

HNSW and approximate nearest neighbour searchPostgreSQL Row-Level SecurityPartitioning by tenantAbstention and knowing when not to answerRecall@k and retrieval metrics

What it is

BM25 scores a passage by how often the query's terms appear in it, discounting terms that appear everywhere and correcting for the fact that longer passages contain more of everything by chance.

Why it is needed

Exact terms matter. Identifiers, statute numbers, defined contract terms and product names must match literally, and an embedding model will happily rank a paraphrase above the exact string you asked for.

How it works

Each term contributes idf x saturated term frequency. The saturation means the tenth occurrence of a word adds far less than the second, and the length normalisation prevents long documents winning on volume alone.

Formula

score(q,d) = sum over terms t in q of
  idf(t) * ( tf(t,d) * (k1+1) ) / ( tf(t,d) + k1 * (1 - b + b * |d|/avgdl) )

idf(t) = log( 1 + (N - df(t) + 0.5) / (df(t) + 0.5) )

tf(t,d)

how often term t appears in passage d

df(t)

how many passages contain t at all

|d| / avgdl

this passage's length against the average

k1 = 1.5

term-frequency saturation; higher means repetition keeps counting

b = 0.75

length normalisation strength; 0 disables it entirely

In this project

Implemented by hand in harness/golden/bm25.py rather than pulled from a library, because it is used to pool judgment candidates for the golden set and that scoring has to be auditable. Postings are stored in array('i') rather than lists of tuples: 51,310 chunks produce roughly 15 million postings, and the tuple form costs gigabytes of object overhead for no benefit. Index build takes about 5 seconds.

Trade-offs

Common mistakes

• Comparing BM25 scores across queries — they are not calibrated and have no fixed range
• Assuming a high score means relevance rather than term overlap

When to use it

How it interacts with the rest of the system

Reciprocal Rank FusionHybrid retrievalEmbeddings and dense retrieval

What it is

Resample the query set with replacement many times, recompute the metric on each resample, and take the middle 95% of the resulting distribution.

Why it is needed

A score over 200 queries is one draw from a distribution. Reporting it as a fact invites a reviewer to point out that a different 200 queries would have given a different number.

How it works

No assumption of normality is needed. The resampling distribution stands in for the sampling distribution directly.

Formula

for b = 1..B:
    Q*_b = sample |Q| queries from Q, with replacement
    theta*_b = metric(Q*_b)

CI95 = [ percentile(theta*, 2.5), percentile(theta*, 97.5) ]

B = 1000

with replacement

a query can appear twice in a resample; that is the point

B = 1000

resamples; more narrows the estimate of the interval, not the interval

In this project

Every quality metric carries a bootstrapped 95% interval over 1,000 resamples. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks, and rightly.

Trade-offs

Common mistakes

• Resampling judgments instead of queries — the query is the sampling unit
• Reporting an interval as if it covered corpus differences too

When to use it

How it interacts with the rest of the system

Recall@k and retrieval metricsNDCG and rank-aware scoringGolden sets and graded relevance

What it is

A 200-page filing cannot be one vector. It is split into passages — here targeting 512 tokens — each embedded and indexed independently.

Why it is needed

Chunking decides what is findable. A fact split across two chunks may be retrievable from neither. This is a real design decision, not a parameter.

How it works

Fixed-token windows slide through the token stream with overlap so a fact near a boundary appears whole in at least one chunk. Section-aware packing instead respects document structure and never lets a chunk span two sections.

At a glance

In this project

Both strategies are implemented behind one interface and run over the same corpus: 51,310 fixed-512 chunks and 59,579 section-aware. Page and section provenance is preserved through both, which is what lets a citation resolve to a location a reader can check.

A correction to a common claim

Section-aware chunking is often described as avoiding cuts mid-sentence. Measured here, it does not: it splits long sections internally using the same overlap, so it starts mid-sentence about as often as fixed windows do. What it actually guarantees is that no chunk spans two sections. Measured over one document per source, fixed-512 produced 4, 17 and 32 chunks crossing a section boundary; section-aware produced zero in every case, by construction.

Why the overlap exists

15% overlap means roughly 77 tokens repeat between consecutive chunks. Without it, a sentence spanning a boundary is truncated in both chunks and retrievable from neither. The cost is index size and some duplicate results.

Trade-offs

Common mistakes

• Chunking before parsing structure, losing page and section provenance
• Assuming section-aware prevents mid-sentence starts

When to use it

How it interacts with the rest of the system

Embeddings and dense retrievalProvenance and reproducibilityRetrieval-Augmented Generation

What it is

Two mechanical checks after generation: does the cited chunk id exist in the index for this run, and does the quoted text appear verbatim in that chunk.

Why it is needed

Models invent citations. They also paraphrase a quotation while presenting it as verbatim. Both are checkable without a model, and both should fail the response rather than reach a reader.

How it works

Citation resolution is a lookup. Quote verification is string containment first, with a model judge only where containment fails — because a legitimate quote can differ by whitespace or an ellipsis.

In this project

Planned for Phase 2 as part of the generation graph. Citation accuracy and citation coverage are defined in the metric content and reported separately: accuracy asks whether citations point at supporting passages, coverage asks whether factual claims carry a citation at all. A system can score well on one and badly on the other.

Trade-offs

Common mistakes

• Checking the citation resolves but not that it supports the claim
• Measuring accuracy without coverage, so a system citing almost nothing scores well

When to use it

How it interacts with the rest of the system

Constrained decoding and structured outputAbstention and knowing when not to answerRetrieval-Augmented Generation

What it is

A grammar compiled from the schema restricts which tokens may be sampled at each step, so the output cannot leave the grammar.

Why it is needed

A retry loop wastes tokens and latency and can loop forever. Constraining at the sampler makes malformed output impossible rather than unlikely.

How it works

The schema is compiled to a state machine. At each decoding step, tokens that would leave the grammar are masked to negative infinity before sampling.

In this project

vLLM 0.27.1 supports xgrammar, guidance, outlines and lm-format-enforcer, with enforcement at the sampler rather than a retry loop. Three findings shaped the design: the backend is process-wide and not per-request; auto silently cascades xgrammar to guidance to outlines, so two runs could use different decoders with no signal in the output, and the backend should be pinned explicitly; and constrained decoding guarantees grammar conformance, not full JSON Schema validity.

What it does not guarantee

Keywords a grammar cannot express — numeric minimum and maximum, minLength, minItems — are not enforced at the sampler. The output is guaranteed to parse and to match the structural grammar. It is not guaranteed to satisfy every assertion in the schema. The design consequence is to keep the contract structural and put semantic assertions in a validation pass afterwards.

Thinking tokens are charged against the output budget

On models that reason before answering, thought tokens count toward maxOutputTokens. The budget runs out mid-JSON and the response arrives truncated but parseable up to the cut, with only finishReason indicating anything is wrong. Pinning the thinking level low removed the truncation and halved latency.

Trade-offs

Common mistakes

• Assuming schema-valid means semantically valid
• Leaving the backend on auto and getting a silent fallback
• Ignoring finishReason and parsing truncated JSON into a real-looking record

When to use it

How it interacts with the rest of the system

Citation resolution and quote verificationModel pinning and reproducibilityRetrieval-Augmented Generation

What it is

Bulk fetching from public APIs with a shared request ceiling, resumable state, and per-document failure isolation.

Why it is needed

Public data providers rate-limit per IP. A run that trips the limit can be blocked for a day; a run that dies at document 200 of 300 without resuming costs hours.

How it works

A token bucket paces requests, Retry-After is honoured, transient failures back off, and each document is checkpointed so a restart resumes rather than restarts.

In this project

662 documents from three sources at a 9 requests/second ceiling. Two implementation details worth knowing: a token bucket that starts full fires a burst of rps requests instantly and only then settles, putting roughly twice the ceiling into the opening second — which is exactly the window a per-second limit measures — so capacity is pinned to 1.0 for strict pacing. And a daily quota is distinguished from a transient 429, because retrying a daily quota inside the same run cannot help and just burns the remaining attempts.

Integrity, not just existence

A resume check that only asks whether a file exists will skip a truncated download forever, and the failure surfaces much later as an inexplicable parser error. Content-Length is verified before the write, files are written through a .part file and renamed atomically, and resume compares the file on disk against the byte count recorded in the manifest.

Trade-offs

Common mistakes

• A token bucket that starts full
• Treating HTTP 200 as success when the body is an error page
• Assuming a file that exists is a file that is complete

When to use it

How it interacts with the rest of the system

Provenance and reproducibilityDeduplication by containment

What it is

Containment measures the fraction of the smaller document's content that appears in the larger one.

Why it is needed

An exhibit contract inside a 200-page filing is a complete subset of it, but Jaccard scores it near zero because the filing is far larger. Containment scores it 1.0.

How it works

Documents are reduced to overlapping word shingles. Containment is the fraction of the smaller document's shingles present in the larger.

Formula

containment(A, B) = |shingles(A)  n  shingles(B)| / |shingles(A)|

with A the smaller document.

Jaccard, for contrast:
  J(A, B) = |A n B| / |A u B|

For a 20 KB contract inside a 400 KB filing, containment = 1.0 and Jaccard = 0.05.

shingle

a sliding window of 5 consecutive words

1 in 16 sampling

keep a shingle when its hash mod 16 is 0 — uniform, so unbiased

threshold 0.80

the containment above which a duplicate is declared

In this project

Five-word shingles, hash-sampled at one in sixteen to keep the index for 140 MB of filing text in the low millions of entries rather than tens of millions. The sampling is uniform, so the estimate is unbiased. Validated with controls before its result was trusted: a contract embedded in a filing scored 1.0000, the same contract reformatted scored 1.0000, an unrelated contract scored 0.0045.

Trade-offs

Common mistakes

• Using Jaccard for subset detection
• Trusting a dedup result without a positive control — a detector that finds nothing may be one that cannot find anything

When to use it

How it interacts with the rest of the system

Provenance and reproducibilityCorpus acquisition and rate limiting

What it is

An embedding model converts a passage into a fixed-length list of numbers — here 384 of them. Similar meanings produce nearby vectors, so retrieval becomes a nearest-neighbour search in that space.

Why it is needed

Users do not use the document's vocabulary. Someone asks about 'ending a contract early' when the document says 'termination for convenience'. Lexical search scores that zero; dense retrieval finds it.

How it works

The model is a small transformer. The vector for the CLS token is taken as the passage representation and normalised to unit length, which makes the dot product of two vectors equal to their cosine similarity — so similarity is one multiply-accumulate rather than a division.

Formula

cos(a,b) = (a . b) / (|a| |b|)

With both vectors L2-normalised, |a| = |b| = 1, so:

cos(a,b) = a . b

And pgvector's cosine distance operator <=> returns 1 - cos(a,b).

a . b

dot product, the sum of elementwise products

384

the dimensionality bge-small produces

<=>

pgvector's cosine distance operator; smaller is more similar

In this project

bge-small-en-v1.5 embeds all 51,310 chunks in about 1.5 minutes on an RTX 5060 at 558 chunks/second, producing a 79 MB float32 matrix. Verified after generation: all L2 norms within 0.0006 of 1.0, no NaN rows, no zero rows. The embeddings are not committed — they are reproducible from the committed corpus manifest.

Trade-offs

Common mistakes

• Forgetting to normalise, then using dot product as if it were cosine
• Mixing vectors from two different models in one index
• Assuming a high cosine means relevance rather than surface similarity

When to use it

How it interacts with the rest of the system

HNSW and approximate nearest neighbour searchReciprocal Rank FusionHybrid retrievalChunking strategies

What it is

A golden set pairs queries with judgments: for each query, which passages are relevant and how relevant, usually on a 0-3 scale.

Why it is needed

Every retrieval metric is a function of these grades. If the grades are wrong, the metrics are wrong in a way no amount of engineering downstream can detect.

How it works

Queries are drafted across categories, candidates are pooled with a lexical retriever, and each candidate is graded. A stratified sample is then re-graded by a human and the agreement rate published.

At a glance

In this project

Targets 200 queries across five categories: single-chunk factual, multi-chunk synthesis, cross-document, tenant-scoped, and genuinely unanswerable. Candidates are pooled 50 per query by BM25, with the passages a query was drafted from always included so a known positive is always in the pool. The set is model-drafted and only partially human-verified, and is described that way rather than as hand-labelled.

Trade-offs

Common mistakes

• Describing a model-drafted set as hand-labelled
• Publishing an agreement rate from a sample that does not discriminate

When to use it

How it interacts with the rest of the system

LLM-as-judge and blind judgingRecall@k and retrieval metricsNDCG and rank-aware scoringAbstention and knowing when not to answer

What it is

HNSW builds a layered graph where each vector links to its neighbours. Search enters at a sparse top layer, greedily walks toward the query, then descends to denser layers to refine.

Why it is needed

Exact nearest-neighbour search compares the query to every vector. At 51,310 chunks that is tolerable; at ten million it is not. HNSW turns a linear scan into a graph walk.

How it works

Two knobs matter. m is how many links each node keeps, fixed at build time. ef_search is how many candidates the search keeps in flight, set per query — larger means more of the graph explored, better recall, more time.

Formula

recall@k = |ANN top-k  n  exact top-k| / k

Measured on this corpus, unrestricted:
  ef_search=40   -> 0.850
  ef_search=100  -> 0.942
  ef_search=200  -> 0.967
  ef_search=800  -> 0.975

m

links per node, fixed at build; higher is better recall and a larger index

ef_construction

candidate list size while building; higher is a better graph and a slower build

ef_search

candidate list size while querying; the runtime recall/latency dial

In this project

Built in PostgreSQL via pgvector 0.8.6 with m=16, ef_construction=64. On the real corpus the index builds in about 8 seconds. Measured here: at the default ef_search=40 an unrestricted role recovers 0.850 of what exact search returns — approximate means approximate, and 15% is lost before access control is even involved.

Trade-offs

Common mistakes

• Assuming ANN results equal exact results
• Benchmarking at ef_search=40 and reporting it as retrieval quality
• Filtering after the index has already chosen — see ANN post-filtering

When to use it

How it interacts with the rest of the system

Embeddings and dense retrievalANN post-filtering under access controlPostgreSQL Row-Level SecurityPartitioning by tenant

What it is

Two retrievers with different failure modes are run over the same corpus and their ranked lists are combined into one.

Why it is needed

Lexical search fails on paraphrase; dense search fails on rare exact strings. Their errors are largely independent, so the union recovers passages either alone would lose.

How it works

Both retrievers return a ranked list. The lists are merged by rank rather than score, because the two scoring scales have nothing in common.

At a glance

In this project

Measured on real queries against the real corpus: of the top 20 from each path, the two agree on only 5 on average. Three quarters of each list is unique to that path. That number is the entire argument for fusion, and it is visible in the walkthrough demo.

Trade-offs

Common mistakes

• Merging by raw score instead of rank
• Assuming more candidates always helps — precision falls

When to use it

How it interacts with the rest of the system

BM25 and lexical searchEmbeddings and dense retrievalReciprocal Rank FusionCross-encoder reranking

What it is

The server sends HTML; the client renders the same tree and attaches it to the existing DOM. If the two disagree, the server HTML is discarded and the page is re-rendered from scratch.

Why it is needed

A mismatch throws away the work the server did and re-renders everything on every load. The only symptom is a minified error in the console.

How it works

HTML forbids flow content inside a paragraph. When a parser meets a div inside a p it auto-closes the paragraph, so the DOM the browser builds does not match the tree the framework rendered.

At a glance

In this project

This project hit exactly that: an inline metric reference rendered its full panel — dialog, section, headings, pre, lists — at the reference point, which sat inside a paragraph of prose. The fix was structural rather than per-tag: the inline trigger returns phrasing content only, and panels render once per page outside the prose tree. A build-time guard now scans the emitted HTML with an explicit element stack and fails the build on flow content inside a paragraph, and it was verified by reintroducing the bug and confirming it caught all four violations.

Trade-offs

Common mistakes

• Rendering block content at an inline reference point
• Testing only in development, where production behaviour differs
• Patching individual tags rather than fixing the structure

When to use it

How it interacts with the rest of the system

Static export and precomputed interaction

What it is

A model grades each query-passage pair against a written rubric, standing in for a human annotator at a fraction of the cost.

Why it is needed

Grading 10,000 candidates by hand is not feasible. A model does it in hours for a couple of dollars. The question is whether the grades mean anything.

How it works

The rubric is published. The judge sees only the question and the passage text. A stratified sample is re-graded by a human and the agreement rate published beside the results.

In this project

Drafting and judging use different models, and the judge is blind by construction: the judging prompt interpolates only the relevance statement, the question and the passages. Whether a passage was one the query was drafted from is recorded in the output but never sent to the judge. Model references are pinned to dated snapshots, and every record carries name@snapshot rather than a bare name.

Why blindness is the whole design

If the judge can see which passages a query was written from, it agrees with them, and the agreement rate then measures conformity rather than correctness. The number goes up, which is exactly what makes it dangerous — a high agreement rate looks like a well-constructed set.

What happened here

The human verification sample is currently unusable and the site says so rather than publishing it. Of 32 verified judgments, 30 were graded 3 and 2 were graded 2 — 94% a single value — while the model graded 91% of the same pool as 0. A sample that does not discriminate cannot measure a judge in either direction, so the agreement rate is withheld with the reason stated, pending re-grading.

Trade-offs

Common mistakes

• Letting the judge see the drafter's labels
• Using a -latest alias, so re-running months later silently changes the judge
• Publishing an agreement rate from a degenerate sample

When to use it

How it interacts with the rest of the system

Golden sets and graded relevancefaithfulnessModel pinning and reproducibility

What it is

Providers expose both moving aliases and dated snapshots. An alias resolves to whatever is current at call time.

Why it is needed

If a judge sets a published number, a floating reference means re-running months later silently uses a different judge, and the number moves with no explanation in the output.

How it works

Resolve the alias to its snapshot at run start and record that, not the name that was requested.

At a glance

In this project

Verified against the provider's model list: a -latest alias reports a floating label as its version, while a dated model reports a snapshot. Every query and judgment records name@snapshot, and a run config captures both model references, the candidate depth, the batch size, the seed and the corpus file.

Trade-offs

Common mistakes

• Using a -latest alias anywhere a number is published
• Recording the requested name rather than the resolved snapshot

When to use it

How it interacts with the rest of the system

LLM-as-judge and blind judgingConstrained decoding and structured output

What it is

Normalised Discounted Cumulative Gain sums the relevance of each result, discounted by how far down the list it sits, then divides by the best possible ordering.

Why it is needed

Recall treats rank 1 and rank 10 identically. For a generator given only the top few passages, that difference decides whether the answer is grounded.

How it works

Gain grows exponentially with grade, so a 3 counts seven times a 1. The discount is logarithmic in rank. Dividing by the ideal ordering puts the result on a 0-1 scale.

Formula

DCG@k = sum over i=1..k of  ( 2^grade(c_i) - 1 ) / log2(i + 1)

NDCG@k = DCG@k / IDCG@k

Gain by grade:   0 -> 0    1 -> 1    2 -> 3    3 -> 7
Discount by rank: 1 -> 1.00  2 -> 0.63  3 -> 0.50  10 -> 0.29

c_i

the passage at rank i

IDCG@k

DCG of the best ordering possible for this query

In this project

Reported at k=10 alongside Recall@10 and MRR@10. Using graded relevance rather than binary is why the golden set grades 0-3 instead of relevant/not.

Trade-offs

Common mistakes

• Comparing NDCG across different grading scales
• Forgetting that IDCG depends on how many relevant passages exist

When to use it

How it interacts with the rest of the system

Golden sets and graded relevanceRecall@k and retrieval metricsCross-encoder reranking

What it is

A LIST partition on tenant keeps each tenant's rows physically separate, and each partition carries its own vector index.

Why it is needed

If an index contains only rows this role may see, there is nothing to post-filter away. The recall cliff disappears because the cause disappears.

How it works

The planner prunes partitions the query cannot match, so only permitted partitions are scanned. Passing the permitted tenant list as an explicit predicate enables that pruning at plan time.

Compared

In this project

Recommended for Phase 2 after the post-filtering measurements. Two things do different jobs here and it is worth being precise: row-level security is the correctness boundary, and the explicit tenant predicate is purely a performance hint. In the earlier synthetic measurement recall was identical with and without the predicate — the predicate never widened access — but plan-time pruning cut latency from 38.5 ms to 1.03 ms. The security test suite must therefore drop the predicate and still pass; that is what proves it is an optimisation and not the boundary.

Trade-offs

Common mistakes

• Treating the explicit predicate as the security boundary
• Partitioning so finely that a partition holds fewer rows than ef_search

When to use it

How it interacts with the rest of the system

PostgreSQL Row-Level SecurityANN post-filtering under access controlMulti-tenancy and tenant assignmentHNSW and approximate nearest neighbour search

What it is

Provenance is the chain from a published figure back to the raw input that produced it: source URL, checksum, script, commit.

Why it is needed

A benchmark nobody can re-run is a marketing claim. The corpus is also too large to redistribute, so the manifest has to stand in for the data.

How it works

Each document records source, identifier, URL, checksum, page count, tenant and licence. Each derived dataset records the script that produced it and the command that regenerates it.

In this project

corpus_manifest.json is committed; the 719 MB of raw documents are not. Replay the URL, verify the checksum. Every precomputed interactive dataset carries describes, generated_by.script, generated_by.regenerate and generated_by.commit — 23 of 23 datasets verified as carrying full provenance. Page counts distinguish real page breaks from estimated ones, because a synthesised number presented as measured is a fabrication regardless of intent.

Trade-offs

Common mistakes

• Committing derived data without the script that made it
• Prose quoting figures that drift from the data beside it

When to use it

How it interacts with the rest of the system

Deduplication by containmentCorpus acquisition and rate limiting

What it is

RAG splits question answering into two jobs: find the right passages, then write an answer using only those passages. The model is never asked to recall facts from its weights.

Why it is needed

A language model trained months ago cannot know your documents, and asking it to recall specifics produces fluent, confident, wrong answers. Retrieval puts the actual source text in front of it, and makes every claim checkable against a citation.

How it works

A query is embedded and searched against an index. The top passages are re-ranked, then passed to the model as context with instructions to answer only from that context. Citations are resolved back to the passages, quotes are verified against source text, and the system abstains when the evidence is insufficient.

In this project

This project is a full RAG pipeline measured end to end rather than a retriever measured in isolation. The corpus is 662 public documents — SEC 10-K filings, commercial contracts and IETF RFCs — chunked to 51,310 passages. Retrieval, grounding, abstention, latency and cost are all measured, because a system that retrieves perfectly and then hallucinates is not a working system.

Why measure the whole pipeline

Retrieval quality and answer quality are different things. A reranker that improves Recall@10 can still produce worse answers if it promotes passages the generator misreads. Measuring only retrieval hides that; measuring only answers hides which stage caused the failure.

Where RAG systems actually fail

In practice the failures cluster in three places: the retriever never surfaces the passage (a recall failure), the generator ignores the passage it was given (a grounding failure), or the system answers confidently when nothing relevant was retrieved (an abstention failure). Each has its own metric family here.

Trade-offs

Common mistakes

• Measuring retrieval alone and assuming answers follow
• Treating an empty retrieval result as evidence of absence
• Assuming metrics from one corpus transfer to another

When to use it

How it interacts with the rest of the system

Chunking strategiesEmbeddings and dense retrievalBM25 and lexical searchReciprocal Rank FusionCross-encoder rerankingAbstention and knowing when not to answerCitation resolution and quote verification

What it is

Recall@k is the proportion of queries for which at least one sufficiently relevant passage appears in the top k.

Why it is needed

It is the ceiling on everything downstream. A passage the retriever never returns cannot be reranked, cited or quoted.

How it works

For each query, look at the top k, ask whether any of them is graded 2 or higher, and average that across the query set.

Formula

Recall@k = |{ q in Q : max grade(c) >= 2 for c in top_k(q) }| / |Q|

MRR@k = (1/|Q|) * sum over q of 1 / rank of first relevant

Context precision = |{ c in top_k : grade(c) >= 2 }| / k

Q

the answerable subset of the query set

grade(c)

graded relevance 0-3 from the golden set

>= 2

the relevance boundary: contains a substantial part of the answer

In this project

Measured at k=5 and k=10. Reported with a bootstrapped 95% confidence interval, and always labelled with the corpus it was measured on, because retrieval metrics are properties of a corpus-and-query-set pair rather than of an architecture.

Trade-offs

Common mistakes

• Measuring as an unrestricted role when the real system is access-controlled
• Publishing a point estimate with no interval
• Comparing across corpora

When to use it

How it interacts with the rest of the system

Golden sets and graded relevanceNDCG and rank-aware scoringBootstrap confidence intervalsANN post-filtering under access control

What it is

A bi-encoder embeds query and passage separately and compares vectors. A cross-encoder feeds both into one model at once, so every query token can attend to every passage token.

Why it is needed

Separate encoding forces each passage into a single vector before it knows what was asked. A cross-encoder decides relevance with the question in hand, which is strictly more informative and measurably more accurate.

How it works

Too expensive to run over a whole corpus — it is one forward pass per query-passage pair. So it runs only over the shortlist retrieval already produced, typically the top 50, and reorders it.

Flow

1. 151,310 chunks
2. 2retrieve top 50
3. 3cross-encode 50 pairs
4. 4reordered top 10

Cost grows with shortlist depth, not corpus size. That is what makes the stage affordable.

In this project

bge-reranker-base runs over the top 50 fused candidates in harness/interactive/retrieval.py. The walkthrough shows movement per result, so the reader can see how much reordering the reranker actually does and judge whether it earned its latency.

Trade-offs

Common mistakes

• Reranking a shortlist so shallow the right passage was never in it
• Ignoring the latency it adds to p95

When to use it

How it interacts with the rest of the system

Reciprocal Rank FusionHybrid retrievallatency

What it is

A policy is a boolean expression attached to a table. PostgreSQL adds it to every query against that table automatically, as a security-barrier condition the planner may not optimise around.

Why it is needed

Filtering in application code means every query path must remember to filter. One forgotten WHERE clause is a data leak. Enforcing in the database makes the rule structural: there is no query that bypasses it.

How it works

The policy typically checks the row's tenant against a session variable set per request. It is evaluated before any user-supplied condition, so a crafted predicate cannot see rows the policy excludes.

In this project

Policies check the row's tenant against an ACL table keyed on a session setting. Every measurement connects as zeroth_app, which is NOSUPERUSER NOBYPASSRLS, and the demo script asserts rolbypassrls is false before taking a single reading.

Two silent bypasses

A superuser is exempt from row-level security entirely. Connect as postgres and every policy silently does nothing — no error, no warning, and every access-control test passes for the wrong reason. Separately, a table's OWNER is also exempt unless FORCE ROW LEVEL SECURITY is set, because relforcerowsecurity defaults to false. Both were confirmed here: running a probe as the table owner returned recall of 0.95 with no restriction applied at all.

The ACL table is itself readable

If a policy reads an ACL table, the querying role needs SELECT on it, which means any query can enumerate the entire authorisation matrix. Putting it behind a SECURITY DEFINER function or its own policy is the fix.

Trade-offs

Common mistakes

• Connecting as postgres and concluding the tests pass
• Forgetting FORCE ROW LEVEL SECURITY
• Assuming correct enforcement means unchanged recall

When to use it

How it interacts with the rest of the system

ANN post-filtering under access controlPartitioning by tenantMulti-tenancy and tenant assignment

What it is

Each list contributes 1/(k + rank) for every item it ranks. Contributions are summed, and the total orders the fused list.

Why it is needed

This is the step most people get wrong. BM25 scores are unbounded and query-dependent; cosine similarities sit in a narrow band near 1. Normalising them to a common scale requires assumptions that are wrong in different ways for every query. Rank sidesteps the problem entirely — rank 3 means rank 3 in any scale.

How it works

The constant k (60 here, the value from the original paper) damps the influence of the very top ranks. Without it, a single first place would dominate everything. With it, appearing in both lists at middling rank beats appearing first in one and nowhere in the other.

Formula

RRF(d) = sum over lists L of  1 / (k + rank_L(d))

k = 60

A passage at rank 1 in one list only:   1/61          = 0.016393
A passage at rank 5 in both lists:      2 * 1/65      = 0.030769

rank_L(d)

1-indexed position of d in list L; absent means no contribution

k = 60

damping constant from the original paper; larger flattens the top ranks further

In this project

Implemented in harness/interactive/retrieval.py and shown in the walkthrough: expanding a fused result reveals its actual arithmetic, for example lexical 1/(60+3) plus dense 1/(60+11) equals 0.029968. The reader watches ranks combine rather than reading the formula.

Trade-offs

Common mistakes

• Using 0 for a missing rank instead of no contribution
• Tuning k per query, which overfits immediately

When to use it

How it interacts with the rest of the system

BM25 and lexical searchEmbeddings and dense retrievalHybrid retrievalCross-encoder reranking

What it is

Every route is rendered to HTML at build time. There is no server process, no API and no database behind the site.

Why it is needed

Hosting is free and the attack surface is essentially zero. The constraint is that nothing can run a live query.

How it works

Interactive elements replay real state captured offline: run the pipeline, capture the intermediate results, commit them as JSON, and let the client move between them.

In this project

The site renders committed JSON and never queries the platform. Interactive demos read from data/interactive/, where each dataset carries the script that produced it and the command that regenerates it. The demos degrade honestly: with JavaScript disabled the reader still gets the full table of figures rendered server-side, not a blank box.

Trade-offs

Common mistakes

• Synthesising demo data because the real pipeline is not ready
• Interactive states with no number attached

When to use it

How it interacts with the rest of the system

Provenance and reproducibilityHydration and HTML nesting

What it is

A tenant is the unit of isolation: a role is granted some tenants and must never see the rest.

Why it is needed

Testing access control needs tenants that actually differ and that hold enough documents for a result set to be measurable. One document per tenant proves nothing.

How it works

Each document is assigned a tenant deterministically from its source, and the assignment is recorded per document so it can be audited.

At a glance

In this project

47 tenants over 662 documents. SEC filings partition by filing company, which is the natural unit. RFCs partition by working group. CUAD could not: its 510 contracts come from 463 distinct filers, so one tenant per counterparty would hold about 1.1 documents each. Contract type is used instead — bounded, deterministic from the filename, and semantically coherent, so contracts sharing a tenant genuinely resemble each other, which is the property that makes retrieval under access control behave like the real system rather than like random filtering. Tenants below a 500-chunk floor are folded into a semantic sibling, and every document records both its final tenant and the unmerged original.

Trade-offs

Common mistakes

• Tenants so small that per-tenant indexes are meaningless
• Hash-bucketing tenants, which destroys the clustering that makes tests realistic

When to use it

How it interacts with the rest of the system

PostgreSQL Row-Level SecurityPartitioning by tenantANN post-filtering under access control