Section 2

Methodology

How each number is produced, what it depends on, and where it can go wrong.

Metrics are implemented explicitly rather than through an evaluation framework. The scoring logic is the credibility of the project, so it is written to be read. Start with or .

2.1 Metrics

Grouped by family. The three-letter tag before each name is the family; the colour reinforces it but never carries it alone.

Retrieval

— Proportion of retrieved chunks that are actually relevant.
— Mean reciprocal rank of the first relevant chunk. Rewards getting one right answer high.
— Discounted cumulative gain over graded relevance, normalised against the ideal ranking.
— Proportion of queries where at least one chunk graded 2 or higher appears in the top 10.
— The same measure at rank 5. Harder, and more sensitive to reranking.

Grounding

— Agreement with the reference answer, judged against a published rubric.
— Whether the answer addresses the question that was actually asked.
— Proportion of citations that resolve to a chunk actually supporting the cited claim.
— Proportion of factual claims that carry a citation at all.
— Proportion of generated claims entailed by the retrieved chunks.

Abstention

— Proportion of genuinely unanswerable queries the system correctly declines.

Performance

— 95th percentile end-to-end time, retrieval through verification.

Cost

— Token counts multiplied by the rates in the run config. Local models cost zero, stated openly.

Point at a node for the metrics it carries and what moves them.

Figure 8. Retrieval quality feeds grounding and abstention, so a retrieval failure moves numbers in components that are not themselves at fault.

2.2 Confidence intervals

Every quality metric carries a bootstrapped 95% confidence interval over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks, and rightly.

2.3 One factor at a time

Each published variant changes exactly one factor from the baseline, so every difference is attributable to a single change rather than to a bundle of them.

2.4 Chunking

Two strategies behind one interface, so the comparison is a run variable rather than a rewrite. Both preserve page and section provenance, which is what lets a citation resolve to a location a reader can check.

Figure 6. The same document under both strategies. Fixed windows overlap and cut across boundaries; section-aware packing never crosses one.

2.5 How the golden set is built

The query set is model-drafted and partially human-verified. It is not hand-labelled and is not described as such. A stratified quarter of it — 25% of each category independently, not a random draw — is graded by hand, and the agreement rate between the model judgments and those human grades is published.

Figure 5. The judge never sees which passages a query was drafted from. Only the question and the passage text cross that line.

2.6 Known limitations

Stated here rather than discovered by readers. Each is added as it is established by measurement.

The golden set is model-drafted and only partially human-verified. The agreement rate is published beside it.
Access control applies differently to the two retrieval paths. Lexical search filters before ranking and loses no recall. Vector search filters after approximate selection, so a restrictive role can lose candidates exact search would have returned. Measured, and reported separately from headline numbers.
CUAD tenants are contract type, not counterparty. The 510 contracts come from 463 distinct filers, so per-counterparty tenants would hold about 1.1 documents each and isolation could not be tested meaningfully. Contract type is bounded, deterministic from the source, and semantically coherent, so contracts sharing a tenant genuinely resemble each other. Tenants below a chunk floor are folded into a semantic sibling for the same reason, and every document records both its final tenant and the unmerged original.
Two document shapes from one publisher, not three independent sources. CUAD contracts are themselves drawn from EDGAR, so they are a different shape from the same publisher. Deduplication against the filing set is mandatory rather than optional.