A benchmark is only as trustworthy as the system underneath it. This is that system: what was chosen, what it was measured at, and what each choice cost.
Every figure here is either read from a committed file or recorded in the build log. Nothing is projected, rounded up, or copied from a vendor benchmark. Where a number has not been measured yet the entry says so instead of estimating one.
Everything runs on one machine: NVIDIA RTX 5060 Laptop, 8 GB VRAM. The constraint is the point. Every decision below — fp16, batch size, model choice, co-residency — was forced by an 8 GB budget rather than chosen on a machine where it did not matter.
9.1 Decisions, and what they cost
Each entry names the alternative it was chosen over. A decision recorded without its alternative is a preference; recorded with one, it is an argument that can be checked.
Measured on a tuning sweep: defaults reached 0.836 overall recall, the tuned parameters reached 0.998.
Cost
Index build cost roughly triples. Paid once per ingest, never on the query path.
Why
A vector index shipped on defaults is an untested assumption. The sweep turned it into a number with a price attached.
Fusion
Reciprocal rank fusion over ranks, k=60
argued, not measured
Instead of
Normalising and summing BM25 and cosine scores
Evidence
BM25 scores and cosine similarities are not on comparable scales; any normalisation encodes an assumption that is wrong differently for every query.
Cost
Discards score magnitude, keeping only order.
Why
Fusing by position needs no assumption about either scale. It is the cheaper claim to defend.
Generation
5 passages of 700 characters, max 2,048 output tokens, temperature 0
measured
Instead of
Ten passages of 1,200 characters
Evidence
A 3B model given ten 1,200-character passages produced no claims at all. Fewer, shorter passages left it enough attention to copy a quote accurately.
Cost
Less context per answer, so recall of the retriever matters more.
Why
Verbatim quoting is what the citation check verifies. A context budget that defeats quoting defeats the verification.
Generation
Schema-constrained decoding via XGrammar under vLLM
argued, not measured
Instead of
Asking for JSON in the prompt and parsing what comes back
Evidence
The answer schema becomes a decoding constraint rather than a request, so citations are parseable by construction.
Cost
Ties generation to a serving stack that supports grammars.
Why
A citation format that depends on the model's goodwill is not a format.
Embedding
fp16, batch 64, on bge-small-en-v1.5 (384-dimension)
measured
Instead of
fp32, or a larger embedding model
Evidence
51,310 chunks embedded in 1.5 minutes at 558 chunks/s, producing 79 MB of vectors. 6.87 GB of the 8 GB budget remained free with the embedder loaded.
Cost
A smaller embedding model than the leaderboards favour.
Why
Headroom was the requirement: the embedder and the generator have to be co-resident on one 8 GB card.
Access control
PostgreSQL row-level security inside the retrieval query, under a NOSUPERUSER NOBYPASSRLS role
measured
Instead of
Filtering results in application code after retrieval
Evidence
The application connection asserts rolsuper=false and rolbypassrls=false before it is handed to a caller. A superuser is exempt from RLS silently — no error, no warning.
Cost
Post-filtering an approximate index costs recall, which is measured and reported separately rather than folded into headline numbers.
Why
Access control enforced in application code is a control that a bug can skip. Enforced in the query, it is a control the database applies.
Access control
One partition and one index per tenant
argued, not measured
Instead of
A single index across all tenants on the parent table
Evidence
A partitioned index on the parent is built across every tenant, which loses the property that makes partitioning useful.
Cost
More indexes to build and maintain.
Why
A tenant's data being in its own partition is what makes isolation structural rather than a WHERE clause.
9.2 How the numbers are kept honest
The measurement apparatus is the part of this project that took the longest and is the easiest to skip. Every practice below exists because skipping it produces a number that looks fine and is not.
One factor per variant
Each published run differs from the baseline by exactly one configuration key, so any difference is attributable to that key. The comparison view flags it when more than one differs.
Bootstrapped 95% confidence intervals
1,000 resamples of the query set on every quality metric. A delta whose intervals overlap is reported as not separated rather than given a number that looks decisive.
The judge is measured, not trusted
Relevance grades are drafted by a pinned model and a stratified sample is re-graded by hand. Of 32 judgments checked so far the human and the model agreed on 4. No retrieval score is published until that is understood.
Failure modes are published, not hidden
Nine documented failure modes, each marked observed or prevented-by-design, each naming the metrics it corrupts and how it was detected.
Every figure carries its provenance
Script, regenerate command, commit and corpus id sit beside each figure, so a number can be traced to the code that produced it.
Absent numbers stay absent
No placeholder figures and no illustrative values anywhere. A run that has not happened renders as an empty state naming the phase that delivers it.
9.3 Guardrails
A retrieved passage is untrusted input. It reaches a prompt, and anything in it that reads as an instruction can be followed. These are the boundaries that assumption produces.
Instruction stripping at ingestion
Six narrow pattern classes — override, role reassignment, chat-role markers, tag injection, exfiltration, answer override. Removals are counted and marked in the chunk, never silent, and the patterns are deliberately narrow so ordinary contract and standards prose survives.
Retrieved text cannot forge prompt structure
Each passage is fenced and the fence literal is stripped from the body first, so a document cannot impersonate the header of a passage that was never retrieved.
Citations must resolve to passages actually shown
A cited chunk id is checked against the set the model was given. A citation to anything else fails rather than being displayed.
Quotes are checked character by character
Each quoted span is matched against the source text, with the longest verbatim run recorded so an altered quote is distinguishable from an invented one.
Abstention is a measured outcome
The system declines when the evidence does not support an answer, and how often it declines correctly is one of the reported metric families rather than a footnote.
9.4 The stack, pinned
Versions are pinned because a benchmark whose dependencies float is not reproducible. A model that silently updates changes every number measured against it.