Zeroth

A reproducible benchmark of end-to-end pipeline quality, measured on a public corpus.

Anant Sharma · 2026

Abstract

Zeroth is an open reconstruction of a production confidential-document retrieval platform, rebuilt from scratch over public documents so the design can be inspected and argued with. Every number published here was measured on the corpus described in the methodology, and applies only to it.

The benchmark measures a full pipeline rather than a retriever in isolation: hybrid retrieval, cross-encoder reranking, constrained generation, citation resolution, quote verification, and abstention. Access control is enforced inside the retrieval query itself, and its effect on measured recall is reported separately rather than folded into headline numbers.

Point at a stage for what it does. Solid = implemented; dashed = planned, with the phase that delivers it.

Figure 1. The pipeline under measurement. Solid stages are implemented; dashed stages are planned, marked with the phase that delivers them.

What is measured

Retrieval quality by and ; grounding by and ; and , which is the one most benchmarks leave out — whether the system declines when the evidence is not there.

Figure 7. Corpus composition, read from the committed manifest rather than hardcoded.

What has been measured

One result exists. Enforcing access control inside an approximate retrieval query costs recall in proportion to how restrictive the role is: at the default search width a role seeing all 47 tenants recalls 0.850 of what exact search returns, and a single-tenant role recalls 0.300 — returning nothing at all for 6 of 12 queries that exact search answers under the same policy.

It needs no relevance judgments, because exact search is its own ground truth. That is why it is published and the quality metrics below are not. Read the finding in full, including the earlier measurement it contradicts.

Results

The evaluation harness is built and the security suite passes. What is missing is relevance judgments: the query set has twelve questions and only five carry graded candidates, because grading the rest costs API quota the project has currently exhausted. Five queries produce confidence intervals half a point wide, which is not a result — so nothing is published here rather than published with a caveat.

Completing the set is roughly two dollars of inference and half an hour. It is the smallest blocker in the project and the reason this table is empty.

In progress — Phase 5

Nine configurations, each differing from the baseline by exactly one factor, so every difference is attributable to a single change.

Waiting on relevance judgments for the query set