Section 11

About

Zeroth is an open reconstruction of a production confidential-document retrieval platform, rebuilt over public documents so the architecture can be inspected, measured and argued with.

1 · Project overview

The original system was built for an employer over a private corpus and cannot leave. This is a from-scratch rebuild of the same architecture over public documents, together with a public evaluation board that measures it. Every number published here was measured on the corpus described in the methodology and applies only to it.

That framing is technical rather than merely ethical. Retrieval metrics are properties of a corpus-and-query-set pair, not of an architecture, so numbers measured here cannot validate or stand in for numbers measured on a different corpus.

2 · What this is, and what it is not

It would be easy to call this a research paper. It is not one yet, and the distinction is worth stating plainly rather than leaving a reader to discover it.

A reproducible retrieval benchmark, published methodology-first.

The corpus, the golden set, the guardrails, the failure modes and the evaluation apparatus are all published BEFORE any number is. That order is unusual — benchmarks normally publish results and leave the method implicit — and it is the part a reader can check today.

What it is not

Digital research paper: A paper reports findings. This has none yet — the board is empty, and deliberately so. Calling it a paper before the first run would be claiming the one thing it does not have.
RAG demo: Too weak in the other direction. A demo does not ship a 246-case red-team suite, a plan guard, bootstrapped confidence intervals, or a published golden set with its judge's error rate measured.
Technical case study: Closer, but a case study describes a system that already shipped. This measures one, and the measurement apparatus is most of the work.

What is actually differentiated

Four things, and the caveat beside each is part of the claim rather than a hedge attached to it.

differentiated

What access control costs approximate retrieval, measured

An approximate index returns nearest neighbours by distance, and only then does the policy discard the ones a role may not see. Nothing refills the discarded slots. The effect is measured here per role across a search-width sweep and reported separately from headline numbers rather than folded into them.

The mechanism is well understood in principle. What is uncommon is publishing the measurement rather than mentioning the risk.

differentiated

The silent planner flip, and a guard against it

The same retrieval query executes as either an HNSW index scan or a sequential scan depending on table statistics and machine-level planner settings. The flip is silent, and it flips TOWARD exact search — so recall rises and nothing looks broken. A run simply scores higher on one machine than another. The guard pins the planner, asserts the executed plan shape, fails the run on mismatch, and records an md5 fingerprint so two runs can be compared for planner equivalence.

One of the two documented triggers does not reproduce at the current corpus size, and that is recorded rather than quietly dropped.

uncommon practice

The judge is measured, not trusted

Relevance grades are drafted by a pinned model, a stratified sample is re-graded by hand, and the disagreement rate is published — 4 agreements in 32 checks. Checking that set against itself then found contradictions the disagreement rate alone had hidden.

Verification is not blind: the tool shows the model's grade before asking for the human's. A blind pass would be a stronger instrument.

positioning

Refusing to publish is the argument

No retrieval score appears anywhere, because the golden set is small and disagrees with itself. An empty results table with the reason stated is a stronger claim about method than a filled one with a footnote.

What is established technique

Hybrid retrieval, reciprocal rank fusion, cross-encoder reranking — all published technique, implemented carefully and claimed as nothing more.
HNSW approximate nearest-neighbour search via pgvector, and bge embedding and reranking models.
Schema-constrained decoding for parseable citations.
Row-level security as a Postgres feature. The application of it to a retrieval path, and the measurement of its cost, is the part that is not routine.
The corpus pipeline: fetch, parse, chunk, deduplicate, checksum. Careful, conventional, and not a contribution.

Why this is not a PDF

The pipeline is steppable: A reader can follow one chunk through lexical, dense, fused and reranked, and see where each stage found or nearly lost it. A PDF can only describe that.
The corpus is browsable: All 663 documents, with source, tenant, licence and checksum, searchable. In print this is an appendix nobody opens.
The golden set is inspectable: Every query, every judgment, and every human/model disagreement, per row. Papers cite their evaluation set; they rarely let you read it.
Every figure carries its regenerate command: Script, commit and corpus id sit beside the figure, so a number traces to the code that produced it without leaving the page.
Definitions travel with the text: A metric name opens its formula, range and the failure modes that threaten it, in place. A printed paper pushes that to a table the reader has to hold in their head.

What would change the claim

A published baseline run over a golden set that does not contradict itself would make 'benchmark' unqualified rather than methodology-first.
Nine one-factor variants with separated confidence intervals would make it a paper with findings.
Neither has happened yet, and the positioning above will change when they do rather than before.

3 · Problem

Retrieval systems are usually reported as a single quality number, and that number hides almost everything that determines whether the system works. It hides whether the retriever surfaced the passage, whether the generator used it, whether the system declined when there was nothing to say, and what access control did to any of it.

The project's objective is narrower and harder than a leaderboard score: measure a full pipeline end to end, publish the confidence intervals, publish the failure modes, and make every figure traceable to committed data so a stranger can re-run it.

4 · How it works

Documents are fetched from three public sources, parsed with page and section provenance preserved, chunked two ways, assigned to tenants and deduplicated. Chunks are embedded and indexed. A query runs through lexical and dense retrieval in parallel, the two ranked lists are fused by rank, a cross-encoder reorders the shortlist, and a language model answers under a schema constraint using only the retrieved passages. Citations are resolved, quotes are verified, and the system abstains when the evidence does not support an answer.

Point at a stage for what it does. Solid = implemented; dashed = planned, with the phase that delivers it.

Figure 1. The pipeline under measurement. Solid stages are implemented; dashed stages are planned, marked with the phase that delivers them.

5 · System architecture

The platform runs locally in Docker. Only the site is publicly hosted, and it is a static export that renders committed JSON and never queries the platform. That split is what keeps hosting free and the public attack surface at zero.

Layer, responsibility and technology
Layer	Responsibility	Built with
Acquisition	Fetch three public sources, rate limited and resumable	Python standard library only — urllib, no third-party HTTP client
Parsing	Extract text with page and section provenance	lxml for filing HTML; hand-written parsers for plain text
Chunking	Two strategies behind one interface	transformers tokenizer (bge-small vocabulary)
Embedding	384-dimension vectors for every chunk	BAAI/bge-small-en-v1.5 on PyTorch, CUDA
Lexical index	BM25 over the chunk corpus	Hand-rolled, array-backed postings, standard library
Vector index	Approximate nearest-neighbour search	PostgreSQL 16 with pgvector 0.8.6, HNSW
Access control	Tenant isolation inside the query	PostgreSQL row-level security, non-superuser role
Reranking	Reorder the shortlist	BAAI/bge-reranker-base cross-encoder
Generation	Schema-constrained answers	vLLM 0.27.1, xgrammar constrained decoding
Golden set	Query drafting and relevance judging	Gemini, pinned to dated snapshots
Site	Static export, no runtime	Next.js App Router, TypeScript, Tailwind CSS v4
Figures	Diagrams and charts	Hand-written inline SVG — no chart library

6 · The corpus

662 documents, 24,155 pages and 51,310 chunks across 47 tenants, from annual filings, the contract set and s. Corpus id edgar-cuad-rfc-v1.

Raw documents are not committed — the manifest is. It records source, identifier, URL, checksum, page count, licence and tenant per document, which is what lets someone reproduce the corpus without redistributing gigabytes.

7 · Key technical decisions

Metrics implemented explicitly, without an evaluation framework. The scoring logic is the credibility of the project, so it is written to be read.
Access control enforced in the database, not in application code. A forgotten filter in one query path is a data leak; a row-level security policy applies to every path.
Access-control effects reported separately from headline numbers. Approximate search under a policy loses recall, and folding that into a headline figure would misattribute it to retrieval quality.
Interactive demos replay committed measurements. The site has no backend, so every control moves over data captured by running the real pipeline offline. Nothing is simulated.
No fabricated data anywhere. Empty states say what has not happened yet. An agreement rate is withheld rather than published when the sample cannot support it.

8 · Statement of origin

Zeroth is an open reconstruction of a production confidential-document platform. The original was built for an employer over a private corpus and is not public. This is a from-scratch rebuild of the same architecture over public documents. Every number published here was measured on the public corpus described in the methodology, and applies only to it.

9 · Glossary

Every abbreviation the paper uses, written out. In the prose each one also carries its full form on hover or focus; this is the list for reading straight through.

Family tags

ABS: Abstention
CST: Cost
GRD: Grounding
PRF: Performance
RET: Retrieval

Metric names

Cost per query: Cost per query, in US dollarsCost
MRR@10: Mean Reciprocal Rank at rank 10Retrieval
NDCG@10: Normalised Discounted Cumulative Gain at rank 10Retrieval
p95 latency: 95th-percentile end-to-end latency, in secondsPerformance
Recall@10: Recall at rank 10Retrieval
Recall@5: Recall at rank 5Retrieval

Terms

ANN: Approximate Nearest NeighbourTrades exactness for speed: the index may miss a true neighbour, which is why post-filtering under access control can silently cost recall.
BAAI: Beijing Academy of Artificial IntelligencePublisher of the bge embedding and reranker models used here.
BCP: Best Current PracticeAn RFC sub-series carrying operational practice rather than a wire protocol.
CC BY: Creative Commons AttributionA licence permitting reuse with attribution.
CI: Confidence intervalHere always a bootstrapped 95% interval over 1,000 resamples of the query set.
CUAD: Contract Understanding Atticus DatasetA set of commercial contracts with expert clause annotations, released by The Atticus Project.
CUDA: Compute Unified Device ArchitectureNVIDIA's GPU computing platform.
DCG: Discounted Cumulative GainSums graded relevance down the ranking, discounting each position logarithmically.
EDGAR: Electronic Data Gathering, Analysis, and RetrievalThe SEC's public filing system, and the source of the 10-K filings in the corpus.
HNSW: Hierarchical Navigable Small WorldThe graph index used for approximate nearest-neighbour search over the embedding vectors.
IDCG: Ideal Discounted Cumulative GainThe DCG of the best possible ordering, used as the normaliser.
IETF: Internet Engineering Task Force
RAG: Retrieval-Augmented GenerationGeneration conditioned on documents fetched at query time rather than on model weights alone.
RFC: Request for CommentsThe document series in which internet standards are published.
SEC: US Securities and Exchange Commission
TLS: Transport Layer Security

10 · References

The works this project is built on and measured against. Each entry names the part of the paper that relies on it. Where a claim here rests on someone else's result rather than on a measurement of this system, the citation is the difference between an argument and an assertion.

Retrieval and generation

Cormack, G. V., Clarke, C. L. A., Büttcher, S. (2009) Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR '09, 758–759.
Fusion — Why the two ranked lists are combined by rank position and never by raw score.
Robertson, S., Zaragoza, H. (2009) The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 333–389.
Lexical retrieval — The ranking function behind the lexical half of the hybrid retriever.
Nogueira, R., Cho, K. (2019) Passage Re-ranking with BERT. arXiv preprint.
Cross-encoder reranking — The cross-encoder reranking stage: accurate over a shortlist, far too slow over a corpus.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., Kiela, D. (2020) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS).
The architecture under measurement — The paper that named the pattern this project rebuilds and measures.

Vector search

Malkov, Y. A., Yashunin, D. A. (2020) Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(4), 824–836.
The vector index, and what post-filtering does to it — The index whose approximate nature makes access-control post-filtering cost recall.
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., Nie, J. (2023) C-Pack: Packed Resources For General Chinese Embeddings. arXiv preprint.
Embedding and reranking models — Introduces the BGE family, from which bge-small-en-v1.5 and bge-reranker-base are used here.

Evaluation

Efron, B. (1979) Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7(1), 1–26.
Confidence intervals — The resampling method behind every 95% interval published here.
Voorhees, E. M. (1999) The TREC-8 Question Answering Track Report. Proceedings of the 8th Text REtrieval Conference (TREC-8).
MRR@10 — Where mean reciprocal rank entered retrieval evaluation as a standard measure.
Järvelin, K., Kekäläinen, J. (2002) Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems 20(4), 422–446.
NDCG@10 — The definition NDCG is computed from, including the discount and the ideal-ranking normaliser.

Corpus

Bradner, S., Contreras, J. (2008) BCP 78 — Rights Contributors Provide to the IETF Trust (RFC 5378). IETF.
Corpus sources and attribution — The terms under which the RFCs in the corpus are redistributable.
Hendrycks, D., Burns, C., Chen, A., Ball, S. (2021) CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. NeurIPS 2021 Datasets and Benchmarks Track.
Corpus sources — Licensed CC BY 4.0. Chunking and re-indexing constitute modification and are indicated as such.
US Securities and Exchange Commission EDGAR — Electronic Data Gathering, Analysis, and Retrieval. Public filing system.
Corpus sources — Source of the 10-K filings. Requests carry an identifying User-Agent, as EDGAR requires.

Serving and infrastructure

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., Stoica, I. (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP '23.
Generation — vLLM, which serves the generator.
Dong, Y., Ruan, C. F., Cai, Y., Lai, R., Xu, Z., Zhao, Y., Chen, T. (2024) XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv preprint.
Constrained decoding — The grammar engine that makes the answer schema a decoding constraint rather than a request.
Kane, A. pgvector — open-source vector similarity search for Postgres. Software.
Vector index — Version 0.8.6, with an HNSW index, inside PostgreSQL 16.
PostgreSQL Global Development Group Row Security Policies. PostgreSQL 16 documentation.
Access control — Row-level security, and the documented fact that superusers and BYPASSRLS roles are exempt from it.

11 · Citation

Cite this

Sharma, Anant (2026). Zeroth: a reproducible benchmark of end-to-end RAG pipeline quality. NthLabs. Corpus edgar-cuad-rfc-v1. Retrieved 2026-09-20, from https://zeroth-nthlabs.netlify.app

@misc{sharma_zeroth_2026,
  author       = {Sharma, Anant},
  title        = {Zeroth: a reproducible benchmark of end-to-end RAG pipeline quality},
  year         = {2026},
  howpublished = {\url{https://zeroth-nthlabs.netlify.app}},
  note         = {Corpus edgar-cuad-rfc-v1. Accessed 2026-09-20},
  organization = {NthLabs}
}

The corpus id is part of the citation on purpose: every number here is a property of that corpus and does not transfer to another one.

Provisional address. The project is hosted on a platform subdomain. A permanent domain has not been registered, so this address may change; cite the corpus id and access date alongside it.

12 · Author and links

Built by Anant Sharma. AI Engineer who builds production Python systems that think in steps — agentic workflows, retrieval pipelines, and orchestration middleware where every output is gated by automated evals before it ships. A year and more turning generative AI research into shipped infrastructure. The project carries the platform, the harness, the corpus manifest and this site.