Section 8
About
Zeroth is an open reconstruction of a production confidential-document retrieval platform, rebuilt over public documents so the architecture can be inspected, measured and argued with.
1 · Project overview
The original system was built for an employer over a private corpus and cannot leave. This is a from-scratch rebuild of the same architecture over public documents, together with a public evaluation board that measures it. Every number published here was measured on the corpus described in the methodology and applies only to it.
That framing is technical rather than merely ethical. Retrieval metrics are properties of a corpus-and-query-set pair, not of an architecture, so numbers measured here cannot validate or stand in for numbers measured on a different corpus.
2 · Problem
Retrieval systems are usually reported as a single quality number, and that number hides almost everything that determines whether the system works. It hides whether the retriever surfaced the passage, whether the generator used it, whether the system declined when there was nothing to say, and what access control did to any of it.
The project's objective is narrower and harder than a leaderboard score: measure a full pipeline end to end, publish the confidence intervals, publish the failure modes, and make every figure traceable to committed data so a stranger can re-run it.
3 · How it works
Documents are fetched from three public sources, parsed with page and section provenance preserved, chunked two ways, assigned to tenants and deduplicated. Chunks are embedded and indexed. A query runs through lexical and dense retrieval in parallel, the two ranked lists are fused by rank, a cross-encoder reorders the shortlist, and a language model answers under a schema constraint using only the retrieved passages. Citations are resolved, quotes are verified, and the system abstains when the evidence does not support an answer.
Point at a stage for what it does. Solid = implemented; dashed = planned, with the phase that delivers it.
4 · System architecture
The platform runs locally in Docker. Only the site is publicly hosted, and it is a static export that renders committed JSON and never queries the platform. That split is what keeps hosting free and the public attack surface at zero.
| Layer | Responsibility | Built with |
|---|---|---|
| Acquisition | Fetch three public sources, rate limited and resumable | Python standard library only — urllib, no third-party HTTP client |
| Parsing | Extract text with page and section provenance | lxml for filing HTML; hand-written parsers for plain text |
| Chunking | Two strategies behind one interface | transformers tokenizer (bge-small vocabulary) |
| Embedding | 384-dimension vectors for every chunk | BAAI/bge-small-en-v1.5 on PyTorch, CUDA |
| Lexical index | BM25 over the chunk corpus | Hand-rolled, array-backed postings, standard library |
| Vector index | Approximate nearest-neighbour search | PostgreSQL 16 with pgvector 0.8.6, HNSW |
| Access control | Tenant isolation inside the query | PostgreSQL row-level security, non-superuser role |
| Reranking | Reorder the shortlist | BAAI/bge-reranker-base cross-encoder |
| Generation | Schema-constrained answers | vLLM 0.27.1, xgrammar constrained decoding |
| Golden set | Query drafting and relevance judging | Gemini, pinned to dated snapshots |
| Site | Static export, no runtime | Next.js App Router, TypeScript, Tailwind CSS v4 |
| Figures | Diagrams and charts | Hand-written inline SVG — no chart library |
5 · The corpus
662 documents, 24,155 pages and 51,310 chunks across 47 tenants, from annual filings, the contract set and s. Corpus id edgar-cuad-rfc-v1.
Raw documents are not committed — the manifest is. It records source, identifier, URL, checksum, page count, licence and tenant per document, which is what lets someone reproduce the corpus without redistributing gigabytes.
6 · Key technical decisions
- Metrics implemented explicitly, without an evaluation framework. The scoring logic is the credibility of the project, so it is written to be read.
- Access control enforced in the database, not in application code. A forgotten filter in one query path is a data leak; a row-level security policy applies to every path.
- Access-control effects reported separately from headline numbers. Approximate search under a policy loses recall, and folding that into a headline figure would misattribute it to retrieval quality.
- Interactive demos replay committed measurements. The site has no backend, so every control moves over data captured by running the real pipeline offline. Nothing is simulated.
- No fabricated data anywhere. Empty states say what has not happened yet. An agreement rate is withheld rather than published when the sample cannot support it.
7 · Statement of origin
Zeroth is an open reconstruction of a production confidential-document platform. The original was built for an employer over a private corpus and is not public. This is a from-scratch rebuild of the same architecture over public documents. Every number published here was measured on the public corpus described in the methodology, and applies only to it.
8 · Glossary
Every abbreviation the paper uses, written out. In the prose each one also carries its full form on hover or focus; this is the list for reading straight through.
Family tags
- ABS
- Abstention
- CST
- Cost
- GRD
- Grounding
- PRF
- Performance
- RET
- Retrieval
Terms
- ANN
- Approximate Nearest NeighbourTrades exactness for speed: the index may miss a true neighbour, which is why post-filtering under access control can silently cost recall.
- BAAI
- Beijing Academy of Artificial IntelligencePublisher of the bge embedding and reranker models used here.
- BCP
- Best Current PracticeAn RFC sub-series carrying operational practice rather than a wire protocol.
- CC BY
- Creative Commons AttributionA licence permitting reuse with attribution.
- CI
- Confidence intervalHere always a bootstrapped 95% interval over 1,000 resamples of the query set.
- CUAD
- Contract Understanding Atticus DatasetA set of commercial contracts with expert clause annotations, released by The Atticus Project.
- CUDA
- Compute Unified Device ArchitectureNVIDIA's GPU computing platform.
- DCG
- Discounted Cumulative GainSums graded relevance down the ranking, discounting each position logarithmically.
- EDGAR
- Electronic Data Gathering, Analysis, and RetrievalThe SEC's public filing system, and the source of the 10-K filings in the corpus.
- HNSW
- Hierarchical Navigable Small WorldThe graph index used for approximate nearest-neighbour search over the embedding vectors.
- IDCG
- Ideal Discounted Cumulative GainThe DCG of the best possible ordering, used as the normaliser.
- IETF
- Internet Engineering Task Force
- RAG
- Retrieval-Augmented GenerationGeneration conditioned on documents fetched at query time rather than on model weights alone.
- RFC
- Request for CommentsThe document series in which internet standards are published.
- SEC
- US Securities and Exchange Commission
- TLS
- Transport Layer Security
9 · Author and links
Built by Anant Sharma. AI Engineer who builds production Python systems that think in steps — agentic workflows, retrieval pipelines, and orchestration middleware where every output is gated by automated evals before it ships. A year and more turning generative AI research into shipped infrastructure. The project carries the platform, the harness, the corpus manifest and this site.