Skip to content
Contents

Section 3

Corpus

Public documents, chosen to be long, messy, cross-referencing, and naturally partitioned into tenants.

The manifest is what makes the corpus reproducible without redistributing gigabytes: replay the URL, verify the checksum. Raw documents are cached locally and not committed.

Corpus edgar-cuad-rfc-v1: 662 documents, 24155 pages, 51310 chunks across 47 tenants.documents by sourcecuad509edgar123rfc30documents662pages24,155chunks (fixed-512)51,310tenants4715,959 pages counted from real page breaks; 8,196 estimated where the source carries no page structure.corpus edgar-cuad-rfc-v1 · read from the committed manifest
Figure 7. Composition by source, with page provenance separated into counted and estimated.

3.1 Sources

SEC EDGAR 10-K filings

Long, structurally messy, heavy with cross-references and tables. They partition naturally by filing company, which is what makes tenant isolation meaningful to test rather than theatrical. Only the primary document of each filing is fetched.

CUAD — Contract Understanding Atticus Dataset

Commercial contracts annotated with clause spans, CC BY 4.0, which supplies ground truth for a subset of queries. These contracts are themselves drawn from EDGAR, so they are a different document shape from the same publisher rather than an independent source, and they are deduplicated against the filing set by containment.

RFCs

Documents from the HTTP and families. Freely redistributable, densely cross-referencing, and a clean hard-mode subset — the only genuinely independent third source.

3.2 Attribution

: Hendrycks et al., NeurIPS 2021; The Atticus Project. Licensed 4.0. Chunking and re-indexing constitute modification and are indicated as such. filings are published by the US Securities and Exchange Commission. s are published by the and RFC Editor under 78.

Corpus edgar-cuad-rfc-v1

ABS Abstention

Abstention (correct)

What it measures

Proportion of genuinely unanswerable queries the system correctly declines.

Formula

Abst = |{ q in U : system declined }| / |U|
U
the unanswerable subset of the query set
declined
the abstention gate fired rather than an answer being produced

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · abstention_correct()

Arrives in Phase 4.

What can go wrong

Related — Abstention family

faithfulness · recall_at_10

GRD Grounding

Answer correctness

What it measures

Agreement with the reference answer, judged against a published rubric.

Formula

correctness = judge(answer, reference) in {0, 1}
reference
the golden set answer, model-drafted and partially human-verified

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · answer_correctness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_relevance · faithfulness

GRD Grounding

Answer relevance

What it measures

Whether the answer addresses the question that was actually asked.

Formula

relevance = judge(answer, question) in {0, 1}
judge
LLM against a published rubric; the model reference is pinned per run

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · answer_relevance()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_correctness

GRD Grounding

Citation accuracy

What it measures

Proportion of citations that resolve to a chunk actually supporting the cited claim.

Formula

A = |{ citations resolving to a supporting chunk }| / |citations|
resolve
the cited chunk id exists in the index for this run
supporting
string containment first; LLM judge only where containment fails

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

platform/generation/verify.py · citation_accuracy()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_coverage · faithfulness

GRD Grounding

Citation coverage

What it measures

Proportion of factual claims that carry a citation at all.

Formula

C = |{ claims with >= 1 citation }| / |factual claims|
factual claim
an assertion that could be checked against a source

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

platform/generation/verify.py · citation_coverage()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_accuracy · faithfulness

RET Retrieval

Context precision

What it measures

Proportion of retrieved chunks that are actually relevant.

Formula

P(q) = |{ c in top_k(q) : grade(c) >= 2 }| / k
k
retrieval depth, recorded per run in the config

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · context_precision()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · faithfulness

CST Cost

Cost per query

What it measures

Token counts multiplied by the rates in the run config. Local models cost zero, stated openly.

Formula

cost = (in_tokens * rate_in + out_tokens * rate_out) / 1e6
rate
from configs/pricing.yaml, recorded per run
local models
0.00 by definition; hardware cost is not amortised into this number

US dollars per query. Zero for fully local runs, which is stated rather than hidden.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

harness/eval/cost.py · cost_per_query()

Arrives in Phase 4.

What can go wrong

Related — Cost family

latency_p95_s

GRD Grounding

Faithfulness

What it measures

Proportion of generated claims entailed by the retrieved chunks.

Formula

F(a) = |{ claims in a entailed by context }| / |claims in a|
claims
atomic assertions extracted from the answer
entailed
judged by an LLM against a published rubric, not string match

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · faithfulness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

citation_accuracy · citation_coverage · answer_correctness

PRF Performance

p95 latency

What it measures

95th percentile end-to-end time, retrieval through verification.

Formula

p95 = quantile(latencies, 0.95) over 3 repeats per query
end-to-end
retrieval, rerank, generation, citation resolution, verification
repeats
three per query; the distribution is over all of them

Wall clock seconds. Lower is better. Machine-dependent, so it is only comparable within a run set.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

harness/eval/runner.py · latency_percentiles()

Arrives in Phase 4.

What can go wrong

Related — Performance family

cost_per_query

RET Retrieval

MRR@10

What it measures

Mean reciprocal rank of the first relevant chunk. Rewards getting one right answer high.

Formula

MRR@k = (1/|Q|) * sum_q 1 / rank_first_relevant(q)
rank_first_relevant(q)
1-indexed rank of the first chunk graded >= 2, or 0 contribution if none

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · mrr_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10

RET Retrieval

NDCG@10

What it measures

Discounted cumulative gain over graded relevance, normalised against the ideal ranking.

Formula

DCG@k = sum_i (2^grade(c_i) - 1) / log2(i + 1)
NDCG@k = DCG@k / IDCG@k
c_i
the chunk at rank i
IDCG@k
DCG of the best possible ordering
grade
0-3, so a 3 counts 7x a 1

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · ndcg_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · mrr_at_10

RET Retrieval

Recall@10

What it measures

Proportion of queries where at least one chunk graded 2 or higher appears in the top 10.

Formula

R@k = |{ q in Q : max grade(c) >= 2 for c in top_k(q) }| / |Q|
grade(c)
graded relevance 0-3 from the golden set
top_k(q)
the k chunks the retriever ranked highest for q
Q
the answerable query set

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_5 · ndcg_at_10 · mrr_at_10 · context_precision

RET Retrieval

Recall@5

What it measures

The same measure at rank 5. Harder, and more sensitive to reranking.

Formula

R@5 = |{ q in Q : max grade(c) >= 2 for c in top_5(q) }| / |Q|
top_5(q)
the five chunks ranked highest for q

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

harness/eval/scorers.py · recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10