Skip to content
Contents

Section 4

Corpus

Public documents, chosen to be long, messy, cross-referencing, and naturally partitioned into tenants.

The manifest is what makes the corpus reproducible without redistributing gigabytes: replay the URL, verify the checksum. Raw documents are cached locally and not committed.

Corpus edgar-cuad-rfc-v1: 662 documents, 24155 pages, 51310 chunks across 47 tenants.documents by sourcecuad509edgar123rfc30documents662pages24,155chunks (fixed-512)51,310tenants4715,959 pages counted from real page breaks; 8,196 estimated where the source carries no page structure.corpus edgar-cuad-rfc-v1 · read from the committed manifest
Figure 7. Composition by source, with page provenance separated into counted and estimated.

4.1 Sources

SEC EDGAR 10-K filings

Long, structurally messy, heavy with cross-references and tables. They partition naturally by filing company, which is what makes tenant isolation meaningful to test rather than theatrical. Only the primary document of each filing is fetched.

CUAD — Contract Understanding Atticus Dataset

Commercial contracts annotated with clause spans, CC BY 4.0, which supplies ground truth for a subset of queries. These contracts are themselves drawn from EDGAR, so they are a different document shape from the same publisher rather than an independent source, and they are deduplicated against the filing set by containment.

RFCs

Documents from the HTTP and families. Freely redistributable, densely cross-referencing, and a clean hard-mode subset — the only genuinely independent third source.

4.2 Attribution

: Hendrycks et al., NeurIPS 2021; The Atticus Project. Licensed 4.0. Chunking and re-indexing constitute modification and are indicated as such. filings are published by the US Securities and Exchange Commission. s are published by the and RFC Editor under 78.

Corpus edgar-cuad-rfc-v1

4.3 Every document

The manifest, not a summary of it. Each row is one document as it was acquired: its identifier at the source, the tenant it was assigned to, its page count, its licence, and the checksum that pins the bytes. Raw documents are not redistributed — this is what makes the corpus reproducible without them.

Source

Showing the first 25 of 663 documents; loading the rest…

Corpus documents with source, tenant, pages, licence and checksum
documentsourcetenantpageslicencesha256
cuad-2themartcominc-19990826-10-12g-ex-10-10--03981d032ThemartComInc_19990826_10-12G_EX-10.10_6700288_EX-10.10_Co-Branding Agreement_ Agency Agreement.txtcuadcuad-distribution10CC-BY-4.06cdfab95eb5b
cuad-abilityinc-06-15-2020-ex-4-25-services-a-b8665596ABILITYINC_06_15_2020-EX-4.25-SERVICES AGREEMENT.txtcuadcuad-services6CC-BY-4.0e335c24cfd77
cuad-acceleratedtechnologiesholdingcorp-04-24-a3a8c094ACCELERATEDTECHNOLOGIESHOLDINGCORP_04_24_2003-EX-10.13-JOINT VENTURE AGREEMENT.txtcuadcuad-alliance4CC-BY-4.085182a8d5506
cuad-accurayinc-09-01-2010-ex-10-31-distribut-37e9f4a4ACCURAYINC_09_01_2010-EX-10.31-DISTRIBUTOR AGREEMENT.txtcuadcuad-distribution24CC-BY-4.014932c769a7a
cuad-adamsgolfinc-03-21-2005-ex-10-17-endorse-3fb30638ADAMSGOLFINC_03_21_2005-EX-10.17-ENDORSEMENT AGREEMENT.txtcuadcuad-marketing8CC-BY-4.0ca9acda7d271
cuad-adaptimmunetherapeuticsplc-04-06-2017-ex-452133ffADAPTIMMUNETHERAPEUTICSPLC_04_06_2017-EX-10.11-STRATEGIC ALLIANCE AGREEMENT.txtcuadcuad-alliance26CC-BY-4.0f20f55c32b8c
cuad-adianutrition-inc-04-01-2005-ex-10-d2-re-c4680e34ADIANUTRITION,INC_04_01_2005-EX-10.D2-RESELLER AGREEMENT.txtcuadcuad-distribution8CC-BY-4.0c757bb1d6007
cuad-adma-biomanufacturing-llc-amendment-3-to-984353e4ADMA BioManufacturing, LLC - Amendment #3 to Manufacturing Agreement .txtcuadcuad-supply5CC-BY-4.0bce002bc4b17
cuad-adurobiotech-inc-06-02-2020-ex-10-7-cons-37c8f729ADUROBIOTECH,INC_06_02_2020-EX-10.7-CONSULTING AGREEMENT.txtcuadcuad-services4CC-BY-4.023255988c8f3
cuad-adurobiotech-inc-06-02-2020-ex-10-7-cons-8c41c577ADUROBIOTECH,INC_06_02_2020-EX-10.7-CONSULTING AGREEMENT(1).txtcuadcuad-services4CC-BY-4.023255988c8f3
cuad-afsalabancorpinc-08-01-1996-ex-1-1-agenc-2bd356c8AFSALABANCORPINC_08_01_1996-EX-1.1-AGENCY AGREEMENT.txtcuadcuad-distribution34CC-BY-4.0ba14c927c1ce
cuad-agapeatpcorp-20191202-10-ka-ex-10-1-1191-18a45d4aAgapeAtpCorp_20191202_10-KA_EX-10.1_11911128_EX-10.1_Supply Agreement.txtcuadcuad-supply6CC-BY-4.06d43065ff242
cuad-aimmunetherapeuticsinc-20200205-8-k-ex-1-cef2400bAimmuneTherapeuticsInc_20200205_8-K_EX-10.3_11967170_EX-10.3_Development Agreement.txtcuadcuad-alliance50CC-BY-4.03060a01d1ea3
cuad-airspannetworksinc-04-11-2000-ex-10-5-di-ac5f28eaAIRSPANNETWORKSINC_04_11_2000-EX-10.5-Distributor Agreement.txtcuadcuad-distribution22CC-BY-4.0ef5d06f1cc29
cuad-airtechinternationalgroupinc-05-08-2000--b21d32e3AIRTECHINTERNATIONALGROUPINC_05_08_2000-EX-10.4-FRANCHISE AGREEMENT.txtcuadcuad-other23CC-BY-4.0c9a4d1006bce
cuad-alamogordofinancialcorp-12-16-1999-ex-1--5f78f2f8ALAMOGORDOFINANCIALCORP_12_16_1999-EX-1-AGENCY AGREEMENT.txtcuadcuad-distribution40CC-BY-4.03c70a981a210
cuad-alcostoresinc-12-14-2005-ex-10-26-agency-2cd7fb46ALCOSTORESINC_12_14_2005-EX-10.26-AGENCY AGREEMENT.txtcuadcuad-distribution4CC-BY-4.00d8be99620d3
cuad-alliancebancorpincofpennsylvania-10-18-2-1c21a095ALLIANCEBANCORPINCOFPENNSYLVANIA_10_18_2006-EX-1.2-AGENCY AGREEMENT.txtcuadcuad-distribution37CC-BY-4.0d22c5d084ff7
cuad-alliedesportsentertainmentinc-20190815-8-5e579832AlliedEsportsEntertainmentInc_20190815_8-K_EX-10.19_11788293_EX-10.19_Content License Agreement.txtcuadcuad-license-ip7CC-BY-4.082411b6956e9
cuad-alliedesportsentertainmentinc-20190815-8-9b1ac24fAlliedEsportsEntertainmentInc_20190815_8-K_EX-10.34_11788308_EX-10.34_Sponsorship Agreement.txtcuadcuad-marketing10CC-BY-4.0e9a7e542a3a9
cuad-allisontransmissionholdingsinc-12-15-201-728b7acaALLISONTRANSMISSIONHOLDINGSINC_12_15_2014-EX-99.1-COOPERATION AGREEMENT.txtcuadcuad-other10CC-BY-4.0cf7ccac86b58
cuad-ambassadoreyeweargroupinc-11-17-1997-ex--223f2cefAMBASSADOREYEWEARGROUPINC_11_17_1997-EX-10.28-ENDORSEMENT AGREEMENT.txtcuadcuad-marketing12CC-BY-4.08793df7d80cd
cuad-americanphysicianscapitalinc-03-31-2003--151e8bb6AMERICANPHYSICIANSCAPITALINC_03_31_2003-EX-10.26-AGENCY AGREEMENT.txtcuadcuad-distribution8CC-BY-4.0ed3202c56ebd
cuad-americasshoppingmallinc-12-10-1999-ex-10-3997452cAMERICASSHOPPINGMALLINC_12_10_1999-EX-10.2-SITE DEVELOPMENT AND HOSTING AGREEMENT.txtcuadcuad-services6CC-BY-4.0800a8e3e69ce
cuad-anixabiosciencesinc-06-09-2020-ex-10-1-c-8d790441ANIXABIOSCIENCESINC_06_09_2020-EX-10.1-COLLABORATION AGREEMENT.txtcuadcuad-alliance12CC-BY-4.0b700fd095802

ABS Abstention

Abstention (correct)

What it measures

Proportion of genuinely unanswerable queries the system correctly declines.

Formula

Abst = |{ q in U : system declined }| / |U|
U
the unanswerable subset of the query set
declined
the abstention gate fired rather than an answer being produced

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

abstention_correct()

Arrives in Phase 4.

What can go wrong

Related — Abstention family

faithfulness · recall_at_10

GRD Grounding

Answer correctness

What it measures

Agreement with the reference answer, judged against a published rubric.

Formula

correctness = judge(answer, reference) in {0, 1}
reference
the golden set answer, model-drafted and partially human-verified

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

answer_correctness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_relevance · faithfulness

GRD Grounding

Answer relevance

What it measures

Whether the answer addresses the question that was actually asked.

Formula

relevance = judge(answer, question) in {0, 1}
judge
LLM against a published rubric; the model reference is pinned per run

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

answer_relevance()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_correctness

GRD Grounding

Citation accuracy

What it measures

Proportion of citations that resolve to a chunk actually supporting the cited claim.

Formula

A = |{ citations resolving to a supporting chunk }| / |citations|
resolve
the cited chunk id exists in the index for this run
supporting
string containment first; LLM judge only where containment fails

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

citation_accuracy()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_coverage · faithfulness

GRD Grounding

Citation coverage

What it measures

Proportion of factual claims that carry a citation at all.

Formula

C = |{ claims with >= 1 citation }| / |factual claims|
factual claim
an assertion that could be checked against a source

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

citation_coverage()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_accuracy · faithfulness

RET Retrieval

Context precision

What it measures

Proportion of retrieved chunks that are actually relevant.

Formula

P(q) = |{ c in top_k(q) : grade(c) >= 2 }| / k
k
retrieval depth, recorded per run in the config

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

context_precision()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · faithfulness

CST Cost

Cost per query

What it measures

Token counts multiplied by the rates in the run config. Local models cost zero, stated openly.

Formula

cost = (in_tokens * rate_in + out_tokens * rate_out) / 1e6
rate
from configs/pricing.yaml, recorded per run
local models
0.00 by definition; hardware cost is not amortised into this number

US dollars per query. Zero for fully local runs, which is stated rather than hidden.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

cost_per_query()

Arrives in Phase 4.

What can go wrong

Related — Cost family

latency_p95_s

GRD Grounding

Faithfulness

What it measures

Proportion of generated claims entailed by the retrieved chunks.

Formula

F(a) = |{ claims in a entailed by context }| / |claims in a|
claims
atomic assertions extracted from the answer
entailed
judged by an LLM against a published rubric, not string match

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

faithfulness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

citation_accuracy · citation_coverage · answer_correctness

PRF Performance

p95 latency

What it measures

95th percentile end-to-end time, retrieval through verification.

Formula

p95 = quantile(latencies, 0.95) over 3 repeats per query
end-to-end
retrieval, rerank, generation, citation resolution, verification
repeats
three per query; the distribution is over all of them

Wall clock seconds. Lower is better. Machine-dependent, so it is only comparable within a run set.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

latency_percentiles()

Arrives in Phase 4.

What can go wrong

Related — Performance family

cost_per_query_usd

RET Retrieval

MRR@10

What it measures

Mean reciprocal rank of the first relevant chunk. Rewards getting one right answer high.

Formula

MRR@k = (1/|Q|) * sum_q 1 / rank_first_relevant(q)
rank_first_relevant(q)
1-indexed rank of the first chunk graded >= 2, or 0 contribution if none

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

mrr_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10

RET Retrieval

NDCG@10

What it measures

Discounted cumulative gain over graded relevance, normalised against the ideal ranking.

Formula

DCG@k = sum_i (2^grade(c_i) - 1) / log2(i + 1)
NDCG@k = DCG@k / IDCG@k
c_i
the chunk at rank i
IDCG@k
DCG of the best possible ordering
grade
0-3, so a 3 counts 7x a 1

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

ndcg_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · mrr_at_10

RET Retrieval

Recall@10

What it measures

Proportion of queries where at least one chunk graded 2 or higher appears in the top 10.

Formula

R@k = |{ q in Q : max grade(c) >= 2 for c in top_k(q) }| / |Q|
grade(c)
graded relevance 0-3 from the golden set
top_k(q)
the k chunks the retriever ranked highest for q
Q
the answerable query set

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_5 · ndcg_at_10 · mrr_at_10 · context_precision

RET Retrieval

Recall@5

What it measures

The same measure at rank 5. Harder, and more sensitive to reranking.

Formula

R@5 = |{ q in Q : max grade(c) >= 2 for c in top_5(q) }| / |Q|
top_5(q)
the five chunks ranked highest for q

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10