Skip to content
Contents

Section 7 · Learn · Security

PostgreSQL Row-Level Security

Access rules enforced by the database inside the query itself, so a row a role may not see is never returned regardless of how the query was written.

Access rules enforced by the database inside the query itself, so a row a role may not see is never returned regardless of how the query was written.

What it is

A policy is a boolean expression attached to a table. PostgreSQL adds it to every query against that table automatically, as a security-barrier condition the planner may not optimise around.

Why it is needed

Filtering in application code means every query path must remember to filter. One forgotten WHERE clause is a data leak. Enforcing in the database makes the rule structural: there is no query that bypasses it.

How it works

The policy typically checks the row's tenant against a session variable set per request. It is evaluated before any user-supplied condition, so a crafted predicate cannot see rows the policy excludes.

In this project

Policies check the row's tenant against an ACL table keyed on a session setting. Every measurement connects as zeroth_app, which is NOSUPERUSER NOBYPASSRLS, and the demo script asserts rolbypassrls is false before taking a single reading.

Two silent bypasses

A superuser is exempt from row-level security entirely. Connect as postgres and every policy silently does nothing — no error, no warning, and every access-control test passes for the wrong reason. Separately, a table's OWNER is also exempt unless FORCE ROW LEVEL SECURITY is set, because relforcerowsecurity defaults to false. Both were confirmed here: running a probe as the table owner returned recall of 0.95 with no restriction applied at all.

The ACL table is itself readable

If a policy reads an ACL table, the querying role needs SELECT on it, which means any query can enumerate the entire authorisation matrix. Putting it behind a SECURITY DEFINER function or its own policy is the fix.

Trade-offs

Advantages

  • Cannot be forgotten by application code
  • Applies to every query path, including ad-hoc ones
  • Enforced by the database, not by convention

Limitations

  • Policy expressions run per row and can be costly
  • Interacts badly with approximate index scans
  • Bypassed silently by superusers and table owners

Common mistakes

  • Connecting as postgres and concluding the tests pass
  • Forgetting FORCE ROW LEVEL SECURITY
  • Assuming correct enforcement means unchanged recall

When to use it

Reach for it when

  • Multi-tenant data where isolation must be structural

Avoid it when

  • Single-tenant systems where the complexity buys nothing

How it interacts with the rest of the system

ANN post-filtering under access controlPartitioning by tenantMulti-tenancy and tenant assignment

← All topics

ABS Abstention

Abstention (correct)

What it measures

Proportion of genuinely unanswerable queries the system correctly declines.

Formula

Abst = |{ q in U : system declined }| / |U|
U
the unanswerable subset of the query set
declined
the abstention gate fired rather than an answer being produced

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

abstention_correct()

Arrives in Phase 4.

What can go wrong

Related — Abstention family

faithfulness · recall_at_10

GRD Grounding

Answer correctness

What it measures

Agreement with the reference answer, judged against a published rubric.

Formula

correctness = judge(answer, reference) in {0, 1}
reference
the golden set answer, model-drafted and partially human-verified

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

answer_correctness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_relevance · faithfulness

GRD Grounding

Answer relevance

What it measures

Whether the answer addresses the question that was actually asked.

Formula

relevance = judge(answer, question) in {0, 1}
judge
LLM against a published rubric; the model reference is pinned per run

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

answer_relevance()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

answer_correctness

GRD Grounding

Citation accuracy

What it measures

Proportion of citations that resolve to a chunk actually supporting the cited claim.

Formula

A = |{ citations resolving to a supporting chunk }| / |citations|
resolve
the cited chunk id exists in the index for this run
supporting
string containment first; LLM judge only where containment fails

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

citation_accuracy()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_coverage · faithfulness

GRD Grounding

Citation coverage

What it measures

Proportion of factual claims that carry a citation at all.

Formula

C = |{ claims with >= 1 citation }| / |factual claims|
factual claim
an assertion that could be checked against a source

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

citation_coverage()

Arrives in Phase 2.

What can go wrong

Related — Grounding family

citation_accuracy · faithfulness

RET Retrieval

Context precision

What it measures

Proportion of retrieved chunks that are actually relevant.

Formula

P(q) = |{ c in top_k(q) : grade(c) >= 2 }| / k
k
retrieval depth, recorded per run in the config

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

context_precision()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · faithfulness

CST Cost

Cost per query

What it measures

Token counts multiplied by the rates in the run config. Local models cost zero, stated openly.

Formula

cost = (in_tokens * rate_in + out_tokens * rate_out) / 1e6
rate
from configs/pricing.yaml, recorded per run
local models
0.00 by definition; hardware cost is not amortised into this number

US dollars per query. Zero for fully local runs, which is stated rather than hidden.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

cost_per_query()

Arrives in Phase 4.

What can go wrong

Related — Cost family

latency_p95_s

GRD Grounding

Faithfulness

What it measures

Proportion of generated claims entailed by the retrieved chunks.

Formula

F(a) = |{ claims in a entailed by context }| / |claims in a|
claims
atomic assertions extracted from the answer
entailed
judged by an LLM against a published rubric, not string match

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

faithfulness()

Arrives in Phase 4.

What can go wrong

Related — Grounding family

citation_accuracy · citation_coverage · answer_correctness

PRF Performance

p95 latency

What it measures

95th percentile end-to-end time, retrieval through verification.

Formula

p95 = quantile(latencies, 0.95) over 3 repeats per query
end-to-end
retrieval, rerank, generation, citation resolution, verification
repeats
three per query; the distribution is over all of them

Wall clock seconds. Lower is better. Machine-dependent, so it is only comparable within a run set.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

Reported as a point estimate. This is a measured quantity of the run rather than a sample statistic over queries.

How this project computes it

latency_percentiles()

Arrives in Phase 4.

What can go wrong

Related — Performance family

cost_per_query_usd

RET Retrieval

MRR@10

What it measures

Mean reciprocal rank of the first relevant chunk. Rewards getting one right answer high.

Formula

MRR@k = (1/|Q|) * sum_q 1 / rank_first_relevant(q)
rank_first_relevant(q)
1-indexed rank of the first chunk graded >= 2, or 0 contribution if none

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

mrr_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10

RET Retrieval

NDCG@10

What it measures

Discounted cumulative gain over graded relevance, normalised against the ideal ranking.

Formula

DCG@k = sum_i (2^grade(c_i) - 1) / log2(i + 1)
NDCG@k = DCG@k / IDCG@k
c_i
the chunk at rank i
IDCG@k
DCG of the best possible ordering
grade
0-3, so a 3 counts 7x a 1

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

ndcg_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · mrr_at_10

RET Retrieval

Recall@10

What it measures

Proportion of queries where at least one chunk graded 2 or higher appears in the top 10.

Formula

R@k = |{ q in Q : max grade(c) >= 2 for c in top_k(q) }| / |Q|
grade(c)
graded relevance 0-3 from the golden set
top_k(q)
the k chunks the retriever ranked highest for q
Q
the answerable query set

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_5 · ndcg_at_10 · mrr_at_10 · context_precision

RET Retrieval

Recall@5

What it measures

The same measure at rank 5. Harder, and more sensitive to reranking.

Formula

R@5 = |{ q in Q : max grade(c) >= 2 for c in top_5(q) }| / |Q|
top_5(q)
the five chunks ranked highest for q

Mathematical range. No typical value is stated: it would be a number with no run behind it.

Worked example

From the golden set, cross-document, query cross-document-000 — 8 human-verified judgments.

What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?

  • grade 3edgar-0000829224-000082922424000057::fixed-512::00015
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00108
  • grade 3edgar-0000320187-000032018725000047::fixed-512::00183
  • grade 3edgar-0000320187-000032018724000044::fixed-512::00185
  • grade 3edgar-0000796343-000079634324000006::fixed-512::00150
  • grade 3edgar-0000796343-000079634325000004::fixed-512::00152

Confidence interval

95% interval by bootstrap resampling over 1,000 resamples of the query set. A point estimate over a few hundred queries without an interval is the first thing a reviewer attacks.

How this project computes it

recall_at_k()

Arrives in Phase 4.

What can go wrong

Related — Retrieval family

recall_at_10 · ndcg_at_10