Section 5
Walkthroughs
The system's behaviour, not a description of it. Every control below moves over measurements taken on the real corpus.
The retrieval platform is Phase 2 and does not exist yet as a service. What does exist is the corpus, the embedder, a lexical index and a cross-encoder, so these captures were produced by running those over the committed corpus directly.
5.1 What access control does to approximate search
An approximate index returns its nearest neighbours by distance, and only then does the access policy discard the ones this role may not see. Nothing refills the discarded slots. Narrow the role and watch the result shrink.
recall@10
0.300
exact search under the same policy = 1.000
queries returning nothing
6 of 12
permitted matches exist; none were reached
unrestricted, same ef_search
0.850
the ceiling this role is measured against
of the top 10 exact search would return
filled = also returned by approximate search · empty = lost to post-filtering, and nothing refilled it
Data: harness/interactive/rls_demo.py · regenerate with python3 harness/interactive/rls_demo.py · commit 341ea20 · 2026-08-25 · measured as zeroth_app (NOSUPERUSER, NOBYPASSRLS)
| role | ef 40 | ef 100 | ef 200 | ef 400 | ef 800 |
|---|---|---|---|---|---|
| all_tenants 47/47 | 0.850 [0] | 0.942 [0] | 0.967 [0] | 0.967 [0] | 0.975 [0] |
| analyst_broad 35/47 | 0.842 [0] | 0.933 [0] | 0.950 [0] | 0.950 [0] | 0.975 [0] |
| analyst_mid 12/47 | 0.667 [2] | 0.842 [0] | 0.892 [0] | 0.958 [0] | 0.958 [0] |
| analyst_narrow 3/47 | 0.500 [5] | 0.633 [4] | 0.667 [4] | 0.667 [4] | 0.667 [4] |
| single_tenant 1/47 | 0.300 [6] | 0.608 [4] | 0.650 [4] | 0.658 [4] | 0.667 [4] |
recall@10 against exact search under the identical policy · [n] = queries returning nothing
What this shows
- Post-filtering costs recall in proportion to how restrictive the role is: at the default ef_search=40, recall falls from 0.850 with all 47 tenants visible to 0.300 with one.
- Restricted roles return nothing at all for 6 of 12 queries even though exact search under the identical policy returns a full result set.
- Widening the search plateaus below the unrestricted ceiling. A single-tenant role tops out at 0.667 where an unrestricted one reaches 0.975, and 4 queries still return nothing at ef_search=800.
What it does not show
- An earlier measurement on a SYNTHETIC corpus found that raising ef_search from 40 to 800 changed nothing at all. That does not replicate here: recall for a single-tenant role moves from 0.300 to 0.667 and empty results fall from 6 to 4.
The synthetic corpus used generated tenant clusters that were almost perfectly separated (inter-tenant cosine 0.014), so a restricted role's nearest neighbours were entirely other tenants at any search width. Real documents share vocabulary, boilerplate and structure, so tenant regions overlap and a wider search does reach permitted rows. The separated case was the worst case, not the typical one.
Widening the search buys recall back only up to a plateau, and costs latency to do it. Partitioning removes the problem rather than mitigating it: the index contains only permitted rows, so there is nothing to post-filter away.
HNSW graph construction is not deterministic. Re-running this script moves these figures by a few points; the shape of the result is stable, the third decimal is not.
5.2 How a query becomes a ranked list
Four stages, each with its real output. The stage most people get wrong is fusion: reciprocal rank fusion combines the two lists by rank position, never by raw score, because BM25 scores and cosine similarities are not on comparable scales.
What was the total operating income for the North America segment in fiscal 2023, and which XBRL financial taxonomy members are associated with cash flow hedging using foreign exchange contracts during the same fiscal period?
Term overlap. Finds exact wording, misses paraphrase.
Data: harness/interactive/retrieval.py · regenerate with python3 harness/interactive/retrieval.py · commit 341ea20 · 2026-08-25
5.3 Where a document gets cut
Two strategies over the same document. What section-aware chunking actually guarantees is not that a chunk never starts mid-sentence — it splits long sections internally with the same overlap, so it often does — but that a chunk never spans two sections.
chunks
277
285 under section-aware
chunks spanning two sections
17
0 under section-aware
document
106 pp
22 sections · 465,597 chars
edgar-0000732717-000073271724000009
boundaries over the first 6,000 characters
Each line is a chunk start; red spans a section boundary. Point at one for the chunk it opens.
First three chunks, verbatim
#0 · p1 · 512 tokens · "front-matter"
t-20231231 false 2023 FY 0000732717 P3Y P7Y 18 24 http://fasb.org/us-gaap/2023#AccountsPayableAndAccruedLiabilitiesCurrent http://fasb.org/us-gaap/2023#AccountsPayableAndAccruedLia…
#1 · p1 · 512 tokens · "front-matter" · starts mid-sentence
-gaap/2023#FairValueMeasuredAtNetAssetValuePerShareMember http://fasb.org/us-gaap/2023#FairValueMeasuredAtNetAssetValuePerShareMember http://fasb.org/us-gaap/2023#FairValueMeasured…
#2 · p1 · 512 tokens · "front-matter" · starts mid-sentence
PayableAndAccruedLiabilitiesCurrent http://fasb.org/us-gaap/2023#AccountsPayableAndAccruedLiabilitiesCurrent 0000732717 2023-01-01 2023-12-31 0000732717 us-gaap:CommonStockMember 2…
Data: harness/interactive/chunking.py · regenerate with python3 harness/interactive/chunking.py · commit 341ea20 · 2026-08-25