The benchmark could read the answers from the back of the book
Testing 9 does not extend Testing 8. It replaces the harness underneath it. An audit found that the corpus being searched already contained every fix the benchmark asked retrieval to locate, so every number published in Testing 1–8 was measured against contaminated ground. This is the re-measurement: a temporal split, three repositories, 662 paired instances, and confidence intervals on every claim.
2026-08-25
Instances were drawn from the history of the same tree that was being searched.
Gold came from git log -400 HEAD while the corpus was the working tree at HEAD. Every instance’s fix was therefore already present in the code retrieval was asked to search — including the explanatory comments those commits added, which paraphrase the very issue text used as the query.
Two independent defects followed from it, and both were measured before anything was changed.
The audit
| Measure | Naib repo | Playwright |
|---|---|---|
| Instances whose issue vocabulary appears in lines that same commit added | 95.2% | 71.6% |
| Gold files that did not exist until the commit created them | 35.5% | 1.5% |
| Instances containing at least one such unreachable file | 63.1% | 4.0% |
A retriever cannot return a file the repository does not yet contain, so unreachable targets held maximum achievable recall well below 1.0 — by an amount that varied per instance and was never reported.
What the leak was worth
Playwright, identical arms, contaminated corpus versus a corpus pinned before the instances. Recall@10 and hit rate, train split.
| Arm | Contaminated | Leak-free | Change |
|---|---|---|---|
| Grep | 0.087 / 19% | 0.100 / 20% | +15% |
| Naib: lexical | 0.375 / 72% | 0.224 / 42% | −40% |
| Naib: lexical + symbol | 0.402 / 72% | 0.269 / 50% | −33% |
Retrieval lost a third of its recall. Grep did not move. That asymmetry is the leak isolated: grep reads file contents too, but its failure mode was never vocabulary — it ranks large files. Naib was reading the fix descriptions that its own source-body indexing had ingested.
The fix: a temporal split
Naib’s own repository had no valid epoch under the old rules: it held 40 files 450 commits ago against 395 today, so essentially every gold file postdated any epoch that preceded the instances. The greenfield rule is what makes it measurable at all — and it contributes 32 of the 177 cold-start instances below.
Corpora
| Repository | Epoch | Files at epoch | Instances | Mature / greenfield |
|---|---|---|---|---|
| Playwright | HEAD~400 | 3,216 | 151 | 130 / 21 |
| Playwright | HEAD~800 | 3,173 | 141 | 112 / 29 |
| Playwright | HEAD~1200 | 3,087 | 122 | 80 / 42 |
| Playwright | HEAD~1600 | 3,134 | 104 | — |
| Naib | HEAD~200 | 275 | 55 | 23 / 32 |
| Pydantic (Python) | HEAD~400 | 759 | 89 | 82 / 7 |
| Total | 662 | 485 / 177 |
Three repositories rather than more epochs of one: instances within a repository share files, so additional epochs extend existing clusters instead of adding independent ones. Pydantic is Python, parsed by a different backend, and is the only test of whether any of this generalizes past TypeScript.
Leak-free baseline
Playwright, epoch 07b5acd9. Every arm charged identically for the file contents an agent actually reads walking its own ranking. Cost per success is tokens read per instance where gold was located.
| Arm | Train (n=100) | Holdout (n=51) | ||||
|---|---|---|---|---|---|---|
| Recall@10 | Hit | Cost/success | Recall@10 | Hit | Cost/success | |
| Grep | 0.100 | 20% | 5,623,481 | 0.062 | 18% | 6,392,192 |
| Naib: lexical | 0.224 | 42% | 364,902 | 0.303 | 63% | 177,075 |
| Naib: + symbol | 0.269 | 50% | 388,614 | 0.299 | 63% | 251,353 |
| Naib: + semantic | 0.287 | 55% | 255,826 | 0.356 | 63% | 160,673 |
Against an agent driving grep, Naib retrieves 2.9× more of the gold set on train and 5.7× more on holdout, at 22× and 40× lower cost per success. Absolute recall of 0.29–0.36 is the honest difficulty of the task once the answer is not already in the corpus.
Which differences are real
Paired bootstrap on the per-instance recall difference, 10,000 resamples, 95% interval. “Established” means the interval excludes zero.
| Comparison | Train | Holdout | ||||
|---|---|---|---|---|---|---|
| Δ recall | 95% CI | Verdict | Δ recall | 95% CI | Verdict | |
| lexical → + symbol | +0.0453 | [0.0130, 0.0815] | Established | −0.0040 | [−0.0490, 0.0392] | Not established |
| + symbol → + semantic | +0.0185 | [−0.0284, 0.0664] | Not established | +0.0574 | [0.0065, 0.1146] | Established |
| grep → full stack | +0.1878 | [0.1048, 0.2689] | Established | +0.2940 | [0.1838, 0.4121] | Established |
Only the comparison against grep clears zero on both splits. Each individual leg clears on one split and not the other, which is the signature of effects sitting at the resolution limit of a single epoch rather than of genuine per-split differences.
An effect that five smaller samples could not resolve.
The vector leg saw only the first 1,400 characters of each file — 9.3% of Playwright’s source. Splitting oversized files into window-sized chunks at symbol boundaries, scoring the chunks and collapsing them back to one hit per file, was tested five times and returned an interval containing zero every time. The point estimate never moved: +0.022 at n=151, +0.0224 at n=662. The sample was the problem, not the effect.
| Stratum | n | Δ recall | 95% CI | Hit rate | Verdict |
|---|---|---|---|---|---|
| All instances | 662 | +0.0224 | [0.0064, 0.0390] | 58% → 63% | Established |
| — train | 439 | +0.0203 | [0.0004, 0.0402] | 58% → 62% | Established |
| — holdout | 223 | +0.0266 | [0.0004, 0.0535] | 59% → 65% | Established |
| — mature code | 485 | +0.0284 | [0.0081, 0.0487] | 63% → 67% | Established |
| — greenfield | 177 | +0.0058 | [−0.0187, 0.0297] | 45% → 52% | Not established |
| — TypeScript | 573 | +0.0178 | [0.0009, 0.0351] | 58% → 62% | Established |
| — Python | 89 | +0.0520 | [0.0078, 0.0993] | 60% → 65% | Established |
A cluster bootstrap that resamples whole epochs rather than instances gives [0.0017, 0.0425] — still excluding zero, so the result does not depend on treating correlated instances as independent.
Python gains three times what TypeScript does
+0.052 on Pydantic against +0.018 on TypeScript, and Pydantic is the only single corpus that clears zero on its own. Chunking generalizes past the ecosystem it was built in and works better there — flatter module structure and longer functions mean the fixed window was starving Python worse.
Cold start gains nothing
On the 177 greenfield instances the interval is [−0.019, 0.030] and contains zero. Their hit rate is 45% against mature code’s 63%: retrieval for work that creates new files is substantially harder, and nothing in this round improves it.
What it costs
Pooled over all 662 paired instances. Both accountings charge for real file contents; the second charges only the matching span rather than the whole file.
| Arm | Tokens to first gold | Files opened | Resolved | Cost/success | Span cost/success |
|---|---|---|---|---|---|
| Baseline | 107,683 | 6.1 | 58% | 184,678 | 3,502 |
| Chunked | 146,033 | 5.8 | 63% | 232,949 | 3,164 |
| Change | +35.6% | −4.9% | +5 pts | +26.1% | −9.6% |
Chunking opens fewer files but larger ones. A file’s score is its best chunk, and a large file split into many chunks gets many independent chances at a high score — a correction for that bias was implemented and measured, and did not remove the effect.
The −9.6% span figure in this table is a simulation: it re-charged rankings already produced at a flat 350 tokens per served file. Span delivery has since been built and measured, and the section below supersedes it.
Span delivery works. It does not make chunking pay.
Retrieval now returns the fragment that matched — file_path, line bounds, skeleton context and span content — verified against every chunk in this repository: 3,972 spans, 100% reproduce their chunk exactly when sliced out of the file by the reported bounds, none out of range.
Re-measured with the benchmark charging what is actually delivered rather than the file it came from, across all 662 paired instances:
| Population | n | Change in tokens to first gold | 95% CI | Verdict |
|---|---|---|---|---|
| All instances | 662 | +127.19% | [+75.81%, +195.14%] | Costs more |
| — received a span | 190 | +31.00% | [−5.60%, +80.83%] | Not established |
| — received none | 472 | +165.91% | [+94.03%, +257.84%] | Costs more |
A cluster bootstrap resampling whole epochs gives [+64.87%, +202.44%] — still entirely above zero, so this is not an artefact of treating correlated instances as independent. All six epochs cost more individually.
The damage is on the instances that never received a span. Chunking changes the ranking for every query, but only 0.36 results in ten carry a fragment. On the 190 instances that got a span the effect is indistinguishable from zero; on the 472 that did not, it is +165.9%. Chunking pays a ranking cost across the whole result set to narrow one result in three.
KEEL_CHUNK stays off, permanently. A 2.2% recall gain does not justify more than doubling the tokens an agent reads to reach it — for a system whose purpose is spending fewer tokens on files nobody needed. The chunking and span-delivery code remain merged, correct and tested; what is rejected is the economic case for enabling them.
How the pooled number lied
Pooled across the same 662 instances, span delivery looks like a 10.9% saving. Per instance it is a 127% regression. Both are arithmetic on identical data.
The pooled figure is a ratio of sums — total tokens divided by total resolved instances. That statistic is dominated by its largest terms, and file sizes in these corpora span four orders of magnitude, up to 1.1 million characters. A handful of instances where chunking happened to avoid one enormous file outweighed hundreds where it cost more.
An agent does not experience a ratio of sums. It experiences one query at a time, so the per-instance relative change is the statistic that matches the thing being optimised. Averaging it inverts the sign.
This is the third measurement in this programme to look like a result and turn out to be an accounting artefact, after the 815× cost claim and the +43% recall gain. Each survived because nobody asked what the denominator was.
Why sample size was the whole story
| Instances | 95% CI half-width | Observed effect | Verdict |
|---|---|---|---|
| 151 (one epoch) | ±0.045 | +0.022 | Not established |
| 518 (four epochs) | ±0.024 | +0.022 | Marginal |
| 662 (three repositories) | ±0.016 | +0.022 | Established |
The preregistered rule is that a change ships only when the paired interval excludes zero. Chunking failed it five times and was left unmerged each time. Nothing about the change improved between those runs; only the measurement did.
Not established
Cold-start retrieval. 177 greenfield instances show no chunking benefit and a 45% hit rate. Unaddressed.
A cheaper serve window. With chunking rejected, retrieval delivers whole files and the context cost is unaddressed. Whether the serve limit of ten is right at all is untested: if the ranking is confident in its top two, results three through ten are pollution.
Sufficiency. Recall measures whether gold files were returned, not whether an agent received enough context to make the change correctly. Grading spans against file-level gold cannot answer that; it needs line-level ground truth that has not been built.
Gold is still commit-derived. A commit’s file set is what was edited, not what had to be read. A file consulted and left unchanged scores as a false positive, equally for every arm.
Method: keel/scripts/make-temporal-gold.mjs (temporal split, denominator and greenfield rules), keel/scripts/run-retrieval-benchmark.mjs (per-arm cost accounting). Corpora: Playwright at four epochs, Pydantic, and this repository, each searched at a checkout of its own epoch commit. Baseline and chunked arms come from one build via the KEEL_NO_CHUNK ablation switch, so the paired comparison is not confounded by anything else that changed between them.