Are we finding the right code?
A plain-language view of the retrieval experiments. Every result is tied to a split, method, file budget, and corpus fingerprint.
blocked
What we are actually testing
Our immediate goal is narrow: find the right code.
For a task such as “preserve this compiler behavior,” we want the retrieval method to put the files a developer needs at the top of the list, while keeping irrelevant files out. We are not yet claiming that retrieval saves tokens or makes an agent finish tasks more cheaply.
Token accounting is the next phase. We will complete it only after the file-finding capability is demonstrable, because a small token count is not useful if it omits the file that contains the answer.
Freeze real tasks
We pin a repository commit. Each task is a real commit subject, and the files changed by that commit are the objectively known target files.
Build the search index
Naib turns each repository file into a local derived page containing facts such as its path, language, exports, and imports. The source code stays local.
Run every method identically
Naib lexical, grep, and Repowise receive the same task text and may return at most 10 files. No method gets extra search budget.
Score file finding
We compare each returned path with the commit’s changed-file list, calculate recall and precision, and record how many files were served.
Repeat and protect the holdout
Replicates check stability. We use the train split to find bugs and choose settings; we leave the holdout untouched until the method is frozen.
What the numbers mean
We want more of the files that really changed, with fewer irrelevant files handed to the agent.
Recall
Of the files that really changed, how many did the method find? Higher is better.
Precision
Of the files returned, how many really changed? Higher means less noise.
Files served
How many files the method puts in front of the agent. This is the reading burden.
Top-k
The maximum number of results allowed per task. The validated artifact records the configured value for every arm.
Replicate
Repeating a cell to check stability. Deterministic cells should match byte-for-byte.
Corpus fingerprint
A hash of indexed pages. If it changes, old and new scores are different experiments.
Holdout scorecard
All methods get the same tasks and the artifact-recorded file maximum on the evaluation split.
| Method | Recall | Precision | Files | Meaning |
|---|---|---|---|---|
| Keel lexical · 999-file pilot | 0.106 | 0.030 | 1564 | fail · One-replicate pilot holdout recall is 0.106 and precision is 0.030; below recall 1.000 and precision 0.800. |
| Keel grep · 999-file pilot | 0.232 | 0.065 | 1584 | fail · One-replicate pilot holdout recall is 0.232 and precision is 0.065; grep outperformed lexical retrieval but neither met the task gate. |
| Repowise Ollama · all-minilm · 999-file pilot | 0.053 | 0.015 | 1600 | fail · Real Repowise Ollama pilot completed one replicate; holdout recall is 0.053 and precision is 0.015. Repowise did not emit token accounting. |
| Repowise Ollama · nomic-embed-text · 999-file pilot | 0.030 | 0.009 | 1600 | fail · Real Repowise Ollama pilot completed one replicate; holdout recall is 0.030 and precision is 0.009. Repowise did not emit token accounting. |
| Repowise Ollama · embeddinggemma · 999-file pilot | 0.072 | 0.020 | 1600 | fail · Real Repowise Ollama pilot completed one replicate; holdout recall is 0.072 and precision is 0.020. This was the strongest embedding arm, but it did not meet the task gate. Repowise did not emit token accounting. |
Holdout recall
Did the method find the files that actually changed?
Current reading: The latest artifact is blocked.
Blocked arms are shown honestly and are not treated as passing evidence.
Run details and model accounting
These values come directly from the validated JSON artifact. A dash means the runner did not report that field.
| Arm / model | Dimensions | Tests | Files served | Tokens | Cost | Wall time |
|---|---|---|---|---|---|---|
| Keel lexical · 999-file pilot lexical | — | 160/160 | 1,564 | 463,523 | $0.0000 | 15.8s |
| Keel grep · 999-file pilot lexical | — | 160/160 | 1,584 | 429,943,315 | $0.0000 | 17.0s |
| Repowise Ollama · all-minilm · 999-file pilot all-minilm | 384 | 160/160 | 1,600 | 0 | $0.0000 | 325.0s |
| Repowise Ollama · nomic-embed-text · 999-file pilot nomic-embed-text | 768 | 160/160 | 1,600 | 0 | $0.0000 | 795.8s |
| Repowise Ollama · embeddinggemma · 999-file pilot embeddinggemma | 768 | 160/160 | 1,600 | 0 | $0.0000 | 737.5s |
The bounded pilot completed. The final acceptance run remains blocked until five independent replicates and a sealed holdout are run; web build passes, while the pre-existing web lint baseline remains.
Running results table
Every completed test stays here. A dash means that metric was not part of that test, not that it was zero.
| Test | Repository / split | Method | Recall | Precision | Files | What this tells us |
|---|---|---|---|---|---|---|
| fail | tracked_files_only_bounded_pilot · train + holdout | Keel lexical · 999-file pilot | 0.106 | 0.030 | 1564 | One-replicate pilot holdout recall is 0.106 and precision is 0.030; below recall 1.000 and precision 0.800. |
| fail | tracked_files_only_bounded_pilot · train + holdout | Keel grep · 999-file pilot | 0.232 | 0.065 | 1584 | One-replicate pilot holdout recall is 0.232 and precision is 0.065; grep outperformed lexical retrieval but neither met the task gate. |
| fail | tracked_files_only_bounded_pilot · train + holdout | Repowise Ollama · all-minilm · 999-file pilot | 0.053 | 0.015 | 1600 | Real Repowise Ollama pilot completed one replicate; holdout recall is 0.053 and precision is 0.015. Repowise did not emit token accounting. |
| fail | tracked_files_only_bounded_pilot · train + holdout | Repowise Ollama · nomic-embed-text · 999-file pilot | 0.030 | 0.009 | 1600 | Real Repowise Ollama pilot completed one replicate; holdout recall is 0.030 and precision is 0.009. Repowise did not emit token accounting. |
| fail | tracked_files_only_bounded_pilot · train + holdout | Repowise Ollama · embeddinggemma · 999-file pilot | 0.072 | 0.020 | 1600 | Real Repowise Ollama pilot completed one replicate; holdout recall is 0.072 and precision is 0.020. This was the strongest embedding arm, but it did not meet the task gate. Repowise did not emit token accounting. |
Token status for this table: deferred. We will add input-token and effective-cost columns after the file-finding gate is established and both methods are priced at the same useful output level.
Progress over time
Pending work stays visible instead of looking complete.
Train versus holdout
Train = tune
Use these tasks to find bugs and choose thresholds. A good train score is encouraging, not proof.
Holdout = judge
Keep these tasks untouched while choices are made. After freezing the method, they test generalization.
Why both?
If we keep changing the method after seeing holdout results, holdout becomes another train set.
Source: testing-results.json, validated by the shared retrieval result schema.