Reproducible retrieval study
I compared seven local knowledge-search paths on the same 3,633 documents and 323 public queries. The result is a practical map of where BM25, agentic query planning, embeddings, Jev reranking, and local rank fusion actually help.
Evaluation setup
Every headline result uses the complete BEIR NFCorpus test split: 3,633 documents, 323 test queries, the same qrels, and the same metric implementation. Candidate depths and remote stages are disclosed. These are project-run results, not an official MTEB submission.
The central lesson: retrieval, judgment, ranking policy, and generation are separate layers. A reranker can reorder candidates, but it cannot recover evidence that the first stage never retrieved.
Measured results
| Pipeline | nDCG@10 | MRR@10 | Recall@10 | Median latency | p95 latency |
|---|---|---|---|---|---|
| BM25 top 30 | 0.305654 | 0.512697 | 0.147309 | 1.92 ms | 6.60 ms |
| BM25 top 30 + Jev | 0.353235 | 0.585817 | 0.158667 | 1.08 s | 16.95 s |
| BM25 top 50 + Jev | 0.362468 | 0.593023 | 0.164474 | 1.92 s | 17.84 s |
| Two-level Jev Line Search | 0.366280 | 0.657660 | 0.169397 | 11.74 s | 35.75 s |
| Hybrid + Unified Passage Gate | 0.376298 | 0.618043 | 0.166977 | 1.56 s | 17.47 s |
| Agentic lexical top 50 | 0.380168 | 0.597940 | 0.185464 | 6.99 s | 24.39 s |
| BM25 + embedding RRF top 50 | 0.396712 | 0.632089 | 0.193977 | 1.38 s | 1.49 s |
| Multi-round Agentic Hybrid top 50 | 0.424145 | 0.637722 | 0.206303 | 6.99 s | 24.39 s |
| Agentic lexical top 50 + Jev | 0.430969 | 0.644041 | 0.204138 | 7.52 s | 26.35 s |
| Hybrid top 50 + Jev | 0.444327 | 0.654583 | 0.214907 | 3.17 s | 17.99 s |
| Agentic Hybrid + Jev/retrieval fusion | 0.450750 | 0.652606 | 0.220885 | 9.24 s | 38.72 s |
Latency covers retrieval and reranking only. One-time corpus indexing and final answer generation are excluded. Expensive remote pipelines use labeled cold pilots or per-query stage estimates, and provider latency can change.
Inspect exact configuration, cost, caveats, and reproduction commands → Inspect latency provenance →
The smallest path is local SQLite FTS5/BM25 retrieval followed by Jev evidence reranking. Adding Jev to BM25 top 30 raised nDCG@10 from 0.305654 to 0.353235. It is the most inspectable default and requires no embeddings or vector database.
Two planning rounds generate alternative lexical searches, execute them locally, and fuse the results before Jev. It improves vocabulary coverage without a document embedding index and reached 0.430969 nDCG@10, at the cost of online planner latency.
BM25 and embedding rankings are combined with reciprocal rank fusion before Jev. The wider candidate pool increased Recall@50 from 0.209810 to 0.318075; Jev then raised nDCG@10 to 0.444327. This is the strongest current quality/latency compromise.
Multi-round lexical planning runs beside the original-query embedding lookup. Jev alone reached 0.445761 nDCG@10; a zero-call local blend of Jev rank and retrieval rank reached 0.450750. The gain suggests that semantic judgment and retrieval priors contain complementary information.
A two-level corpus taxonomy improved candidate recall but finished slightly below ordinary Hybrid + Jev. More relevant candidates raised the ceiling, yet also made the ranking problem harder. Higher candidate recall was necessary, not sufficient.
The gate judged relevance, supporting evidence, contradiction, and prompt-injection risk. Its fixed policy excluded 84.4% of candidates and reduced nDCG@10 below bare Hybrid. It is a useful negative result: safety and evidence thresholds need task-specific calibration.
Line Search sent bounded windows of the full index through Jev Choice, then globally ranked the finalists. It achieved the strongest first-hit behavior, but weaker evidence-set ranking and the highest observed cost. It fits small, high-value corpora rather than a default retrieval path.
Practical choice
| Priority | Recommended mode |
|---|---|
| Minimum infrastructure and exact terms | BM25 + Jev |
| Vocabulary mismatch without embeddings | Agentic lexical + Jev |
| Strong quality with manageable latency | Hybrid + Jev |
| Maximum measured ranking quality | Agentic Hybrid |
| Evidence governance experiments | Passage Gate |
| Small-corpus deep search | Line Search |
The project is independent and is not affiliated with or endorsed by TypeSafe AI, OpenRouter, or MiniMax.