Complete 323-query evaluation
BM25 + Jev reranking,
without a vector database.
Most RAG stacks begin by adding embeddings and a vector database. This benchmark asks a narrower question: how far can a transparent local retriever go when Jev judges its evidence before generation?
The smallest path
Keep retrieval, judgment and generation separate.
- SQLite FTS5/BM25 searches local files.
- Up to 30 passages become a bounded candidate set.
- Jev judges whether each passage contains evidence for the question.
- Application code sorts the returned probabilities.
- A separate answer model writes from retained passages and cites the files.
There is no vector database in the default path. Embeddings remain available as an optional retrieval path when lexical recall is not enough.
Measured result
Reranking helped; wider recall helped more.
The evaluation uses the complete BEIR NFCorpus test split: 3,633 documents and 323 queries. Every path uses the same qrels and metric code. The primary metric is nDCG@10.
| Pipeline | nDCG@10 | Change from BM25 top 30 |
|---|---|---|
| BM25 top 30 | 0.305654 | Baseline |
| BM25 top 30 + Jev | 0.353235 | +15.57% |
| BM25 top 50 + Jev | 0.362468 | +18.59% |
| Agentic lexical top 50 + Jev | 0.430969 | +41.00% |
| Hybrid top 50 + Jev | 0.444327 | +45.37% |
| Agentic Hybrid + rank fusion | 0.450750 | +47.47% |
These are project-run results, not an official MTEB submission. Reproduce the comparison and evaluate it on your own corpus before choosing a pipeline.
Clean comparison
Jev improved the order of the same BM25 candidate pool.
BM25 top 30 and BM25 top 30 + Jev use the same candidates. Jev raised nDCG@10 from 0.305654 to 0.353235, a 15.57% relative gain. That isolates the contribution of reranking from changes in first-stage recall.
The larger gains came from widening the candidate pool before Jev. Agentic lexical search generates several bounded searches, executes them locally, fuses the results, and then sends the shortlist to Jev. It reached 0.430969 without maintaining a document embedding index.
The highest point estimate came from Agentic Hybrid with local rank fusion at 0.450750. That path adds online planning and embedding retrieval, so the highest score is not automatically the best operational choice.
Failure diagnosis
Reranking cannot repair a missing candidate.
Jev can improve the order of passages that the retriever found. It cannot judge a passage that never entered the candidate set. That gives each failure a different repair:
- If the answer-bearing passage is absent, improve retrieval recall, query planning, or chunking.
- If the passage is present but ranked too low, improve reranking.
- If the right evidence is retained but the answer is wrong, inspect the answer prompt and generation stage.
Negative result
A more elaborate Jev program was not automatically better.
A unified Jev Passage Gate combined relevance and policy-style questions in one fixed gate. It reached 0.376298 nDCG@10 and excluded 84.4% of candidates. That was worse than the stronger retrieval-plus-reranking paths.
Publishing the miss matters: questions and thresholds need calibration against the target corpus. Complexity by itself is not an improvement.
Data boundary
Local-first does not mean fully offline.
Source files, the SQLite index, and lexical retrieval stay on the machine. In the default hosted-provider configuration, selected passage text is sent to the Jev provider, and retained evidence is sent to the answer model. If documents must never leave the machine, replace those stages with local models.
Practical choice
Start with the smallest path that addresses the failure.
Test BM25 + Jev first when the corpus contains product names, identifiers, technical terms, code symbols, or other language that overlaps with user queries. Add agentic lexical search when several exact-term searches are needed but you still want to avoid a document embedding index. Add hybrid retrieval when synonyms and paraphrases make lexical recall fail.