Evaluation methodology and results

Search quality here is measured, not asserted. The metrics on this page are computed in CI, and a ranking regression fails the build. Every number is regenerated by the commands in Reproduce the numbers; nothing is hand-entered. A search score is only as trustworthy as the corpus and the judges behind it, so this page states both before it states a number.

​

The judgment sets

Two corpora are measured, and they are never merged into one score. A single blended number would let an easy corpus carry a hard one, which is precisely how a search system comes to be described as excellent by the team that built it. The 202 gold judgments are split into two slices by a hash of the query, so neither slice is the easy half:

• dev, inspectable and tunable, like any development set.
• locked, measured and never diagnosed. When a locked query fails, the failure is reported in aggregate and left alone. The harness deliberately refuses to print which locked queries failed, because a held-out set whose failures you study is just a slower training set. That discipline is enforced by the tool, not by remembering to look away.

133 of the 202 judgments are multi-relevant (2 to 4 graded answers that must be ordered, the sibling-discrimination case), and by construction the great majority share few or no path tokens with their best answer, so lexical overlap alone cannot solve them.

​

Grading and metrics

Relevance grades are explicit and judge-assigned (3 clearly-best, 2 strongly relevant, 1 partially relevant), never inferred from a result’s position in the list. Two equally-good answers both get 3. Every proportion is reported with a Wilson 95% confidence interval, and each slice carries CI regression floors, set from the first measured run, slightly below it, so they are a guard against getting worse, not a target nobody has hit. A change that drops a slice below its floor fails the build.

​

Results

Regenerated 2026-08-16. The 20-document and 2,000-document sets are deterministic and reproduce exactly; the 18,450-document rows re-derive their silver distractors from a pinned upstream source, so those figures move by a point or two per rebuild (stated plainly, because it is true).

​

At small scale: 20 real listings

A near-perfect score over a handful of hand-written fixture APIs reflects the authoring, not the ranker. Measured over 20 real captured listings, sixteen of them near-identical siblings of one service (the genuinely hard case), the ranker still leads: But 20 documents is a small ruler, and we do not stop there: the next two tables are the honest ones.

​

At realistic scale: 2,000 real listings

The same 202 blind judgments, scored against all 2,000 real captured listings: Against a realistic catalog the ranker puts the right resource first about half the time and surfaces it in the top ten two-thirds of the time. That is the number to quote, and improving it is real work rather than a tuning pass. The two slices agreeing to within about 2 points on p@1 is the evidence the dev/locked split is fair. The hybrid earns its place here: an ablation on this set moves BM25’s nDCG@10 from 0.518 to the hybrid’s 0.594 (MRR 0.551 to 0.626), a per-query win (82 better, 53 worse, 67 unchanged) that a two-sided sign test puts at p = 0.016, reproducible with pnpm ablation.

​

Under stress: 18,450 documents

The robustness test: the same real listings and the same judgments, buried under about 16,000 off-distribution distractors. Does the right real answer survive? Read recall@10 first here: with the corpus nine times larger, the right answer still lands in the top ten more than half the time on gold (54.0%), and the typo slice (every query deliberately misspelled) still recovers it 42.6% of the time, evidence the typo rescue is doing real work at scale. The zero-result rate is 0.0% on every slice above: the ranker always returns something to rank, so a low precision score is a ranking problem to improve, never a silent empty response. MCP tools, where one endpoint multiplexes several tools keyed on toolName, are measured on their own slice against the full catalog (locked p@1 85.7%, recall@10 71.4%, n=7). That n is a tripwire, not evidence; see below.

​

How to read a search-quality claim

The numbers above are worth exactly as much as the questions you can ask of them. These are the questions, vendor-neutral, and the ones this page has tried to answer in advance.

1. What is n, and what is the confidence interval? A bare “92% precision” or “nDCG 0.93” is a point estimate. At 10 to 50 queries the 95% interval is often 30 to 50 points wide, wide enough that the headline and a coin flip are hard to tell apart. A credible claim reports the sample size and the interval beside every number.
2. How big is the corpus, and where did the documents come from? Real captured listings or invented fixtures? A near-perfect score over a few dozen hand-written APIs measures the authoring, not the ranking. Known-item retrieval over a tiny self-made corpus scores near the ceiling of the instrument regardless of ranker quality.
3. Who wrote the relevance judgments, and had they seen the results first? The strongest evidence is blind human judgments: a query written from a service’s own description, before any results are seen. Weakest is judgments written by whoever wrote the ranker, or by an LLM (worse still, the same model that authored the corpus). “Human-reviewed” is not “human-graded”; ask which.
4. Is there a held-out slice that is never tuned against, and is it actually withheld? If every tuning decision is validated on the set it is reported on, the test set has quietly become the training set. Look for a locked split chosen reproducibly, ideally with a harness that refuses to reveal which held-out queries failed.
5. Can you re-run it, and is the corpus pinned? A real claim ships a command that reproduces the number and a content hash pinning the exact corpus, so the dataset cannot be silently swapped to flatter the result.
6. Was the number measured through what is actually deployed? The reported retrieval method should equal the served one (same model, same weights, same code path), against a live endpoint returning the spec-correct shape. A number measured on a model the production service does not run proves nothing about what an agent experiences.
7. Is abuse resistance measured, not just asserted? “Spam-resistant” should come with a rank delta: how far did keyword-stuffing or fake popularity actually move an attacker, and can the popularity signal be self-manufactured for free?

The flip side: a lower, well-caveated number is usually more trustworthy than a high one with none. A ranker that scores 100% on a synthetic set and about 50% on a real blind-judged one, and publishes both, is showing you how much of any score is the corpus rather than the ranker.

​

What we deliberately do not claim

​

Next steps