> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rail402.dev/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Rail402 is an x402 payment facilitator, Stellar-native Bazaar discovery layer, and agent tooling for the Stellar network. It currently targets stellar:testnet.
> The live testnet facilitator is https://facilitator.rail402.dev with endpoints /verify, /settle, /supported, /health, and /discovery/*.
> Payment amounts use 7-decimal SEP-41 integer (stroop) arithmetic. Never use floating-point math for amounts.
> Every rejection returns a machine-readable error code and a non-null human-readable reason. When explaining a failure, surface both.
# Evaluation methodology and results
> How Rail402 measures Bazaar search quality: blind graded human judgments over real captured listings, a tunable dev slice and a measured-once locked slice, standard IR metrics with Wilson confidence intervals and CI regression floors, and results across 20, 2,000, and 18,450 documents.
Search quality here is measured, not asserted. The metrics on this page are computed in CI, and a ranking regression fails the build. Every number is regenerated by the commands in [Reproduce the numbers](/architecture/reproduce); nothing is hand-entered.
A search score is only as trustworthy as the corpus and the judges behind it, so this page states both before it states a number.
## The judgment sets
Two corpora are measured, and they are **never merged into one score**. A single blended number would let an easy corpus carry a hard one, which is precisely how a search system comes to be described as excellent by the team that built it.
2,000 real listings captured from a live x402 Bazaar, with **202 blind, graded human judgments**. Queries were written by reading each resource's purpose and phrasing a request an agent would actually make, **before** measuring anything.
ToolACE-derived queries, each written against its own answer. Lexical overlap flatters any ranker, so silver measures **recall and robustness under load, not precision truth**, and never gates a release.
The 202 gold judgments are split into two slices by a hash of the query, so neither slice is the easy half:
* **`dev`**, inspectable and tunable, like any development set.
* **`locked`**, measured and **never diagnosed**. When a locked query fails, the failure is reported in aggregate and left alone. The harness deliberately **refuses to print** which locked queries failed, because a held-out set whose failures you study is just a slower training set. That discipline is enforced by the tool, not by remembering to look away.
133 of the 202 judgments are multi-relevant (2 to 4 graded answers that must be *ordered*, the sibling-discrimination case), and by construction the great majority share few or no path tokens with their best answer, so lexical overlap alone cannot solve them.
## Grading and metrics
Relevance grades are **explicit and judge-assigned** (`3` clearly-best, `2` strongly relevant, `1` partially relevant), never inferred from a result's position in the list. Two equally-good answers both get `3`.
| Metric | What it captures | Note |
| --------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------- |
| precision\@1 | Is the first result right? | The number an agent feels most. |
| recall\@5, recall\@10 | Did the right answer(s) surface at all? | Lead metric under heavy distractor load. |
| MRR | How high did the first relevant result land? | |
| nDCG\@10 | Graded ranking quality | Exponential-gain form `(2^g - 1)` (Burges et al., ICML 2005), not the linear J\&K form. |
| zero-result rate | How often does a query return nothing? | An online signal, measured offline too. |
Every proportion is reported with a **Wilson 95% confidence interval**, and each slice carries **CI regression floors**, set from the first measured run, slightly below it, so they are a guard against getting worse, not a target nobody has hit. A change that drops a slice below its floor fails the build.
**precision\@5 is deliberately not featured.** Most gold queries have only 1 to 2 relevant documents, so precision\@5 is structurally capped near 20% no matter how good the ranking is, and quoting it unqualified would be meaningless in either direction. recall\@k and nDCG\@10 are the honest metrics for this shape of task.
## Results
Regenerated 2026-08-16. The 20-document and 2,000-document sets are deterministic and reproduce exactly; the 18,450-document rows re-derive their silver distractors from a pinned upstream source, so those figures move by a point or two per rebuild (stated plainly, because it is true).
### At small scale: 20 real listings
A near-perfect score over a handful of hand-written fixture APIs reflects the authoring, not the ranker. Measured over 20 *real* captured listings, sixteen of them near-identical siblings of one service (the genuinely hard case), the ranker still leads:
| Slice | n | p\@1 | recall\@5 | recall\@10 | MRR | nDCG\@10 | zero-result |
| ------ | -- | ----- | --------- | ---------- | ----- | -------- | ----------- |
| locked | 10 | 80.0% | 100% | 100% | 0.900 | 0.926 | 0% |
| dev | 10 | 100% | 100% | 100% | 1.000 | 0.980 | 0% |
But 20 documents is a small ruler, and we do not stop there: the next two tables are the honest ones.
### At realistic scale: 2,000 real listings
The same 202 blind judgments, scored against all 2,000 real captured listings:
| Slice | n | p\@1 (95% CI) | recall\@5 | recall\@10 | MRR | nDCG\@10 | zero-result |
| ------ | --- | --------------------- | --------- | ---------- | ----- | -------- | ----------- |
| locked | 105 | 50.5% \[41.1%, 59.9%] | 54.0% | 66.5% | 0.616 | 0.569 | 0% |
| dev | 97 | 48.5% \[38.8%, 58.3%] | 58.4% | 67.9% | 0.632 | 0.605 | 0% |
Against a realistic catalog the ranker puts the right resource first about half the time and surfaces it in the top ten two-thirds of the time. That is the number to quote, and improving it is real work rather than a tuning pass. The two slices agreeing to within about 2 points on p\@1 is the evidence the dev/locked split is fair.
The hybrid earns its place here: an ablation on this set moves BM25's nDCG\@10 from 0.518 to the hybrid's 0.594 (MRR 0.551 to 0.626), a per-query win (82 better, 53 worse, 67 unchanged) that a two-sided sign test puts at `p = 0.016`, reproducible with `pnpm ablation`.
### Under stress: 18,450 documents
The robustness test: the same real listings and the same judgments, buried under about 16,000 off-distribution distractors. Does the right *real* answer survive?
| Slice | Kind | n | p\@1 (95% CI) | recall\@10 | MRR | nDCG\@10 |
| ---------------- | ------- | ----- | --------------------- | ---------- | ----- | -------- |
| gold · locked | gold | 105 | 30.5% \[22.5%, 39.8%] | 54.0% | 0.437 | 0.424 |
| gold · dev | gold | 97 | 29.9% \[21.7%, 39.6%] | 49.9% | 0.419 | 0.411 |
| silver · toolace | silver | 2,000 | 40.5% \[38.4%, 42.7%] | 68.0% | 0.510 | 0.532 |
| keyword | derived | 97 | 30.9% \[22.6%, 40.7%] | 39.7% | 0.372 | 0.341 |
| typo | derived | 105 | 19.0% \[12.7%, 27.6%] | 42.6% | 0.303 | 0.304 |
Read **recall\@10** first here: with the corpus nine times larger, the right answer still lands in the top ten more than half the time on gold (54.0%), and the typo slice (every query deliberately misspelled) still recovers it 42.6% of the time, evidence the typo rescue is doing real work at scale.
**Robustness delta (gold · locked, 2,000 to 18,450 documents):** p\@1 50.5% to 30.5%, recall\@10 66.5% to 54.0%, nDCG\@10 0.569 to 0.424. Precision degrades gracefully under a 9× distractor load rather than collapsing, which is the property a stress test exists to show.
The **zero-result rate is 0.0% on every slice above**: the ranker always returns something to rank, so a low precision score is a ranking problem to improve, never a silent empty response.
MCP tools, where one endpoint multiplexes several tools keyed on `toolName`, are measured on their own slice against the full catalog (locked p\@1 85.7%, recall\@10 71.4%, n=7). That `n` is a tripwire, not evidence; see below.
## How to read a search-quality claim
The numbers above are worth exactly as much as the questions you can ask of them. These are the questions, vendor-neutral, and the ones this page has tried to answer in advance.
1. **What is `n`, and what is the confidence interval?** A bare "92% precision" or "nDCG 0.93" is a point estimate. At 10 to 50 queries the 95% interval is often 30 to 50 points wide, wide enough that the headline and a coin flip are hard to tell apart. A credible claim reports the sample size *and* the interval beside every number.
2. **How big is the corpus, and where did the documents come from?** Real captured listings or invented fixtures? A near-perfect score over a few dozen hand-written APIs measures the authoring, not the ranking. Known-item retrieval over a tiny self-made corpus scores near the ceiling of the instrument regardless of ranker quality.
3. **Who wrote the relevance judgments, and had they seen the results first?** The strongest evidence is *blind human* judgments: a query written from a service's own description, before any results are seen. Weakest is judgments written by whoever wrote the ranker, or by an LLM (worse still, the same model that authored the corpus). "Human-reviewed" is not "human-graded"; ask which.
4. **Is there a held-out slice that is never tuned against, and is it actually withheld?** If every tuning decision is validated on the set it is reported on, the test set has quietly become the training set. Look for a locked split chosen reproducibly, ideally with a harness that refuses to reveal which held-out queries failed.
5. **Can you re-run it, and is the corpus pinned?** A real claim ships a command that reproduces the number and a content hash pinning the exact corpus, so the dataset cannot be silently swapped to flatter the result.
6. **Was the number measured through what is actually deployed?** The reported retrieval method should equal the served one (same model, same weights, same code path), against a live endpoint returning the spec-correct shape. A number measured on a model the production service does not run proves nothing about what an agent experiences.
7. **Is abuse resistance measured, not just asserted?** "Spam-resistant" should come with a rank delta: how far did keyword-stuffing or fake popularity actually move an attacker, and can the popularity signal be self-manufactured for free?
The flip side: a lower, well-caveated number is usually more trustworthy than a high one with none. A ranker that scores 100% on a synthetic set and about 50% on a real blind-judged one, and publishes both, is showing you how much of any score is the corpus rather than the ranker.
## What we deliberately do not claim
* **`n ≤ 10` slices are tripwires, not evidence.** The 20-document and MCP-tool numbers are useful as regression guards and as a floor; a two-query move inside them is noise, and they are never quoted as the headline. The 202-judgment broad set is the set that can actually tell two rankers apart.
* **Silver measures recall and robustness, not precision truth.** Because a silver query is written against its own answer, its lexical overlap flatters every ranker. It is reported apart from gold, always, and never gates a release.
* **The locked slice is never tuned against.** Several field-weight variants beat the shipped weights on the locked slice in ablation; none was adopted, because picking one because it wins on the held-out set converts that set into a training set in a single commit.
* **Ranking over MCP tools is measured against a synthetic tool corpus,** not a large real one yet: the 2,000-document gold corpus is 100% HTTP resources. A real MCP corpus with blind judgments is committed follow-on work.
* **The bottleneck now is judgment count, not corpus size.** Growing the blind judgment set and re-measuring is ongoing, not a one-time pass.
## Next steps
Every command, the eval pack's provenance, and the drift plan.
The ranker these numbers measure.
Wire-level conformance against the upstream e2e suite.
The lighter conceptual introduction.