Skip to main content
Every figure in Evaluation methodology and results is produced by a command in this repository. Nothing on that page is transcribed by hand; anyone can regenerate all of it. This page has the commands, the provenance of the corpus, and how the ranking stays current as the spec moves.

​
Run it

From a checkout, in the apps/bazaar package:
What each prints:
The pnpm eval slices are deterministic (same corpus, same committed judgments), so they reproduce exactly. The 18,450-document rows re-derive their silver distractors from a pinned upstream source, so those figures move by a point or two per rebuild. That is stated on the results page rather than hidden, because it is true.
The dev slice may be inspected and tuned against. The locked slice is held back: the harness reports its failures only in aggregate and refuses to name the queries that failed, so it cannot quietly become a training set.

​
The 18,450-document eval pack

The stress test needs a large, adversarial catalog. It is built from public, license-clean sources and it is an evaluation fixture only, never the live catalog.
The silver documents can never be served, ranked into, or leaked. They use .invalid hosts, their quality signals are zeroed, and their payTo is a sentinel. They exist only in-process during evaluation. The gold set is our 2,000 real captured listings; the silver set is 16,450 off-distribution distractors around them.
Provenance is pinned and checksummed in eval-pack/manifest.json: Both sources are permissively licensed, so the pack adds no license risk to a permissively-licensed project. manifest.json records the SHA-256 of the corpus and judgment files, so anyone can confirm the exact dataset a number was measured against, and so the dataset cannot be silently swapped to flatter a result.

​
Staying current as the spec moves

The x402 discovery conventions keep changing upstream: filters, metadata fields, and cataloging behavior have all moved and will move again. Search and cataloging keep up in four concrete ways:
1

Pinned spec SHAs

Every spec file the implementation conforms to is pinned by commit SHA. A snapshot records which spec text each behavior implements, so drift is a diff, not a guess.
2

A daily spec-drift watch

A scheduled check diffs the pinned upstream spec files every day. When upstream moves, the change is measured against the test suite and a real testnet settlement before the pins are updated.
3

A dual conformance run

The conformance harness runs the upstream e2e suite twice, at the pinned spec and at the latest upstream main, and compares. That is what catches a break introduced by an upstream spec change rather than by our own code.
4

CI ranking floors

The search slices run in CI with regression floors. A ranking change that makes retrieval worse on realistic data fails the build, so quality upkeep is enforced mechanically, not by good intentions.
The judgment set itself is the current bottleneck, not the corpus, so growing the blind judgment set and re-measuring is ongoing work, not a one-time pass. A few facilitator behaviors that search and cataloging depend on are documented in detail elsewhere: Interoperability is handled the same way: listings are shaped so a stock x402 SDK client round-trips them unchanged, the list-versus-search response-key asymmetry is honored exactly (How search works), and the upstream e2e suite validates the wire shape end to end (Conformance).

​
Next steps

Retrieval architecture

The ranker design in full.

Evaluation methodology and results

The numbers and the methodology behind them.

Conformance

Wire-level conformance against the upstream e2e suite.

Error registry

The machine-readable codes every rejection carries.