> ## Documentation Index > Fetch the complete documentation index at: https://docs.rail402.dev/llms.txt > Use this file to discover all available pages before exploring further. # Load test > Discovery and settlement latency of the release image under its production limits, with the script, datasets, configuration and raw results. This page gives the results of the published load-test package in `tools/loadtest/published/2026-10-26/`. The package holds every request of every run (`runs//requests.ndjson.gz`), each run's statistics and configuration, and at the top `summary.json` and `config.json`. `tools/loadtest/README.md` documents the driver, the datasets and the cache states. `tools/loadtest/package.sh` is the script that produced the package. All figures are milliseconds. Every latency is measured from when its request was due (open loop) or sent (closed loop), over every request sent, so a failure counts at the time it took to fail and a timeout at 30 s. ## Stack | Part | Value | | - | - | | Service | The release image built from commit `50e7461` (image ID `sha256:7da468cb9b1c…`), Node.js 24.19.0, ONNX Runtime (`onnxruntime-node`) 1.30.0, model `all-minilm-l6-v2` | | Service limits | 2 CPUs, 3 GiB, 512 processes, read-only root, no capabilities | | Postgres | `postgres:18.6-alpine`, a database of its own, 1.5 CPUs, 2 GiB | | Settings | The production facilitator settings, with the overrides `config.json` lists with their reasons (local private network, corpus assets, upto off, no rate limits, retention) | | Driver | Commit `0efb605`, Node.js 24.19.0, pinned to CPUs 6–7; service and Postgres on CPUs 0–3 | | Host | Intel Core Ultra 5 225, 8 cores, 50 GB | | Settle network | The local private Stellar network (quickstart, standalone), every payment on `stellar:testnet` | | HTTP cache | None at the origin: discovery responses carry no `Cache-Control` | | Dataset | Entries | SHA-256 (JSON Lines) | | - | - | - | | Catalog A | 500 | `9142d5257446d9956e36c352e0489c91259654ae6fda34d15d74a22d3744b2d1` | | Catalog B | 5,000 | `74c9dca16deac3885e2c25617f421bc146e22d9cda01942a48c2b700a71cbccd` | | Queries | 202 | `b6f46ec6325f415a194115e06dd9aec3b393a1ceda652a6e3f2e0cd25e8d5591` | | Uncached sample | 1,000 | `a62129ff40d120e58d662926c3a4d39f11bed07b3b8be5059e0d7ef88b34357d` | Each run's `host.json` records the listings by state before and after it: every listing of the catalog was published throughout. The host is shared, so the package was run with `LOAD_WAIT_BELOW=2`: each run started only once the host's 1-minute load average was below 2 (it was between 0.25 and 1.98 when the runs started). `host.json` records each run's wait (at most 240 s) and the load average before and after it. ## Cached and uncached search Open loop, 120 s after a 30 s warm-up. `warm`: every query's embedding is cached. `uncached`: every query is new, so each runs the embedding model, or is answered lexical-only with `partialResults: true` when the embedder is at its cap (`saturated`). `cold-index`: a listing is added every 10 s, and the first answer at each new catalog revision includes the index rebuild. | Catalog | State | Rate/s | Requests | p50 | p95 | p99 | Errors | Partial | | - | - | - | - | - | - | - | - | - | | A | warm | 50 | 6,000 | 3.8 | 8.2 | 11.6 | 0 | 0 | | A | warm (search) | 100 | 9,600 | 2.9 | 4.3 | 6.2 | 0 | 0 | | A | warm (list page) | 100 | 2,400 | 5.0 | 6.6 | 10.2 | 0 | 0 | | A | uncached | 100 | 12,000 | 8.1 | 10.3 | 15.0 | 0 | 0 | | A | uncached | 300 | 36,000 | 14.7 | 35.3 | 38.1 | 0 | 18,530 | | A | cold-index | 50 | 11 | 49.2 | 64.9 | 64.9 | 0 | 0 | | A | warm-index | 50 | 5,989 | 3.2 | 4.7 | 5.7 | 0 | 0 | | B | warm | 50 | 6,000 | 9.9 | 12.4 | 14.3 | 0 | 1,959 | | B | warm (search) | 100 | 9,600 | 9.1 | 12.1 | 22.1 | 0 | 3,134 | | B | warm (list page) | 100 | 2,400 | 19.6 | 30.3 | 40.5 | 0 | 0 | | B | uncached | 100 | 12,000 | 52.1 | 72.2 | 81.7 | 0 | 5,785 | | B | uncached | 300 | 36,000 | 257.1 | 625.3 | 791.1 | 2,198 | 31,821 | | B | cold-index | 50 | 11 | 376.3 | 446.7 | 446.7 | 0 | 2 | | B | warm-index | 50 | 5,989 | 10.3 | 13.9 | 282.7 | 0 | 1,957 | The `warm (search)` and `warm (list page)` rows are one run: 80 % searches and 20 % `/discovery/resources` pages at 100 requests a second. With `uncached` at 100/s on catalog A every query ran the model; at 300/s the embedder was at its cap for 18,530 of 36,000 queries, which were answered lexical-only. On catalog B, `partialResults` also marks warm answers whose semantic ranking was cut at its depth (about a third), and at 100/s the embedder was at its cap for 3,225 of the 12,000 uncached queries. A `/health` probe at 5/s ran beside the `warm` and `uncached` runs: p50/p95/p99 1.9/4.9/9.1 beside `warm` 50/s, 1.6/7.9/9.9 beside `uncached` 100/s and 4.8/8.0/9.2 beside `uncached` 300/s on catalog A; 6.7/10.4/11.5, 10.9/17.3/24.8 and 57.5/153.1/219.5 on catalog B. Every probe answered 200. ### Over capacity `uncached` at 300/s on catalog B is above what the service completes (see the curve below). The service sheds discovery load (see [Load limits](https://docs.rail402.dev/bazaar/search.md#load-limits)): at most 32 discovery requests run at once and 64 wait, each at most 1 s, and every other request is refused at once with `503` `discovery_overloaded`. Discovery reads the catalog through a database pool of its own, and a page reuses listings read within the last second at the same catalog revision. | Run (catalog B) | Label | Requests | p50 | p95 | p99 | Max | Answers | | - | - | - | - | - | - | - | - | | `uncached` 300/s | search | 36,000 | 257.1 | 625.3 | 791.1 | 1,181 | 33,802 × 200, 2,198 × 503 | | `uncached` 300/s | `/health` | 600 | 57.5 | 153.1 | 219.5 | 534.7 | 600 × 200 | | `uncached` 300/s with the settle probe | search | 36,000 | 244.8 | 615.7 | 809.0 | 3,608 | 33,956 × 200, 2,044 × 503 | The service answered 33,802 searches (282 a second, about the 285 a second of the 64-client closed loop) and refused 2,198 with `503` (median 29 ms to refuse), with no `500` and no timeout. 30,991 of the answered searches were lexical-only (`saturated`, `partialResults: true`). Every `/health` probe succeeded. The driver records statuses, not error codes; the service's log shows no error during the runs (one warning per stack: the corpus's testnet assets are absent from the private network). ## Concurrency curve Closed loop, 60 s after a 10 s warm-up: each client sends its next request when its previous one answers. Throughput is completed requests per second; latency is service time at that throughput. | Catalog | State | Clients | Requests/s | p50 | p95 | p99 | Saturated | | - | - | - | - | - | - | - | - | | A | warm | 1 | 375.2 | 2.5 | 3.4 | 5.6 | 0 | | A | warm | 4 | 1,139 | 3.1 | 5.3 | 9.6 | 0 | | A | warm | 16 | 1,196 | 12.8 | 24.1 | 29.2 | 0 | | A | warm | 64 | 1,010 | 66.0 | 119.3 | 149.8 | 0 | | A | uncached | 1 | 113.2 | 8.4 | 10.7 | 13.5 | 0 | | A | uncached | 4 | 149.1 | 26.8 | 35.1 | 37.9 | 1 | | A | uncached | 16 | 844.9 | 14.1 | 50.0 | 59.5 | 45,186 | | A | uncached | 64 | 711.0 | 101.9 | 143.5 | 162.7 | 36,988 | | B | warm | 1 | 113.7 | 8.8 | 10.8 | 12.4 | 0 | | B | warm | 4 | 154.2 | 26.3 | 34.6 | 39.2 | 0 | | B | warm | 16 | 157.4 | 105.1 | 137.9 | 159.1 | 0 | | B | warm | 64 | 153.3 | 444.7 | 587.9 | 633.0 | 0 | | B | uncached | 1 | 66.9 | 14.7 | 16.9 | 18.4 | 0 | | B | uncached | 4 | 81.4 | 48.7 | 63.2 | 67.9 | 0 | | B | uncached | 16 | 276.5 | 43.4 | 163.5 | 204.0 | 14,899 | | B | uncached | 64 | 285.3 | 155.9 | 496.6 | 653.8 | 15,717 | No closed-loop request failed. Past 4 clients, uncached throughput rises only because the embedder's cap answers most queries lexical-only (`Saturated`). ## Settlement under a search flood Real `exact` payments settled through `POST /settle` of the same process, 0.9 a second for 120 s, alone and beside an `uncached` flood, on each catalog. Every settlement succeeded. | Catalog | Run | Settlements | p50 | p95 | p99 | | - | - | - | - | - | - | | A | Alone | 108 | 1,214 | 2,222 | 2,234 | | A | Beside `uncached` at 100/s | 108 | 1,304 | 2,326 | 2,350 | | A | Beside `uncached` at 300/s | 108 | 1,484 | 2,496 | 2,528 | | B | Alone | 108 | 1,221 | 2,224 | 2,229 | | B | Beside `uncached` at 100/s | 108 | 1,770 | 2,812 | 2,880 | | B | Beside `uncached` at 300/s | 108 | 2,920 | 4,063 | 4,245 | Settle latency on the private network has two modes, about one and about two ledger intervals, and the share of each differs from run to run, which moves p50 between them. Within each mode the flood's cost shows: on catalog A the fast mode's median was 1,206 alone, 1,281 beside 100/s and 1,469 beside 300/s, the slow mode's 2,214, 2,295 and 2,484. On catalog B, where each uncached search takes more of the service's two CPUs, the flood costs more: 47 of 108 settlements fell in the slow mode alone (median 2,217), 97 beside 100/s (median 2,579) and 106 beside 300/s (median 2,928). The private network closes ledgers at its own pace, so the absolute figures are not those of a public network. ## Reproduce ```sh theme={"theme":{"light":"github-light","dark":"github-dark"}} docker build --build-arg RAIL402_VERSION=$(git rev-parse --short HEAD) -t rail402-facilitator:loadtest . docker compose --profile stellar up -d stellar IMAGE=rail402-facilitator:loadtest LOAD_WAIT_BELOW=2 tools/loadtest/package.sh ``` The run took 81 minutes on this host, waits for an idle host included, and writes a new package under `tools/loadtest/published//`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.