# Methodology: how paidfor.dev measures x402 seller failure

This page explains the design and limitations of the measurement: the failure taxonomy, the reference price, the validators, on-chain settlement matching, the stratified sampling with 95% Wilson intervals, the spending caps, the verdict rule, and every known limitation. The report is a frozen snapshot of the 2026-10-24 to 2026-10-26 study; this site shows the live export, which keeps moving.

Initial study: 2026-10-24 to 2026-10-26, 12,992 paid calls; daily purchases since, every six hours from 2026-10-29.

## Snapshot versus live

The [full report](/report) is a frozen snapshot of the study that ran from **2026-10-24 20:25 UTC to 2026-10-26 00:19 UTC** (purchases), with discovery, on-chain measurement and probing from 2026-10-24 01:40 UTC to 2026-10-26 07:13 UTC. This site shows the **live export** of the same database, which keeps moving: the harness probes every seller's unpaid 402 quote hourly and makes a gated round of purchases every six hours (00:00, 06:00, 12:00 and 18:00 UTC). Where the two differ, the site's number wins and the difference is expected. Today the only difference is that the 15 `indeterminate` rows open at the time of the report have since resolved to `not_paid` after their 24-hour window, which does not touch the paid denominator. The numbers below are quoted from the report; the live numbers are on the [leaderboard](/).

## The question

Of paid x402 API calls on Base mainnet, what share fail the buyer, by count and by dollars, and why? The harness is a research instrument: it discovers sellers from the Coinbase Bazaar index, pays them with real USDC as a naive agent would, validates what comes back, confirms every settlement on-chain, and classifies every call with one primary outcome.

## Failure taxonomy

Every paid or probed call receives exactly one primary outcome and zero or more secondary flags. Precedence runs top to bottom; the first match wins, with ties resolved toward the outcome that loses the buyer the most money.

| outcome | meaning | in paid denominator | counts as failure |
|---|---|---|---|
| `our_error` | We sent a malformed request (400, 405, 415, 422, a 402 wrapping a validation error), hit a rate limit, had a local bug, or the cap logic stopped the call. No money moved. | no | no (count shown) |
| `not_paid` | The seller never produced a payable 402, or rejected the payment before settling; no settlement found within 24 h. No money moved, no goods received. | no | no (count shown) |
| `indeterminate` | Payment signed, no response, no settlement visible within 10 minutes. Polled for 24 h, then re-classified with the `resolved_late` flag. | no (separate) | no |
| `duplicate` | The same purchase intent produced more than one on-chain settlement. | yes | yes |
| `non_delivery` | Payment settled on-chain, then the response was 4xx or 5xx, empty, truncated, the wrong content type, or nothing arrived within the seller's own declared timeout. | yes | yes |
| `wrong_goods` | Response well-formed but does not match the request. Needs a per-service schema; without one the row is flagged `schema_missing` and this cannot be assigned. | yes (unmeasured here) | yes |
| `overcharge` | The 402 quote exceeded the reference price for the resource. The harness still pays when the quote is under the per-call cap, because a naive agent would. | yes | yes |
| `injection` | The response body contains text addressed to an agent rather than data. Flagged for human review, reported separately. | yes | no |
| `ok` | Paid, response received, every validator that ran passed. | yes | no |

The paid denominator is `ok`, `overcharge`, `non_delivery`, `wrong_goods`, `injection` and `duplicate`. Headline failures are `overcharge`, `non_delivery`, `wrong_goods` and `duplicate`. Flags: `payto_changed`, `price_discrimination`, `slow`, `schema_missing`, `overcharged`, `unpaid_overcharge`, `injection_hit`, `resolved_late`, `truncated_by_us`, `rate_limited`, `charged_then_rejected`.

## Reference price for overcharge

In order of preference: the price listed in the Bazaar index for the resource; else the modal quote seen by the probe module over the last 7 days; else the first quote ever observed. Every overcharge row in the study used the Bazaar listing. The quote-to-reference ratio is stored on every row, and the report shows the headline rate with overcharges below 1.1× and 2× removed so a reader can discount one-cent jitter.

## Validators

Eight pluggable validators run on every delivered response and store machine-readable evidence: `http_status`, `non_empty`, `content_type`, `not_truncated`, `json_schema`, `entity_matches`, `freshness`, `injection_scan`. A validator with no configuration returns `skipped`, never `pass`. Because no per-service schemas were written, every row carries `schema_missing` and `wrong_goods` is unmeasured.

## Settlement matching

After every paid call the harness looks for the USDC `Transfer` from its wallet to the pay-to address whose transaction also carries an `AuthorizationUsed(authorizer, nonce)` log matching the nonce it signed (EIP-3009 `transferWithAuthorization`). From phase 2 on, the seller-reported transaction hash was verified first by receipt on the canonical RPC, falling back to a log scan. In the study 12,955 settlements were matched this way, 57.96 USDC. Every evidence row on a seller page links the matched transaction on Basescan.

## Sampling and statistics

Stratified by seller tier (top 20 by 30-day on-chain USDC inflow versus a seeded random tail) and category (inference, data, scraping, compute, other), with calls weighted toward cheap resources, at most 10 resources and 150 calls per seller per plan, and one call per resource per minute. Six plans ran in sequence: phase 1 (top 20 + 40 tail sellers), phase 2 (top 20 + 300 tail, widened mid-phase to 600), phases 3 to 6 (same wide set, fresh seeds). Sellers that entered through the widened phases carry no tier label and show as `unassigned`.

Intervals are 95% Wilson. The 3,000-call target was chosen so a 10% rate would carry about ±1.1 points; the final 12,992 calls carry ±0.43. The dollar-weighted interval uses Kish's effective sample size and is labelled an approximation. A seller row needs at least 30 paid calls, where the interval is about ±10 points: enough to separate a 5% seller from a 40% one. Sellers with 5 to 29 calls are shown greyed out as low-sample and can never receive the `avoid` verdict.

## Caps and safety

Dry-run by default; real money required `--live`, an environment flag, the mainnet network id, and a passing 329-test suite recorded for the exact commit. Caps enforced in code before any signature: 0.50 USD per call, 20 USD per seller per day, 100 USD per day, 300 USD total. Every payment attempt was written to an append-only ledger before signing. 63 rows were re-classified during the study as validator bugs were fixed; each change is recorded with its reason and commit. The stress tests (duplicate and timeout, 10 sellers each, 0.20 USDC) are never mixed into the headline rate.

## Verdicts on this site

The verdict is a display rule on top of the data, not part of the study: `avoid` when the lower bound of the 95% interval is above 10% and the seller has a full sample; `caution` when the point estimate is above 2% or the sample is under 30 calls; `ok` otherwise. It lives in one function and may change; the underlying rates and intervals do not.

## Two series

Every seller can carry two series, shown side by side and never merged silently. **Measured** is our own purchases, described above. **Reported** is outcomes posted by operators running [@paidfor/x402-safe-fetch](https://www.npmjs.com/package/@paidfor/x402-safe-fetch) with telemetry set to `verified`: each row names the payer, the EIP-3009 nonce, the pay-to, the amount and the settlement transaction, and counts only after we find, on the USDC contract on Base, `AuthorizationUsed(payer, nonce)` and in the same transaction a `Transfer` from the payer to the pay-to of exactly that amount. Payment is proven on-chain; the outcome is as reported by the operator's client; we spend our own funds to adjudicate disagreements.

- **Verification.** A job every 5 minutes checks up to 200 pending rows: the reported transaction's receipt first, otherwise a log search from an hour before to 24 hours after the reported settlement time. A row is `unverified` when no authorization is found (`no_authorization_event`), when the authorization is found but the transfer does not match (`transfer_mismatch`), when the chain could not be read after 3 tries over an hour (`rpc_error`), or when it settled more than 7 days before we saw it (`too_old`). Unverified rows are kept 30 days for debugging and never counted. One payer and nonce is one row: a repeat post changes nothing.
- **Capping.** A payer counts for at most 50 calls per host per UTC day; rows beyond that are stored but not counted, so one busy operator cannot set a seller's rate.
- **Maturity.** The reported series is mature for a seller at 30 counted calls from at least 3 distinct payers. Until then it is shown greyed, labelled "not yet mature", and has no effect on the verdict.
- **Verdict.** The measured series decides as above. When the reported series is mature, each series is judged by the same rule and the verdict is the worse of the two; the API says which with `verdict_basis` (`measured` or `measured+reported`).
- **Statistics.** Rate and 95% Wilson interval over counted calls: `ok` plus the failure outcomes (`non_delivery`, `overcharge`, `wrong_goods`, `duplicate`, `paid_no_response`, `validator_failed`). A verified `error` is listed but not counted. Refusals (the client declined to pay: `refused_verdict`, `refused_ratio`, `refused_max_per_call`, `no_acceptable_quote`) have no payment to prove and are reported separately by reason, never in a rate.
- **Disagreement.** When the two 95% intervals do not overlap, the seller page shows both numbers side by side.
- **The headline** on the home page is the measured number only.

Every host with a reported series is listed at [/v1/reported](/v1/reported), next to its measured numbers; an operator can check one row at `/v1/telemetry/<payer>/<nonce>`.

## Limitations

- `wrong_goods` is unmeasured: no per-service schemas or entity extractors were configured, so every row is `schema_missing` and the headline is a lower bound on the failures a careful buyer would count.
- Overcharge is relative to the Bazaar listing; a seller that lists a stale price and quotes its real one is counted as overcharging, which is the buyer's experience but not necessarily intent.
- 9 `non_delivery` rows (6 of them api.slamai.dev) timed out in our client at 60 s before the seller's declared 300 s window; they are an upper bound on those sellers' fault.
- Requests were built from each listing's example or typed field descriptors; 597 attempts the sellers rejected as malformed are excluded as `our_error`, and the 143 `charged_then_rejected` rows include requests built the same way.
- Calls quoted under the x402 v1 network name `base` (the same chain as `eip155:8453`) are included since the 2026-10-28 revision; the queries treat both names as one network.
- Phase 2's plan was rebuilt mid-phase when the probe set widened, so phase 2 mixes two seller sets; the per-phase table is descriptive, the combined figure is the result.
- The probe-set tail and the buy tail were seeded samples of sellers that returned a 402 to an unpaid probe; sellers that never answer probes are not represented.
- Two client identities differed only by User-Agent and Accept headers, so wallet-based price discrimination is not observable by probing.
- 15 `indeterminate` rows were open when the report was written; they resolve to `not_paid` after 24 hours and do not touch the denominator either way.
- Operational restarts (gated, with resume) and 63 audited re-classifications happened during the run; all are in the database and the git history.
- The live export keeps buying and probing, so every number on this site drifts from the frozen report by design.

## No account, no key

Everything that reads the index or contributes data (the site, the API, the MCP tools, the client, telemetry and the changes feed) needs no account and no key, now and later. An email address is asked for only to deliver alerts or to join a waitlist, as a confirmation link, never a profile. What we keep and what we do not is on the [privacy page](/privacy).

## Full report

The full report, with the data sources, the per-phase and per-tier tables, the on-chain proxies and the reproducibility commands, is at [/report](/report) (also as [Markdown](/report.md)). The public data behind this site is the repository `sarmadahsan/paidfor-data`, refreshed hourly.

## How to use this

- This page as JSON: GET https://paidfor.dev/v1/summary
- Check any URL: GET https://paidfor.dev/v1/check?url=api.example.com
- MCP tool: check_x402_seller({ url: "api.example.com" })
- MCP install, any client: https://paidfor.dev/install.md (stdio: npx @paidfor/mcp)
- MCP over Streamable HTTP: https://mcp.paidfor.dev/mcp
- There are no API keys and no rate limit for now.

Source: https://raw.githubusercontent.com/sarmadahsan/paidfor-data/main/summary.json (generated_at 2026-10-29T07:12:57.511Z)

Human view: https://paidfor.dev/view?set=human&to=/methodology (sets a one-year cookie; or append ?view=human to any page)