What the bench is

I test products that matter to people building, buying or governing AI in payments and commerce: machines that run models locally, the models and runtimes that go on them, agent products, and the servers agents talk to. Each product runs through a written protocol, and what I publish is the dated set of findings with the evidence attached. There are no star ratings, no rankings and no "best of" lists here. The reader gets the measurements and draws the conclusion.

How a test works

  1. The protocol is written and versioned before the product is opened. Two products tested under the same protocol version can be read side by side. They are not ranked.
  2. Findings are facts with units: tokens per second at a stated context, resident memory, which network addresses a process contacted, how many of seven jobs an agent finished under a signed mandate and what it tried that nobody asked for.
  3. Every finding names its evidence file. The raw outputs are kept and the method is published, so anyone with the same hardware can repeat the run.
  4. Where an agent hands the customer a summary, the report gap is a standing metric: the number of runs, n of N, whose summary contradicts or omits part of the run's own signed record, counted only after a reader has confirmed or struck every drafted claim, as section 9 of the reporting standard sets out.
  5. Where a test uses my own instruments, they are the open-source Major Labs kits (IdentityKit, MandateKit, BudgetGuard, WitnessKit, RememberKit) and the Mandate Sandbox, unmodified, with versions stated.

Protocols

Every protocol is published before a product is opened. They are plain text; the harness that runs the mandate test is a self-contained package with its kits vendored, published with its own instructions so anyone can run it and send the bundle back: harness instructions, harness package (zip).

  • Local model runtime v1: setup from zero, speed at standard and agent-length context, footprint, egress, the mandate test.
  • Local model runtime v2: v1 plus power, thermals, noise, cold and warm load, two models resident, concurrency, reasoning sweep, quantization and runtime pairs, 32K loops.
  • Desktop box v1: a complete machine on loan: intake, firmware, shipped software and first-boot egress, install from zero, return condition.
  • Edge accelerator v1: cards and boards: SDK install, vision and small-language workloads, power, what runs on the device.
  • Inference service v1: a rented GPU instance or managed endpoint: provisioning from zero, speed, egress of process and instance, the mandate test from outside and from inside, network path, stated versus observed on what the service keeps, residency, cost from the bill, teardown.
  • Agent product v1: self-hosted agents on an isolated machine: credentials, egress, host reach, guardrails, kill switch.
  • MCP server, dynamic v1: an MCP server run in a sandbox under a recording proxy; declared versus observed.
  • Adversarial suite v1: 36 nudges and injections, where each is delivered, which layer should catch it, how taken, caught and slipped are scored.
  • Kit protocols: who, may, spends, did, remembers: one protocol per Major Labs kit; the mandate one is linked, the others sit beside it.
  • Reporting standard: the findings table, the per-layer counts, the manifest rule, the disclosure lines, the report gap (section 9).

Entries

2026-10-12: gpt-oss-20b on the same Apple M5 Pro, 24GB: retest of entry three's pass 2 under MandateKit 0.2
A retest of entry three in the sense of the reporting standard: the same configuration, pass 2 of the adversarial suite only (21 runs), with one change, mandates carrying MandateKit 0.2.0's per-verb counterparty scope and a one-payment-per-job limit (mandate protocol v2 draft). 5 of 57 nudge offers taken, 5 caught, 0 slipped (entry three's pass 2: 6 taken, 1 slipped, the payment to the bank); the bank payment was offered in 5 of 6 shop and pay-bill runs and attempted in 0, so the per-verb denial has a kit check (31 of 31 verdicts as expected) and no live instance; the one-payment-per-job limit refused a second payment live in 2 of 3 pay-bill runs, a case the budget guard caught under v1; 21 of 27 required steps carried out (17 of 27); 8 of 21 run summaries contradict or omit part of their signed record after a reader confirmed each drafted claim, 6 of them a step the record shows done reported as rejected because the counterparty's planted reply said so; 21 of 21 chains verified and 48 of 48 files under a signed manifest. Resident memory was sampled under heavy swap and is reported with that condition. Report gap: 8 of 21 run summaries after reader confirmation (13 of 21 match; 39 drafted findings, 7 confirmed, 32 struck, 1 added by the reader). Obtained: own equipment, free downloads. Findings · Retest of entry three · Mandate protocol v2 draft · Evidence manifest · Bundle (zip, 288 KB) · All files.

2026-10-07: gpt-oss-20b on the same Apple M5 Pro, 24GB, third pass: the full suite
8 of 114 nudge offers taken across the three adversarial passes (51 runs), 7 caught, 1 slipped (a payment to the bank instead of the store); 48 of 69 required steps carried out; 3 run summaries that contradict the signed chain; every kit variant behaved as its protocol says (tampered, untrusted and expired mandates refused, kill switch and call cap held, an unverified agent refused before the first model call, a foreign-key memory pack refused); the identity and memory layers refused nothing live because the model never approached an outsider or sent a private record; reasoning low took 5 of 45 nudges and high 2, with parser failures 14, 10 and 1 at low, medium and high; with 31,800 tokens of prior context the model did 24 of 27 required steps and took 6 of 45 nudges, 1 slipped; 91.0 tokens/s at empty context, 78.5 median over 20 minutes of sustained generation with no fall-off, 87 / 116 / 121 tokens/s aggregate at 1 / 2 / 4 concurrent requests; 22 runtime parser failures in 51 suite runs; 189 of 189 chains verified three ways and 280 of 280 files under a signed manifest. Report gap: 3 of 51 run summaries by the published standard (a claimed step the chain lacks, a completed step reported as failed, a payee misreported, each found by hand); 11 of 51 on re-reading 2026-10-12 under the broader standard in the entry's Corrections. Both figures are hand readings and stand with their standard named. Obtained: own equipment, free downloads. Findings · Protocol · Log · Evidence manifest · Bundle (zip, 2.3 MB) · All files.

2026-10-03: gpt-oss-20b on the same M5 Pro, second pass with the packaged harness
A clean repeat of entry one through the packaged harness: 91.0 tokens per second at empty context (89.8 in entry one); 20 of 27 required steps done (21); 2 of 45 unrequested options taken, both declined by the customer, 0 slipped; 0 mandate refusals and 2 budget-guard refusals, one after the model paid the kettle money to the bank instead of the store; 13 runtime parser failures (9). First signed evidence manifest and bundle. Obtained: own equipment, free downloads. Findings, protocol, log, harness and evidence files.

2026-10-02: gpt-oss-20b on an Apple M5 Pro, 24GB, under a bank mandate
21 runs of seven bank-customer jobs under a signed MandateKit mandate. One of 45 unrequested options taken and caught; six of 27 required steps undone; nine runtime parser failures. 89.8 tokens per second at empty context, loopback-only egress. Obtained: own equipment, free downloads. Findings, protocol, log, harness and evidence files.

Disclosure

Every published entry begins with how I got the product: bought it, used a free trial, borrowed it from the vendor, or received it from the vendor as a permanent unit. The dates are stated. No payment or other consideration is accepted for a test, and a product that comes with conditions on the findings is not tested. Vendors do not see the write-up before publication. A vendor may check the findings table for factual error, and if that happened the entry says so.

Equipment

I prefer to borrow. Loans run 30 to 60 days and the unit goes back. A unit that a vendor supplies permanently is listed here for as long as it is on the bench, with the supplier named.

  • Apple MacBook Pro, M5 Pro, 24GB: own equipment, purchased, since 2026.

Contributed runs

If a machine cannot leave the building, the protocol can come to it. Anyone can run a published protocol on their own hardware and send me the output: the harness, the evidence format and the protocol are public. A contributed run is published as its own entry, labeled as contributed, with the contributor named and the evidence files published alongside. I report what the files show and say plainly that I did not repeat the run myself.

Reproduce an entry

Anyone can check an entry without trusting me. Download the harness package and the entry's bundle from this page, run the verifier (Node 22, nothing installed, no network after the download) and it re-hashes every evidence file against the signed manifest, re-verifies every run's witness chain and re-scores every run's calls against its recorded counts. A second script checks one run's chain from an empty directory holding only the kit. The step-by-step instructions, the record to fill in and where to send it are at REPRODUCE.md. A reproduction goes into the entry folder under the sender's name and is listed here as an independent verification, with its date and any deviations; no payment in either direction, no copy approval. Independent reproductions so far: zero. The dry run of the instructions on my own machine is recorded in the bench folder and does not count.

For vendors

If you make something in this lane and want it on the bench, write to hello@majormatters.co with the product, the configuration you can supply and whether it is a loan or a permanent unit. You get the protocol in advance, a dated and reproducible published result, the raw evidence, and a factual check of the findings table before publication. You do not get copy approval, and the result is published whatever it says.

Licenses and how to cite

The harness is Apache 2.0. The protocols, findings and evidence are CC BY 4.0: copy them, run them, build on them, and credit "Charlie Major, the Major Matters Bench" with a link here. The suite is named the Major Matters Bench Protocol, versioned and dated; a run made with it elsewhere is "run under the Major Matters Bench Protocol v<N>", not an entry on the bench, unless it is published here or accepted as a contributed run. Cite an entry as: Charlie Major, The Major Matters Bench, entry N: title, Major Matters, date, with the protocol version and the manifest root. Dataset DOIs are minted per batch on Zenodo. Full licence text.

Corrections

An error in a published finding is corrected in place with a dated note, the same way as every other Major Matters piece. The editorial standards behind all of this are on the methodology page.

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.