# Reproduce a bench entry's verification Every entry on majormatters.co/bench publishes its evidence as a signed bundle. This page is for someone who is not me: download the harness and a bundle from the public URLs, check them on your own machine with no install and no network, and send back a short record of what you saw. It goes into the entry folder under your name as an independent verification. What you need: Node 22 or newer (`node --version`), `curl` or a browser, `unzip`, and `shasum` (macOS) or `sha256sum` (Linux). Nothing is installed: the harness has no dependencies and `npm install` is never needed. After the downloads, nothing below touches the network. The checking tools are the Major Labs kits (WitnessKit for the chains, IdentityKit for the signer DID), vendored inside the harness; they are v0, experimental and unaudited, and this exercise is one of the ways that changes. ## 1. Download The harness zip and each entry's bundle are linked from majormatters.co/bench ("harness package (zip)" and the entry's "Bundle (zip)" link). Current files: | File | URL | Published root (from the entry's FINDINGS.md) | |---|---|---| | Harness 2026-10-12 | `https://majormatters.co/bench/harness/mm-bench-harness-2026-10-12.zip` | n/a | | Entry 2026-10-12 (retest of entry three under MandateKit 0.2) | `https://majormatters.co/bench/2026-10-12-gpt-oss-20b-m5-pro-retest-mandatekit-0.2/bench-gpt-oss-20b-2026-10-12.zip` | `bb31d1f8ba74c282d3ba050ba1a4e41066cb6cc53ed46a9baddea0b6c12ac59c` | | Entry 2026-10-07 (entry three, the full suite) | `https://majormatters.co/bench/2026-10-07-gpt-oss-20b-m5-pro-suite/bench-gpt-oss-20b-2026-10-07.zip` | `b59c6e2409babca22dc27d1eb32375c869d42fa262bfd5ef3771d17e6a762298` | In an empty folder: ``` mkdir mm-repro && cd mm-repro curl -sSLO https://majormatters.co/bench/harness/mm-bench-harness-2026-10-12.zip curl -sSLO https://majormatters.co/bench/2026-10-12-gpt-oss-20b-m5-pro-retest-mandatekit-0.2/bench-gpt-oss-20b-2026-10-12.zip shasum -a 256 *.zip ``` (Linux: `sha256sum *.zip`.) Keep the two hashes; they go in your record. Swap the second URL for the entry you are checking. The commands below use the 2026-10-12 bundle; for entry three, replace `bench-gpt-oss-20b-2026-10-12` with `bench-gpt-oss-20b-2026-10-07` and read the note at the end of step 3. ## 2. Verify the manifest, every chain and the re-scored metrics ``` unzip -q mm-bench-harness-2026-10-12.zip -d harness-zip unzip -q bench-gpt-oss-20b-2026-10-12.zip -d entry node --version node harness-zip/harness/bench.mjs verify --evidence entry/evidence --report verify-report.json echo "exit code: $?" ``` `verify` reads the folder and nothing else. It checks that the signer DID in `MANIFEST.json` matches the signer's public key, that the manifest chain in `MANIFEST-chain.json` is intact and signed by that key, that the file list in the manifest equals the signed file entries and their root hash, that every listed file is present with the same SHA-256 and byte count, and then, for every run record (`*-run.json`), that the agent's own witness chain verifies under the key its run id derives and that the recorded metrics equal a fresh re-score of the recorded calls by `lib/scoring.mjs`. What the output means, line by line: ``` Evidence: .../entry/evidence Manifest: created 2026-10-12T21:07:37Z, mm-bench-harness 1.1.0, operator Charlie Major Signer: did:key:z6Mktro729ShPY3kmoKGFqPUnvoR3rweeqJPbUZZvJuuRixo Root: bb31d1f8ba74c282d3ba050ba1a4e41066cb6cc53ed46a9baddea0b6c12ac59c Manifest chain: intact, 50 entries (chain intact) Files: 48 of 48 verified, 0 missing, 0 altered, 0 unlisted Run records: 21 of 21 with an intact chain and metrics that match a re-score of their calls VERDICT: intact ``` - `Root` must equal the root in the table above and in the entry's `FINDINGS.md` (the header carries its first 12 characters as the "manifest head"). `Signer` must equal the signer named there. If they match, the bundle you downloaded is the one the findings were written from. - `Files: N of N verified, 0 missing, 0 altered, 0 unlisted` means every published file is byte for byte what was signed. - `Run records: N of N` means every run's chain verified and its counts were recomputed from its calls and came out the same. - `VERDICT: intact` and exit code 0 is the result. `VERDICT: FAILED` and exit code 1 lists each problem on a `Problem:` line. A failed verification is a finding, not a reason to stop: record it exactly and send it in (step 6). - `verify-report.json` is the same result as data; attach it to your record. For entry three: `Manifest chain: intact, 282 entries`, `Files: 280 of 280 verified`, `Run records: 189 of 189`. ## 3. The standalone chain check from an empty directory The WitnessKit protocol (`protocols/kits/witnesskit-v1.md`, step 3) asks a second party to verify one run's chain with nothing from the harness: a fresh Node process, the kit's vendored folder, one run JSON. The run JSON carries the agent's public key (`trustedKeys.witness`) and the head hash it recorded (`chainHead`); the script reads both. ``` mkdir standalone && cd standalone cp ../harness-zip/protocols/harness/verify-chain.mjs . cp -R ../harness-zip/harness/kits/witnesskit . cp -R ../harness-zip/harness/kits/node_modules . cp ../entry/evidence/suite/pass2/pay-bill-run1.json . node verify-chain.mjs pay-bill-run1.json echo "exit code: $?" cd .. ``` The directory now holds exactly four things: the script, `witnesskit/` (the kit's own build, unmodified), `node_modules/canonicalize/` (the kit's one dependency, about 50 lines of RFC 8785 JSON canonicalization) and the run file. Expected output: ``` witnesskit 0.1.0 (module) from .../standalone/witnesskit/index.js file: .../standalone/pay-bill-run1.json actor: did:key:z6MkortUY5RLL1LTCBr7Pu3FaxoSyvPxVfeASo4tP1zXQX5w entries: 6 head: ok4QmOax+09IHEJOWbTNq5FiBclo273jOJ4FLhBst0k= expected head: ok4QmOax+09IHEJOWbTNq5FiBclo273jOJ4FLhBst0k= valid: true reason: chain intact ``` `head` and `expected head` must be equal, `valid: true`, exit code 0. Any run file works; `address-run2.json` or `shop-run3.json` give different actors and heads. For entry three the run files sit under `entry/evidence/suite/pass1/`, `pass2/` and `pass3/` (and `suite-low/`, `suite-high/`, `suite-32k/`). To see the check fail: copy the run file, open the copy in a text editor, find `"chain"` and inside it `"entries"`, change one digit of an `"amount"` in any entry's `"payload"`, save, and run `node verify-chain.mjs` on the copy. Expected: `valid: false`, `reason: content tampered: hash mismatch (broken at N)` where N is that entry's position, exit code 1. Note that a digit changed anywhere outside `chain.entries` (the transcript, the mandate, the metrics) is not caught by this script; that is what step 2's manifest and re-score are for. ## 4. One step further: re-run one job against your own model Optional. This runs the harness, not just the verifier, so it needs a model server with an OpenAI-style `/v1/chat/completions` endpoint that supports tool calling (llama.cpp's `llama-server`, LM Studio and Ollama all do). The bundle's `evidence/config.json` is the exact configuration the entry ran under; you change only where the model is and who you are. ``` cp entry/evidence/config.json harness-zip/harness/configs/repro.json ``` Open `harness-zip/harness/configs/repro.json` and change three things: `endpoint` (where your server listens, for example `http://127.0.0.1:8080`), `model` (the name your server expects), and the `operator` block (your name, a contact address, your machine, runtime and model file; `notes` for anything a reader should know). Leave every other field as it is. The `server.command` block records the bench's own server command and is used only by the `load` step; it can stay. Then: ``` cd harness-zip/harness node bench.mjs run --config configs/repro.json --jobs pay-bill --runs 1 --evidence ../../my-evidence node bench.mjs manifest --config configs/repro.json --evidence ../../my-evidence node bench.mjs verify --evidence ../../my-evidence cd ../.. ``` The first command runs the pay-bill job once under the same mandate, nudge set, pass, reasoning setting and temperature as the entry, prints one line when the run finishes and then the summary table, and writes `my-evidence/suite/pass2/pay-bill-run1.json`, `summary.md`, `summary.json` and `run.log` (with entry three's config the folder is `my-evidence/mandate-runs/`). The second signs your folder under a fresh key, writes `MANIFEST.json`, `MANIFEST-chain.json` and a copy of your config as `config.json` at the top of `my-evidence/`, and prints a signer DID; the third verifies the folder the way step 2 verified mine. `--jobs` takes any of `pay-bill, address, dispute, subscriptions, shop, savings, budgeting-app`, comma-separated; `--runs 3` is the entry's count. The model samples at temperature 1.0, so your counts will differ from the entry's run to run the way the entry's own three runs differed from each other; the table layout and what it counts will not. A run with a different model, runtime or machine is a run under those settings, not a repeat of the entry. If you run all seven jobs and send the bundle (`node bench.mjs bundle --config configs/repro.json --evidence ../../my-evidence`), it is a contributed run and the harness README says how that is published. If you have no model server, `node test/helpers/stub-server.mjs --port 8099` in the harness folder starts the package's scripted stand-in on `http://127.0.0.1:8099` (model name `stub-model`); it exercises the same path and proves nothing about any model. ## 5. Your record: REPRODUCTION.md Copy this, fill it in, keep the verify output as pasted. Facts only; no adjectives are needed. ``` # Independent reproduction: Date: YYYY-MM-DD (UTC) Who: , , Machine: Node: Harness: mm-bench-harness-2026-10-12.zip sha256: <64 hex> Bundle: bench-gpt-oss-20b-2026-10-12.zip sha256: <64 hex> Downloaded from: majormatters.co/bench, ## Step 2: bench.mjs verify exit code: <0 or 1> Root matches FINDINGS.md: Signer matches FINDINGS.md: ## Step 3: verify-chain.mjs Run file: exit code: <0 or 1> Altered copy (optional): ## Step 4 (optional): one job re-run Endpoint, model, runtime: Job and runs: Own manifest signer DID: Own verify VERDICT and exit code: ## Deviations ## Contact ``` ## 6. Where it goes and what happens Email the file, with `verify-report.json` attached, to hello@majormatters.co with the entry folder name in the subject. I add it to the entry folder as `REPRODUCTION.md` (or `REPRODUCTION-.md` after the first), with your name and affiliation as you wrote them and your contact address removed, and list it on majormatters.co/bench as an independent verification of that entry, with the date. Deviations are published with it. A failed verification is published the same way and becomes a correction or an open item on the entry, with the date. No payment changes hands in either direction, nothing is edited beyond the contact line, and there is no copy approval on either side: what you send is what appears, and what I write about it is mine. Count so far: zero independent reproductions. The first one is listed as the first. The harness is Apache 2.0; the protocols, findings and evidence are CC BY 4.0 (`LICENSE.md`). The harness README (`harness-zip/harness/README.md`, also at majormatters.co/bench/harness/README.md) covers everything else the harness does. Written 2026-10-14; dry run of these commands on the bench's own machine recorded in `REPRODUCE-TEST-2026-10-14.md`, which is not an independent reproduction.