The Only Thing a Measurement Lab Owns Is Whether You Can Check It

I ran my own security scanner against my own code this week, and it flagged three of my repositories. Every one of those flags was wrong. That was the most useful thing that happened all week, and I want to explain why, because it is the same reason for everything else Major Labs shipped.

Major Labs is the measurement side of my work. Where Major Matters reads the architecture against the announcement, Major Labs counts the agentic web: a weekly census of the Model Context Protocol server ecosystem, its security posture, its identity gaps. An independent lab has exactly one asset. Not access, not a model, not a headcount. It is whether a stranger can check the numbers. The moment they cannot, the lab is just another blog with a chart.

A measurement is only worth the trust someone else can place in it without asking me to vouch for it.

So this week I spent on the checking, not the claiming. Here is what that looked like in practice, and why each piece is the same idea wearing different clothes.

Dogfooding, and finding my own tool's blind spot

Major Labs publishes a static security scanner as a one-line GitHub Action, so that any maintainer can run the same check the population sweep runs. If I ship that, I had better pass it. So I ran it against the lab's own open-source primitives.

Three came back flagged. One was the scanner detecting its own pattern definitions, a regex string that describes what a risky call looks like, read as if it were the risky call. One was an outbound request that the code already guards against abuse, on the line directly above the one that got flagged. One was already annotated as reviewed, and the scanner ignored the annotation.

That last one was a real defect, in my tool, not their code. A security scanner that cannot be told "I checked this, it is fine" trains people to ignore it, which is worse than not scanning. So I fixed it: the scanner now honors a suppression comment the way every serious static analyzer does. The fix shipped because I pointed the tool at myself and did not like what I saw. That is dogfooding doing its actual job, which is to fail on you before it fails on a stranger.

Preregistering a result I do not yet have

Major Labs has been tracking whether high-risk MCP servers actually get fixed after their maintainers are told. The early signal is uncomfortable, and I am not going to publish it yet, because the honest version of that study needs a design fixed in advance.

This week I wrote that design down and committed it before sending the next round of outreach: which servers get contacted, in what order, measured how, with the analysis rules set before any results exist. A finding you can shape after you have seen the data is not a finding. Preregistration is the boring machinery that stops a researcher, including me, from quietly steering toward the interesting answer. It went into the public repository, timestamped, ahead of the work it governs.

Correcting in public, not in silence

While preparing that study I found that an earlier Major Labs report had overstated one number. It described a whole cohort as contacted when only the first batch had been. The scan figures were right; the framing was wrong.

The tempting fix is a silent edit. I published a dated correction on the live page instead, saying what the original claimed and what was actually true. A correction that hides itself is not a correction, it is a cover. The Frontier Incident Timeline exists because I think the labs building frontier AI should disclose their failures plainly; I do not get to hold myself to a lower standard than the one I am asking of them.

Making the data leave the building

The rest of the week was plumbing in the same spirit. The measurement series is now a citable public dataset with a permanent identifier, archived on the open research repository that CERN runs, so it can be cited the way a paper is cited rather than the way a tweet is. The incident timeline and the new agent-identity tracker both publish machine-readable feeds, so a researcher does not have to trust my summary when they can read the record. And the whole lab is now queryable over MCP itself: point an agent at the endpoint and the numbers arrive in its context, licensed for reuse, with a citation instruction attached.

None of that is a feature in the product sense. It is all the same move: reduce the amount of trust you have to extend to me before the numbers are useful to you.

Why this is the strategy, not the housekeeping

It would be faster to publish numbers and ask you to believe them. Plenty of the agentic-web commentary works exactly that way, and some of it is even right. But an independent voice that asks for trust it has not earned is spending a balance it does not have.

The bet Major Labs is making is that verifiability compounds and assertion does not. A number you can check, cite, and reproduce gets used, and every use is a small deposit. A number you have to take on faith gets a nod and is forgotten. So the unglamorous week, the dogfooding and the preregistration and the dated correction, is not the overhead around the work. It is the work.

The agentic web is being assembled far faster than anyone is measuring it. I would rather measure it slowly and be checkable than measure it fast and be believed.

If an independent lab's only real asset is that you can check its numbers, what is the first number you would try to check, and would you be able to?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.