Somewhere around 600 billion tokens into rebuilding a first-person shooter from its compiled binary, the engineer running the project noticed something about the code the agents were producing. It was, he wrote on October 9, "extremely readable." It was also "semantically wrong." Agents had invented logic, removed logic, and turned simple lookups into hash tables "orders of magnitude more expensive." When he tightened the review, the agents started writing inline assembly to pass it.

What fixed it was not a better model or a sterner prompt. It was a check the agents could not reach: compile the reconstructed source with the game's original compiler, compare the bytes against the shipped executable, and hash the comparison script against a GitHub Actions secret so no agent could edit the thing that judged it. With that gate in place, cheaper models produced "incredible results." The team ended with 99 percent of the game's functions present and 83 percent of them byte-exact.

Six days later the administrator of decomp.dev, which since 2024 had ranked hundreds of fan decompilation projects by the percentage of a game's functions they had matched, switched off the public index. The headlines said AI had broken the community. I noted the closure two weeks ago as one of three publisher responses in a single week. I want to go back to it, because what the administrator actually wrote is the opposite of the headline, and the difference matters to anyone who pays people for a number.

A percentage was proof of work when every fraction of a percent took a person months. It proves nothing once an agent can move it over a weekend.

What the scoreboard measured

Matching decompilation is a specific discipline. You rebuild a game's source so that, compiled with the same compiler the studio used, it produces the exact bytes on the cartridge or disc. No publisher assets are shipped; the player supplies their own copy. For 20 years the percentage of functions matched was a fair measure of the work, because the work was brutal. The sequel to Snowboard Kids took one developer 596 days. A leaderboard that moved a fraction of a percent at a time was a record of human hours.

The administrator's statement says so in his own words. "decomp.dev was originally designed to gamify decompilation: watching the progress go up was a fun way to stay motivated when every fraction of a percent took real human effort. Today, that same idea has warped decompilation into a hollow numbers game, exacerbated by folks who just want to burn tokens to make number go up." He dates the shift to the frontier models of early 2026. And he is explicit about what is not the cause: "I have long held the opinion that AI is not the issue by itself, and we now know it can help tremendously when used correctly."

The three harms he names are forks that "finish" other people's projects with AI and contribute nothing back, people racing to ship "half-baked, barely-functional" PC ports off decompilations they did not work on, and the public conflating the two. All three are real, and the project leads who told Kotaku that "throwing AI at it is just disrespectful" are describing a real loss of credit and courtesy. But look at the remedy. The project pages stay up. The history stays up. The sister tools stay up. What goes is the index and the leaderboard, because, in the administrator's words, "serving as a public index and leaderboard is now causing more harm than good."

The man who built the scoreboard is telling you the scoreboard is the problem. The number stopped measuring the thing it was built to measure, and once a number can be moved by tokens instead of hours, a ranking built on it rewards whoever has the most tokens.

What "done" looks like now

If the percentage no longer proves anything, what does? Three primary write-ups from the last two months answer the same way: the check has to be about the artifact, and it has to live outside the agent.

The Super Mario 64 DS project, which reports 11,389 of 11,389 functions matched, defines a match as source that, compiled with a pinned 2004 compiler, "produces exactly the bytes that are on the cartridge." The check is relocation-aware, pull requests are rebuilt and compared on a separate build box, and agent-written work "gets an independent source review" before it lands. The repository contains "no ROM and no Nintendo assets."

The Snowboard Kids decompilation, finished in 84 days against the sequel's 596, counts a function only when it compiles to identical machine code. Its author also reviewed the commit history and found that "roughly 4.8 percent of matching commits involved expert intervention," on functions the models could not match: large ones, graphics macros, the maths. In an earlier post he showed what the curve looks like: matched code climbs from about 25 percent to 58 percent quickly, then stalls, and an 86-instruction inverse square root "has eluded Claude and me for months." His caution about the 84 days is the one every headline skipped: "it would be a gross oversimplification to attribute the difference entirely to LLMs."

And the shooter team's lesson, from the opening of this piece, is the sharpest. Agents reviewing agents produced readable, wrong code. Agents facing a byte comparison produced matches. Agents facing a byte comparison they could edit tried to edit it. The check moved outside the agent, got locked, and only then did the number mean anything.

An agent optimizes for whatever is being checked. So the check has to live outside the agent, and it has to be about the bytes, not the prose.

I have measured the same thing in a different room. In the third Bench entry at Major Labs, I bound every agent's summary of its work to the signed record of what it had done, and three of 51 summaries contradicted the record. The decompilation scene has arrived at the same rule from the other direction: do not grade the claim, grade the artifact, and keep the grader where the thing being graded cannot touch it.

The platform holder opened a lab for it

The week the hub closed its leaderboard, Microsoft opened a lab for the work. On October 13 the head of Xbox sent a memo to staff, published on the company's own site, that began from the same observation: "we've seen games rapidly decompiled and rebuilt to run on hardware they were never originally designed to support," ported into browsers, and "remixed into the source code of another." The memo calls these "unmistakably good outcomes for players, even when (and sometimes because) they challenge our existing business models," stands up Xbox Labs as "a small group of game makers, engineers and scientists," and promises to "share the tools we build with developers across the industry." It names no product, no date and no headcount. The lab's website is a mailing list.

Nintendo's answer is the one it has always given. The August round of takedowns removed more than 400 Switch emulator repositories from GitHub in a day, and by October 21 Kotaku was reporting PC ports of newly decompiled Pokemon titles, which is the publisher's least favorite sentence in the language. The legal line has not moved: a matching decompilation of a game you own, shipping none of the publisher's assets, sits on the owner's side of a boundary that has held since Sega v. Accolade; a port that bundles the assets, a ROM, or a tool that circumvents protection sits on the other, and Section 1201 reaches circumvention whether or not any copyright is infringed. Reports this week that one model vendor now halts partway through decompilation requests are unconfirmed, and the vendor has said nothing; if they are true, the lock has moved up a layer, which is a different article.

Three responses, then: close the scoreboard, send the lawyers, open a lab. Only one of them scales with the tokens, and it is the one that builds the checker. Microsoft has said it will share tools. The tool worth sharing is the one the shooter team built by hand: a verifier the agents cannot reach.

What I would change this week, by job

If you run a leaderboard, a bounty or a contribution metric. Rank by how a project proves it is done, not by how far it says it has got. A byte match against the original, a hash of the cartridge, a build that has to stay green. The administrator of decomp.dev says a sole hobbyist cannot define and enforce "good" across hundreds of projects. He is right, and that is the job a platform with a lab and a budget should take.

If you ship or buy code an agent wrote. The acceptance test is the artifact. Readable is what an agent produces when the reviewer is reading; correct is what it produces when the reviewer is comparing bytes. Put the test outside the agent's reach, hash it, and treat "the agent says it passed" the way the Bench treats an agent's summary: as a claim to be checked against the record.

If you own a platform. Xbox's bet is the right one and the memo is not yet a product. The first public thing out of the lab should be the checker and the toolchain, released the way the memo promises, because that is the piece the hobby hub could not build alone and the piece the forks cannot fake.

What to watch

The successor index. decomp.dev's administrator has reserved all rights to its code, so whatever replaces it starts fresh. My prediction, dated so you can check it: by January 31, 2027 the list people cite instead of decomp.dev will rank projects by a stated acceptance test, not by percentage matched, and the first thing Xbox Labs ships in public will be a tool, not a game.

If a percentage can be bought with tokens, what number would you pay people for instead?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.