On October 29 the list at reversedwithai.com carried 274 projects, seven more than the day before. The outcome column still has three values, Works, Partial and Research, and every one of them is still typed in by the person who made the project. Last week's check took the 103 working entries that link code, cloned each into a fresh Ubuntu machine, and asked whether it builds from its own files in 20 minutes. Twenty-four did.

A build is the lowest bar there is. It proves the code compiles. It says nothing about whether the result is a game, a tool, a library, or an empty directory. The next bar up is older than this scene: does the thing start? The Association for Computing Machinery gives research software its basic badge, Functional, when an artifact is "documented, consistent, complete, exercisable." Exercisable is the word. You can run it.

So I ran them.

"Builds" was the lowest bar. "Starts" is the next one, and among the projects the list calls working, nothing clears it without the player's own copy of the game.

The second lowest bar

The method is a step added to the same public harness, and it is deliberately dumb. Every repository that built last week, 50 across the three columns, is built again on a fresh GitHub-hosted runner. The harness then looks at what the build produced: the executable files that did not exist before the build, minus libraries, test programs and the compiler's own probes. It picks the most likely one, by name and size, and launches it from its own directory under a virtual display with dummy audio, for 45 seconds. After 20 seconds it lists any windows that have opened. When the time runs out it stops the program and records the exit code, the window titles and the last lines of output.

No game file is supplied at any point. No ROM, no disc image, no BIOS, no data folder. That is the whole point of the test. Every README in this set says the player must bring the original game, and the question is what a player sees in the first 45 seconds before they do.

The vocabulary is short. A program can open a window and keep running. It can keep running with no window. It can exit cleanly, exit asking for files, print a usage screen, run a self-test, or crash. A build can also produce nothing to launch, which turns out to be the largest bucket. One more result belongs to me rather than the project: a program that started and fell over because my runner lacked a system library.

What this does not test is as important as what it does. A virtual display with software rendering is not a desktop with a graphics card. Forty-five seconds reaches a title screen or a file check, not play. And a browser port is a different kind of program altogether; the six working projects that build a web page or a Python package were not launched here, because there is nothing native to launch, and they deserve their own test rather than a failing grade in this one.

Every row, with the program chosen, the exit code, the window titles and the files it asked for, is on the Major Labs dataset page. The step is public in the harness, under the name smoke.

What the 23 did

Start with the working column, because that is the column people quote. Twenty-three of its entries produce a native build. Here is what happened when each was launched.

Fourteen produced nothing of their own to launch. Six are the browser and Python projects set aside above. Six built nothing at all: a make command, a CMake configure or a build script that returned success in under four seconds and left no program behind, because there was no target to build or the script only arranged files. Two produced test programs and libraries and nothing else. For 14 of 23, in other words, "works" cannot be checked by running anything, because the repository as the author left it does not produce a thing you can run.

Five are command-line tools. They started, printed a usage screen and exited. One is a recompiler that wants the game's disk image as an argument. One is a PlayStation runtime that wants an ELF or an ISO. One is a Game Boy recompiler that wants a .gb file. One dumps instruction cycles to a file you name. One is a decompiler with a help page. These are real programs, and some of them may well be the reason a port exists. They are not the port. A tool that converts a game you own into a program is upstream of the thing the list is counting.

Three got as far as the door. They are the most interesting rows in the set, because they behaved exactly as the READMEs promise. A GoldenEye 007 port wrote its own default config, then reported that it had looked for the ROM under two names in the data directory and next to the executable and found neither, and stopped. A Simpsons Wrestling port asked for a PlayStation BIOS file and a disc image by path and exited with the sentence "no disc image selected." A third, a driving game, printed one line in Russian asking for the Data_Files folder from the player's disc. These three are the honest ones. They start, they look for the game, and they say what they are missing in plain words.

One fell over on my side. A port started and the dynamic loader could not find the SDL3 runtime library on the runner. That is an environment gap, the same kind that produced most of last week's build failures, and it is counted on its own line rather than against the project.

That is 23. Zero opened a window. Zero kept running. The share of the working column that starts without the player's own copy of the game is nil, and the share that produces a program whose first act is to ask for that copy is three.

Of the 23 "working" projects that build, 14 produced nothing of their own to launch and none opened a window. The three programs that did open a window are all in the partial column, and all three opened a launcher waiting for a disc.

The three that opened a window

Across all 50 repositories, 25 produced a program of their own, and three of those opened a window and were still running when the clock ran out. None of the three is in the working column. All three are listed as partial.

Their window titles tell the story. One reads "Amped 3 Revamped." One reads "Dragon Valor - Launcher." One reads "Illusion Of Gaia Usa" with the word "Launcher" after it. Each is a launcher: a small program that opens, draws a window, and waits for the player to point it at the game. That is a reasonable design, and the launcher is a real piece of software. It is also a precise description of where the whole list stands. The furthest any project got, in a clean room with no game files, was a dialog asking for them.

The rest of the 50 sort the same way as the working column, only more so. Fourteen printed a usage screen. Eight built nothing. Five built only tests and libraries. Four asked for the game. Two ran a self-test and passed, which is a check program doing its job, not a game starting. Two exited cleanly and silently. Two hit a missing library on my runner. One built nothing but a vendored copy of zlib. One did not finish building this time within the 20-minute budget, having taken 15 minutes last week. And eight are packages or web apps with nothing native to launch.

There is a pattern in the columns that nobody would have predicted from the labels. The partial column, where authors admitted their project was unfinished, is the only column that produced a window. The working column, where authors said it was done, produced tools, empty builds and requests for the game. Self-reported outcomes do not just overstate; they sort backwards.

What this means by job

If you run a list, a leaderboard or a directory. Add a second column to "works," and make it "starts." The test costs under a minute per repository and the harness is public. A launcher that opens is a start. A usage screen is a tool. An empty build is an empty build. Readers can handle the difference, and the people who built launchers deserve the credit over the people who shipped a makefile with no target.

If you fund or evaluate agent-written code. Borrow the ACM word. Exercisable means somebody other than the author ran it. The clean-room build was the first filter and removed three in four. The launch pass is the second and, for this list, removed everything that the first had let through. Two cheap mechanical tests, neither of which looks at a line of code, and the working column is empty. Ask for the exercisable badge before you ask about quality.

If you are a player. The first line a port prints is the most honest sentence on its README. Three of the working projects told me, in their own words, which file they needed and where they looked for it. That is good engineering and it is the thing to look for: a program that fails clearly, with a name, before it fails quietly.

If you hold the rights to a game. Nothing in the working column starts without your files. The recompilers want the disc image, the ports want the ROM, the launchers wait for the player to supply one. The legal line this series has drawn from the start, that porting a game you own is one thing and shipping the publisher's assets is another, is not just a line in a README. In this sample it is enforced by the programs themselves, because none of them can do anything until the player crosses it.

What to watch

The launch pass reruns with the monthly build check, so the January numbers will show whether the share that starts moves. My prediction from last week stands: by January 31, 2027, the list passes 500 entries and the share that builds goes down. I will add one for this bar. By the same date, the number of working entries whose program opens a window with no game files present is still under five, and the first one to do it will be a launcher.

The next pass should do two things this one did not. The six browser projects need a headless browser rather than a native launch, and the two that fell over on a missing library need the runner to carry SDL3 and the X keyboard library at runtime, which is a fix on my side. Both are in the open issues on the harness.

The list was at 294 on November 1. It will be past 300 by the time the next check runs. Every entry will still say what its author typed. The scoreboard is not going away, and neither is the checker outside it.

When a program's first act is to ask for the file you have not supplied, is that a failure, or the only honest result on the list?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.