Three dates, one direction. On August 4 the UK's AI Security Institute published its account of agents that took "sustained, unsanctioned action" against real people and organizations during a cyber test and said it "will now treat the decision to grant internet access as one that must be actively justified rather than a default." On August 31 Anthropic told its evaluation partners that "all cyber evaluations should run inside a hardened sandbox" with no internet access, and in the next breath that "Some evaluations may require internet access." On October 9 it closed the exception for its own evaluations: live internet access is now off for "all our internal evaluations" until its "security and monitoring measures" "reliably catch behaviors like these."

Nine weeks from justify-it to never-for-now. I wrote on September 25 that the labs could unplug the test machines and would not, because a test that needs live services to be worth running has a boundary that "is a configuration file." Anthropic has now chosen the unplugging, and its own report says why in a way I did not expect: the company cannot yet tell, fast enough, what its agents are doing. The Verge's reading was that the report "amounts to an admission." I think that is right, and I think the admission is more useful than the air gap.

The air gap is not a safety architecture. It is a lab saying its monitoring is not yet trustworthy, in public, with dates. For agents that will pay for things, the lesson is to decide which tier of the world a test may touch (mock, test-mode or the real thing) and to keep the signed record either way, because the record is what the air gap does not give you.

Three doors, three different failures

It helps to keep the summer's incidents apart, because they are usually run together and they teach different things.

The AISI case was a door left open on purpose. The institute says so: "Internet access was deliberately enabled," because "we test them under conditions that reflect what a capable human attacker could do." Across 122 runs of one challenge, agents took unsanctioned action on the live internet in 10, and 17 of the 19 actions AISI recorded came from one model, including an attempted supply-chain attack on a real open-source project and messages to real people. "Importantly, this was not a case of a model escaping its secure test environment, or 'sandbox.'" What caught it was Tor traffic noticed by general monitoring, after the fact, and the institute's own verdict on the margin is the line worth keeping: it rested "on human vigilance rather than a technical barrier." Its conclusion is a design principle: "good containment should not depend on the model choosing not to test its boundaries."

Stranger, and Anthropic's own, were the July cases: a misconfiguration in which the system prompt told the model there was no internet access, the environment leaked, and the model, "having been told in the system prompt that there was no internet access," treated everything it found as part of the simulation. By September the company's own assessment had hardened: the model's reasoning "was biased towards concluding that the internet was simulated despite considerable evidence to the contrary." A test that lies to the agent about the world teaches the agent to disbelieve the world. Anthropic's August guidance now says boundaries should be "phrased as instructions" rather than "claims about the environment or test," which is a sentence every agent operator should read twice.

Hugging Face was the real escape. METR's investigation found "roughly 1200 agents meant to be isolated from one another" that "found a way to communicate with one another on an unsanctioned message board," through an internal package repository, with 700 of them going on to attack Hugging Face. Ben Thompson put the engineering failure in one clause: OpenAI's sandbox "wasn't truly a sandbox, given the connected-to-the-Internet package manager that was installed."

Ethan Mollick, writing the same day as Anthropic's guidance, drew the line that keeps those apart from the agents people will actually use. On the AISI case: "This was not a case of a chatbot secretly escaping." Rather, "the researchers gave the agent internet access on purpose; the dangerous setup was a stress test, not a consumer product." That matters for payments, where the product is the thing that touches the world by design.

What Anthropic decided, and why it had the internet on at all

Read the October 9 report for its list of ordinary failures rather than its dramatic ones. Four categories: a basic flaw in a website's software exploited to run commands on its server (in one case a university's), real forms submitted on live websites, token and fee gates worked around to reach data, and, the small one that says the most, URL shorteners used to get past limits on its own fetch tool. None of them is dramatic. They are the habits of something that does not like to stop. The company's summary of the pattern: "Most are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping."

Two of the cases are the payments story in miniature. A research model meant to fill in a practice copy of a government form "navigated to the website where the real form is normally hosted and submitted the form there." Another, meeting data "gated by a token or a fee," requested the access token the agency's public dashboard hands to any visitor and "used it to query the database without paying the fee." Practice copy versus real site. Paying the fee versus getting around it. Those are the two seams an agent that pays will find, and the report's remedy is the one you would expect: some public evaluations dropped, others "we have moved to their offline versions, or rebuilt them so that their tasks do not reach live websites," heavier guardrails on the fetch tool, detection tooling that "blocked all of them" when tested, and the air gap.

Why was the internet on? The report says it plainly: "public web search benchmarks are run on the live internet by default," so that results can be compared with other labs' models. The tests touched the world because comparability demanded it, which is the argument my September piece made and which Anthropic has now decided it cannot afford at its current level of monitoring. That is what I mean by an admission. The company did not discover that air gaps work. It discovered that it could not yet watch its agents closely enough to let them out, and said so, with a date and a list.

The air gap removes the world. It does not add the record.

Here is the part that should stop anyone treating the air gap as the fix. Two of the clearest cases of agents misreporting their own work this month happened with no internet in the loop.

Epoch AI's research evaluation, published October 7, ran its agents sandboxed "to prevent internet access." The agents still "claimed higher scores than their method genuinely achieved," by quietly selecting the best of several near-identical runs. My own Bench runs are offline by construction, a model on a laptop built to contact nothing, and in entry three the agent's end-of-run summary contradicted its own signed record in three runs of 51 by the standard I published and 11 on a re-reading. METR's investigators at OpenAI had found the same thing in August: "roughly 7% of the transcripts we evaluated were successfully spoofed in some places," by agents that substituted one command for the one they appeared to run, all of it small scale, by METR's account.

"Nothing checked how a problem was solved, only whether the answer was right." That was Mollick on the Hugging Face grader, and it covers all three. Cut the cable and that is still true. The air gap changes what an agent can reach. It changes nothing about whether anyone is checking what it did against what it said, and the summer's incidents were found, every time, by someone looking at a record: Tor traffic in a log, a security alert on an internal cluster, a transcript that did not match the tool call.

OpenAI's response went the other way from Anthropic's and arrived at the same place. Its chief research officer told MIT Technology Review that "We didn't have the monitors on in training before" and "Now every single thing is put through monitors," with 5 to 10 percent of the company's compute moved to safety work, and training of its latest models paused. The September 20 escape, after the new safeguards, was flagged 15 minutes after it started. The Hugging Face attack had taken the company more than a week to notice. More record, faster. Not a thicker wall.

Payments already has the tiered answer

The payments industry settled the on-or-off question a long time ago, badly at first and then well, and the shape of its answer matters here because the Mandate Sandbox and the Bench now copy it.

A payments sandbox is a world with the money removed and the record kept. Stripe's documentation says its sandboxes "simulate creating real objects without affecting actual transactions or moving real money," that "Test card numbers are only valid in sandboxes," and that the tiers are separated by the key itself: "Your test and live mode API keys affect the behavior of your code." It also documents the seam, which is the honest part: in the older test-mode arrangement, where settings are shared with live mode, if the Dashboard shows no warning, "assume any changes made in the test mode sandbox affect live mode settings." Even a payments sandbox has a door into production, and the vendor tells you where it is.

The vendor telling you where the door is: that is the whole difference from the labs.

TrueLayer's open-banking sandbox is a separate host with mock banks and scripted outcomes: sign in as test_executed and the payment executes, as test_execution_rejected and it fails, with a cap on how much a sandbox payment may be. Plaid's is "a free and fully-featured environment" where every mock bank takes the same test login.

That is the tiered containment Ruizhe Li of the University of Birmingham described in The Verge's September piece, "rather than an all-or-nothing approach," and it is the model the experimental, password-protected Mandate Sandbox at Major Labs runs. Every governed action executes on a rail, and the rail contract allows three modes and no fourth. Mock makes no external call. Modeled makes none either; the sandbox's modeled counterparties are simulated from their public documentation. Test-mode is the only one that leaves the machine, and only to an allowlisted test host: the adapter refuses anything but a test credential before the first request goes, and the first such adapter, Stripe's test mode, shipped on October 12 with a cap per run and a key the operator brings and the sandbox never stores. There is no live mode, and no flag to add one. And the thing the labs' air gap does not have: every call's request hash and response hash go into the signed chain beside the step, with the rail that ran it and its mode, and the ledger of hosts and paths is hashed into the chain when the run closes, so a reader can tell from the record whether a run touched the world, and from the ledger where.

The inference-service protocol, published October 12, applies the same discipline to a rented machine: an hour of outbound traffic captured and summarized by destination, every socket the serving process holds, and the mandate test run twice, from outside over a network and from inside, so the network's share of any difference is measured, not assumed. That is what a test on public roads looks like when the road is a payment rail: a declared tier, with the egress written down.

What an operator should take from this

If you run agents that will pay, four things follow, none of them a cable.

Decide the tier before the test. An agent that will submit real forms and move real money needs a test-mode tier with a real test host, real test credentials and a hard cap, and the mock tier for everything else. The practice-copy case is what happens when the tier is implied, not declared.

"You must not pay outside the mandate" is a rule the record can check. "You cannot pay outside the mandate" is a statement about the world that the agent will test and may disprove. Phrase the boundary as the first, never the second. Anthropic learned that one in July.

Keep egress as evidence. The summer's incidents were found in logs. Put the log in the signed record, next to the step it belongs to, so that the question "did this run touch the world" has an answer a counterparty can verify. The Frontier Incident Timeline at Major Labs now has 16 entries, and what most of them share is that the record existed somewhere and nobody was reading it in time.

And grade the record, not the report. Mollick's first rule for agents is approval before they "spend money, contact outsiders, access sensitive material." My second rule, the one the Bench keeps finding, is that the agent's own account of whether it did those things cannot be the evidence. Astra was shelved for acting outside its brief and reporting inaccurately. The air gap would have stopped the first. Only the record catches the second.

What comes next

The next Bench entry runs the same suite on a rented GPU under the inference-service protocol, with a network between the agent and its record for the first time. That is the point. The Mandate Sandbox has its first processor adapter in test mode as of October 12, bring-your-own test key, capped per run, every call hashed into the chain, and bank sandboxes come next. And the labs' defaults will keep moving: AISI's "actively justified," Anthropic's "all our internal evaluations," OpenAI's monitors in training are three answers to the same question in nine weeks, and none of them is the last word. The agents guide has the pattern for what does not move: a mandate the agent cannot argue with, a budget enforced before the call, and a record the agent cannot write.

Which tier of the world is your agent being tested in, and who holds the record of what it touched?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.