When Amazon cut off Meta's Muse agent last week, it gave a reason that sounded like a rule: agents that buy on a customer's behalf "should operate openly and respect service provider decisions about whether or not to participate." I read that and wanted to know what the rule was at the other 99 largest retailers in America. Not what they said about AI in an interview. What their sites actually do when an agent knocks.

There was nowhere to look it up. No survey asked the sites. The consultancies publish "agent readiness" checklists that sell services. The robots.txt trackers count publishers, not stores. The standards bodies writing agent-identity specs had no count of how many merchants would use one. A decision that affects every shopper with an assistant, every company building one, and every retailer's revenue was being made, one site at a time, by a default nobody had read.

So I asked the sites directly. Ninety-two of them, six polite questions each, the same way every week.

The merchant does not know what its bot manager is refusing. The agent does not know the rules. The standards body does not know the demand. Somebody should keep the number.

Three questions, and why those three

The Merchant Agent-Readiness Index asks each storefront three things.

Can an agent read the rules? Every website can publish a small file called robots.txt that tells automated visitors what they may and may not do. It is the only door policy on the web that is written down in a standard place. If an agent that identifies itself cannot fetch that file, the site has no readable rules at all, whatever its executives believe.

Does the agent get a page? The scanner then asks for the homepage the way an agent would, without a browser, saying who it is. What comes back is the real policy: a page, a refusal, a challenge that only a human with a mouse can pass, or an empty shell that only fills in once a browser runs its scripts. This is the decision the bot-management layer makes on the merchant's behalf.

Is anything published for the agent to find? Three files that did not exist two years ago now tell an agent what a merchant intends: llms.txt, a plain-text guide written for AI systems; a Universal Commerce Protocol profile, which declares what an agent may do, down to checkout; and an A2A agent card. Their presence is intent, stated in a form a machine can read.

Rules, enforcement, intent. Those three are the whole door.

The method is deliberately dull. Six plain requests per site, 1.5 seconds apart, under a user agent that names the project and links to it. No login, no form, no cart, no pretending to be a browser or another company's bot, no retry after a refusal. The sample is the National Retail Federation's Top 100, minus the eight with no single storefront. Only aggregates are published; per-merchant rows stay private and nothing rates or ranks a retailer. It runs from a US cloud address every Saturday, because that is where agents live, and it is checked against other vantage points. Every step is on the page, so anyone with a laptop can reproduce every figure.

Two weeks in

The numbers first, then what they are worth.

A third of the 92 will not show a self-identified agent their robots.txt: 29 on September 20, 28 on September 26. Of the 63 or 64 that do, 16 name any AI crawler or agent at all. Two block a user-triggered agent outright, the kind that acts for a live person; Amazon is one of them, and it had published that before Muse existed.

Roughly half serve a readable homepage to a non-browser client: 50, then 49. The rest refuse, challenge, hand back a shell, or never answer. Sixteen publish an llms.txt. Three publish a UCP profile. None has an A2A card.

Now the part that makes the series worth anything. Between the two runs, 89 of 92 sites gave the same answer on the rules question and 89 of 92 on the homepage. The three that flipped are the noise floor: bot-management decisions are not fully deterministic, so one site's answer in one week means little. The aggregate is stable, and a stable measure is the only kind in which a change means something.

The limits, plainly. A refusal cannot tell you why. A timeout is reported as no answer, not a refusal, because a slow site and a deliberate stall look identical from outside. A site that refuses the scanner also cannot show it an llms.txt, so the file counts are undercounts. A UK home connection sees more refusals than a US data center, which is geography and not agents. And one week is worth nothing; the trend is the product. That is why the index will look thin for a while and why I am publishing it anyway.

Who this is for

I did not build this for payments people, though they will use it. Six readers have a stake in the number, and most of them have never had a number to look at.

The retailer. The door policy on your site was almost certainly set by a bot-management vendor's default, tuned for scrapers and card testers years before anyone said "agent." It is doing its job. It is also refusing your customer's assistant, and nobody on your team has read the rule because there is no rule to read. The index is a mirror. If your site is in the sample, the aggregate tells you where your defaults sit against your peers; if it is not, the method tells you how to check in ten minutes.

The agent builder. If you are Meta, you can negotiate. If you are a ten-person company building a shopping assistant, you send your agent to a site and learn the policy by being refused, silently, and you cannot tell whether you are unwelcome or just unlucky. Amazon's phrase was "operate openly." The index shows how little an open agent would find to read at most sites, and it is the argument for a rulebook that both sides can see.

The standards people. The IETF is standardizing Web Bot Auth so an agent can sign its requests. The UCP council is defining what a merchant declares. The FIDO Alliance is carrying the AP2 mandate work. All of that is supply. The index is the demand side, measured: how many large merchants publish anything, and how that number moves as the specs land. If it does not move, the specs are answering a question merchants did not ask.

The bot-management vendors. This is the reader nobody counts, and it is the one setting policy. On September 15, Cloudflare, which sits in front of a large share of the web, began blocking both training crawlers and user-directed agents by default on ad-bearing pages of every new domain joining its network. Its own words: "On those pages, we treat human attention as the end goal, and keep away the bots." That is a door policy for a meaningful slice of the internet, set at onboarding, by a vendor, in a checkbox. Cloudflare also co-authored Web Bot Auth, which is the mechanism that would let it tell a customer's agent from a scraper. The index measures what the vendors' defaults add up to at the front door of retail. They should want that number too.

Policy and competition people. A marketplace refusing a rival's shopping agent is a question. A third of large retail doing the same thing silently, through defaults, with no published policy and no way for a consumer to know, is a bigger one. The Ninth Circuit has said the agent is legally the user. The index shows how often the user, in that form, is turned away. Anyone who will eventually have to write a rule about this needs a baseline from before the rule.

The shopper. You asked an assistant to find a coffee maker and it came back with three options from the same two sites. That is not because the others were worse. It is because the others did not let the assistant in, and neither of you was told. This piece is written so that you can follow it end to end, because the door being measured is yours.

What I will not do

No ranking. No naming of retailers from the scan data, only from the news. No probing, no logins, no impersonation, no second attempt after a refusal. The rules are the same ones behind the MCP security scoreboard, which has run every week since June with per-server findings held privately for disclosure. The reason is not politeness. It is that the point of the index is a number people trust, and the fastest way to lose that is a league table that gets a retailer's name into a headline for a bot-manager default it did not choose.

Simon Taylor wrote in May that the market had found its first real answer to agentic commerce: the AI does not own the checkout, "the merchant does." I think that is right, and the index is what it looks like one layer earlier. The merchant owns the door too. The question the index asks each week is whether the merchant knows what its door is doing.

What would move the number, and how I will know

Three things would, and each is a column the index already has or will add.

Verifiable agent identity arriving at the edge. When a bot manager can check a signature instead of guessing from an IP address, the refusal can become a decision. Web Bot Auth is the candidate; I will add support for it as a measured column when there is something to measure, and I will say the week the first large retailer answers a signed request differently from an unsigned one.

Retailers writing the policy down. Content-Signal lines in robots.txt, named agents with allow rules, an llms.txt that says what the store wants surfaced. Sixteen named an agent this month. That number is the one I most expect to move, and I will report it as it does.

Protocol profiles. Three UCP profiles out of 92 is a signal that the alternative to the bouncer exists. Whether three becomes 30 by spring is the adoption test for the whole protocol layer, and nobody else is counting.

In the MM Trust Layer Model all three are the authorization layer, the one the first year of agentic commerce left empty. The index is a weekly reading on whether it is being filled.

The ask

Use it. Cite it; the feed is open and the license allows it. Argue with the method; every choice is written down and I will change the ones that are wrong. Tell me what to add: agent-readable return policies, price and inventory feeds, whether a signed request gets a different answer. And if you run a retail site, look at your own door before the index does.

This is what Major Matters and Major Labs are for. One measures the supply side of the agent stack, the servers agents call; the other now measures the demand side, the doors agents knock on. Both are read-only, dated, reproducible, and published the same way every week by someone with no product to sell into the gap. The agentic web is being built on defaults nobody is watching. This is one of them, watched.

If the door policy for a third of American retail is a vendor default nobody has read, who is going to be the first to read it?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.