Picture a shopping agent that does everything right. It names itself in its user agent string, asks for robots.txt before touching anything else, makes six plain requests in total and never retries a refusal.

On October 17, 2026, an agent built to that description knocked on the front door of 92 of the largest US retailers. 31 of them would not show it robots.txt: 24 answered with a 403, five did not answer at all, one served an HTML page where the rules should be, and one returned HTTP 418, the code a 1998 April Fools RFC reserved for "I'm a teapot." Only 46 served a homepage with a title and at least ten links. 32 refused or challenged it. Seven returned a 200 with nothing inside, a page that only fills in once JavaScript runs.

Those are the aggregates from the Merchant Agent-Readiness Index, which I run every Saturday against the same 92 storefronts. It names no retailer, and the counts have barely moved since September.

Four days after that run, Walmart announced what a retailer looks like when it builds a front door on purpose: an Agent Gateway, a profile per agent, spending limits of $50, $100 or $200, approval required by default, and no card details handed to the agent at all. Most retailers are still deciding whether to unlock the door.

An agent cannot buy what it cannot read. Before mandates, identity or checkout protocols come into it, one page and five files decide whether a shopping agent gets past your front door.

The six checks below are the ones the index runs, in the order it runs them, each with a weight out of 100. The homepage carries 35 on its own, because an agent that cannot read it cannot do anything else.

Check one: robots.txt, and what it tells 14 named agents

What it looks for. A plain-text file at /robots.txt that answers a 200. RFC 9309, the 2022 standard for the Robots Exclusion Protocol, says the rules "MUST be accessible in a file named /robots.txt (all lowercase) in the top-level path." The probe then reads the groups for 14 named AI agents: eight training crawlers (GPTBot, ClaudeBot, CCBot and their peers), three AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), and three that fetch a page because a person asked: ChatGPT-User, Claude-User and Perplexity-User.

That last family matters most for shopping, and it carries its own 15 points on top of the 15 for the file. OpenAI describes ChatGPT-User as "not used for crawling the web in an automatic fashion." Anthropic says Claude-User is the agent Claude uses "when individuals ask questions." Perplexity says Perplexity-User "generally ignores robots.txt rules." Disallow the first two and you have turned away a customer mid-purchase, not a scraper. Disallow the third and it may not notice.

What fails. Among the 61 readable files on October 17, only 14 named any AI agent at all. Two blocked a user-triggered agent outright.

RFC 9309 adds an irony. If a server answers robots.txt with a 4xx, a crawler "MAY access any resources on the server," so the 24 retailers answering 403 have, in protocol terms, told every crawler that nothing is off limits, even as their bot managers refuse the next request. The standard also says these rules "are not a form of access authorization." Robots.txt is a notice, not a lock.

The fix. Publish a plain-text robots.txt that answers 200 to anyone, and name the agents you accept with Allow rules above the wildcard.

Check two: the homepage, served, challenged, or an empty shell

What it looks for. A GET on / under an honest user agent, sorted into three outcomes. Served is a 200 with a title and at least ten links. Refused or challenged is a 401, 403, 406, 429 or 503, or a 200 carrying bot-challenge wording. Empty shell is a 200 with neither: the page exists, but the catalog only appears after scripts run.

What fails. 46 of 92 storefronts served a readable page. 32 were refused or challenged, most with a 403. Seven were empty shells. Served pages had a median of 123 links; shells had zero, one or two. An agent without a browser reads the shell and reports, truthfully, that the store has nothing on it.

A bot manager that blocks by default is a business decision, and I covered the quiet version of it when the index launched. The question is whether anyone chose it.

The fix. Decide which self-identifying agents you want and let them past the bot manager. If the storefront is a JavaScript shell, server-render the navigation and product pages.

Check three: structured data on the homepage

What it looks for. JSON-LD blocks, and whether any declare a schema.org Product, Offer, ItemList or ProductGroup. An Offer carries price, priceCurrency, availability, seller, shippingDetails and hasMerchantReturnPolicy. A Product carries its offers. Together they let an agent read price and stock without parsing your page layout.

What fails. Of the 46 served homepages, 27 carried JSON-LD. Two carried any product-level type. The rest were Organization, WebSite and a SearchAction: markup for Google's sitelinks box, not for a buyer. Product markup earns the full 10 points; Organization alone earns five.

The fix. Add Product and Offer markup to the pages that have products on them, and keep the markup price the same as the page price.

Check four: llms.txt

What it looks for. A text file at /llms.txt with more than a trivial amount in it. The proposal, published by Jeremy Howard in September 2024, asks for a Markdown file with an H1 naming the site, a blockquote summary, and H2 sections of links, "to provide information to help agents use a website."

What fails. Nothing published, or an HTML page at the path. 15 of the 92 had one.

The fix. Write one page: what the store sells, where the catalog and search live, shipping and returns in a sentence each, and how to reach support. Publish it as text at /llms.txt for 10 points.

Check five: the UCP profile

What it looks for. A JSON file at /.well-known/ucp with a top-level "ucp" key. Google's documentation for the Universal Commerce Protocol says the file "must be publicly accessible and not require any authentication," and that it declares the features the store supports (checkout, fulfillment, discounts, order management), pins the spec versions, and tells a platform where to send requests.

What fails. No file, which was the case for 89 of 92. Three published one. A valid profile earns 10 points.

The fix. If you sell through a UCP-capable platform, publish the profile it generates. If not, watch this one rather than build it.

Check six: the A2A agent card

What it looks for. A JSON document at /.well-known/agent-card.json, or the older /.well-known/agent.json. The A2A specification, at version 1.0 under the Linux Foundation after Google donated it, says "A2A Servers MUST make an Agent Card available" and requires a name, description, version, interfaces, capabilities and skills.

What fails. Almost everyone, and that is fine for now. The index found its first agent card at a large US retailer on October 17. One of 92. The check is worth five points.

The fix. None yet, unless you already run an agent that other agents should find.

What the six checks do not measure

They stop at the door. In the MM Trust Layer Model, these six checks are the discovery layer and nothing else: can an agent find you, read you, and learn what you sell. Mandates, spending caps, telling one agent from another, whether a card number ever reaches the agent: none of that is in scope.

Walmart's announcement is almost entirely about those other layers. Per-agent profiles are identity. Spending limits and default approval are a mandate. Keeping card details away from the agent is settlement design. I wrote about what happens when the platform writes the mandate and the customer never signs it. The course Anatomy of an Agentic Payment walks one transaction through every layer, and the Bench tests agents under a signed mandate on real hardware. For the basics, start with how AI agents work.

Passing all six does not mean an agent can buy from you. Failing the first two means it cannot even try.

Run it on your own store

Hunch is a research instrument from Major Labs. Type in a store domain and it runs the same six probes, in the same order, under the same honest user agent, and shows the raw response from each request next to the score and the fix. Six read-only GET requests, no login, no cart, no retry after a refusal. Nothing is stored or published, and nothing you check is added to the public data.

The playground is the other half: a reference store that is consistent everywhere an agent might look, plus ten traps built wrong on purpose, each with a published answer key. A price that differs between the JSON-LD and the page. A handling fee that appears only at checkout. A product description with instructions addressed to the agent. The mock checkout refuses any order without a spending mandate and records what the agent did, so you can compare its story with the record.

Point your own agent at it before you point it at a customer.

If a customer's agent is turned away at your front door, who do they blame: the agent, or the store?

Charlie Major is a Product Development Manager at Mastercard. The views and opinions expressed in Major Matters are his own and do not represent those of Mastercard.