Most stores blocking ChatGPT never decided to

The finding

Of the 720 stores that answered an ordinary browser, 440 refused a request identifying itself as GPTBot, OpenAI's crawler.

That number on its own is unremarkable. Publishers have been opting out of AI training for two years, and a store is entitled to do the same.

What is remarkable is how few of them said so. A site that means to turn a crawler away writes it in robots.txt, the file every crawler reads first. Only 74 did.

86% of refusals were never declared.

379 of the 440 stores refusing GPTBot have no rule against it in robots.txt. Their servers refuse a crawler their published policy still welcomes.

A block nobody wrote down is a block nobody chose. What these stores have is bot protection doing what it was configured to do years ago, against a category of visitor that did not exist yet.

What gets refused

The five crawlers do different jobs. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT's search answers. ChatGPT-User fetches a page when someone asks ChatGPT to look at it. PerplexityBot and ClaudeBot serve their own assistants.

The split runs along an odd line. GPTBot and ClaudeBot are refused roughly twice as often as the crawlers that actually feed answers to a shopper asking a question. If stores were making a policy choice — keep my content out of AI answers — the five numbers would look alike.

They do not, and the robots.txt evidence says this is not policy. The explanation we find most plausible — an inference, not a measurement — is that GPTBot and ClaudeBot sit on default blocklists shipped by common bot-protection products, while the search-side crawlers are classified as search engines and let through. On that reading the pattern belongs to the vendors, not to the merchants.

One consequence is worth stating plainly. 185 stores — a quarter of those measured — refused all three OpenAI crawlers. For those, ChatGPT cannot read the site by any route: not to learn it, not to index it, not when a customer pastes the URL and asks what the company sells.

Intent against behaviour

Read the other way: of the 692 stores publishing a readable robots.txt, 115 name any AI agent at all. The names that recur are the ones on the vendor lists — CCBot, Bytespider, Amazonbot, GPTBot, Meta's crawler — which suggests even the declarations are often copied rather than composed.

Thirteen sites declare a block their server does not enforce. That direction is harmless, and it is the mirror image of the same problem: the file and the machine were configured by different people at different times, and nobody reconciled them.

How it was measured

Each domain received six requests for its home page, identical but for the User-Agent header, plus one request for robots.txt. A store counts as refusing a crawler when the browser request is served and the crawler request is not.

What this does not show

We identified ourselves by header, not by address. Real crawlers arrive from published IP ranges, and serious bot management verifies them by address rather than by the name they give. A site may refuse an unverified request claiming to be GPTBot while serving the genuine one. These figures are therefore an upper bound on refusal, and the honest reading of any single result is narrower than the headline: this server refuses a request that says it is ChatGPT's crawler.

The sample is not random. 810 domains drawn from our own outreach lists: US ecommerce and B2B suppliers in retail, consumer goods, wholesale, automotive parts and food. 239 are confirmed Adobe Commerce (Magento). It represents mid-market US ecommerce, not the web, and not enterprise retail.

Location changes the answer. Measured from a US datacenter address in Atlanta. On a 30-domain comparison, the identical test run from a Ukrainian address put six stores in the refuse-everything bucket against one from Atlanta — a small sample, but enough to show that geographic filtering will distort any measurement that ignores it. Across the full set, 90 of the 810 domains refused every request we made, browser included, and are excluded from all percentages above.

One moment in time. All requests were made on 7 September 2026. Bot rules change when a WAF vendor updates a default list, which is precisely the mechanism this study is about.

Check your own store

Two requests, thirty seconds. If the first prints a refusal and the second prints 200, your server treats ChatGPT differently from a customer.

curl -s -o /dev/null -w "%{http_code}\n" -A \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://yourstore.com/

curl -s -o /dev/null -w "%{http_code}\n" -A \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
  https://yourstore.com/

If your own result surprises you, the block is probably not yours.