Most stores blocking ChatGPT never decided to

The finding

Of the 720 stores that answered an ordinary browser, 440 refused a request identifying itself as GPTBot, OpenAI's crawler.

That number on its own is unremarkable. Publishers have been opting out of AI training for two years, and a store is entitled to do the same.

What is remarkable is how few of them said so. A site that means to turn a crawler away writes it in robots.txt, the file every crawler reads first. Only 74 did.

86% of refusals were never declared.

379 of the 440 stores refusing GPTBot have no rule against it in robots.txt. Their servers refuse a crawler their published policy still welcomes.

A block nobody wrote down is a block nobody chose. What these stores have is bot protection doing what it was configured to do years ago, against a category of visitor that did not exist yet.

What gets refused

The five crawlers do different jobs. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT's search answers. ChatGPT-User fetches a page when someone asks ChatGPT to look at it. PerplexityBot and ClaudeBot serve their own assistants.

The split runs along an odd line. GPTBot and ClaudeBot are refused roughly twice as often as the crawlers that actually feed answers to a shopper asking a question. If stores were making a policy choice — keep my content out of AI answers — the five numbers would look alike.

They do not, and the robots.txt evidence says this is not policy. The explanation we find most plausible — an inference, not a measurement — is that GPTBot and ClaudeBot sit on default blocklists shipped by common bot-protection products, while the search-side crawlers are classified as search engines and let through. On that reading the pattern belongs to the vendors, not to the merchants.

One consequence is worth stating plainly. 185 stores — a quarter of those measured — refused all three OpenAI crawlers. For those, ChatGPT cannot read the site by any route: not to learn it, not to index it, not when a customer pastes the URL and asks what the company sells.

Intent against behaviour

Read the other way: of the 692 stores whose robots.txt answered normally, 115 tell at least one of twelve named AI agents to stay out of the whole site. The names that recur are the ones on the vendor lists — CCBot, Bytespider, Amazonbot, GPTBot, Meta's crawler — which suggests even the declarations are often copied rather than composed.

Thirteen sites declare a block their server does not enforce. That direction is harmless, and it is the mirror image of the same problem: the file and the machine were configured by different people at different times, and nobody reconciled them.

How it was measured

Each domain received six requests for its home page, identical but for the User-Agent header, plus one request for robots.txt. A store counts as refusing a crawler when the browser request is served and the crawler request is not.

"Served" means a status code from 200 to 399. Network errors on a crawler request count as a refusal: 12 of the 1,399 refusals across all five crawlers. A robots.txt counts as read when it returned 200, and a crawler counts as declared blocked when its own group contains Disallow: /; rules under User-agent: * and partial disallows are not counted.

What this does not show

We identified ourselves by header, not by address. Real crawlers arrive from published IP ranges, and serious bot management verifies them by address rather than by the name they give. A site may refuse an unverified request claiming to be GPTBot while serving the genuine one, which pushes these figures up. One thing pushes them down: we recorded status codes only, so a bot-challenge page sent with a 200 counted as served. The honest reading of any single result is narrower than the headline: this server refuses a request that says it is ChatGPT's crawler.

The sample is not random. 810 domains drawn from our own outreach lists: US ecommerce and B2B suppliers in retail, consumer goods, wholesale, automotive parts and food. 239 are confirmed Adobe Commerce (Magento). It represents mid-market US ecommerce, not the web, and not enterprise retail.

Location changes the answer. Measured through a US datacenter VPN exit in Atlanta; the exit address itself was not logged for this run. On an earlier 30-domain comparison, the identical test run from a Ukrainian address put six stores in the refuse-everything bucket against one from Atlanta — a small sample whose raw results we did not keep, but enough to show that geographic filtering will distort any measurement that ignores it. Across the full set, 90 of the 810 domains did not serve the home page to the browser request: 73 of them refused every crawler as well, and 18 did not answer at all. All 90 are excluded from the percentages above.

One moment in time. All requests were made on 7 September 2026. Bot rules change when a WAF vendor updates a default list, which is precisely the mechanism this study is about.

Check your own store

Two requests, thirty seconds. If the first prints a refusal and the second prints 200, your server treats ChatGPT differently from a customer.

curl -s -o /dev/null -w "%{http_code}\n" -A \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://yourstore.com/

curl -s -o /dev/null -w "%{http_code}\n" -A \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
  https://yourstore.com/

Prefer a browser to a terminal? Run the same test on your domain and read the result on the page.

If your own result surprises you, the block is probably not yours.

Data

Aggregate results, no domain-level data: refusals by crawler, refusal status codes, sample summary, and a README defining every column.

How to cite

ITlab (2026). Most stores blocking ChatGPT never decided to. Measurement of 810 US ecommerce domains, 7 September 2026. https://itlab.top/research/ai-crawler-access-2026/

Corrections and updates

7 October 2026. Checked against the raw data. None of the headline figures change. We corrected four descriptions that said more than the method measured:

• The 115 robots.txt files block at least one of twelve named AI agents from the whole site; we had written that they name one.
• The 90 excluded domains are those that did not serve the browser request; we had written that they refused every request. 73 did, and 18 did not answer at all.
• The figures are not a pure upper bound: bot-challenge pages sent with a 200 counted as served.
• Atlanta was a datacenter VPN exit, and the exit address was not logged for this run.

We also added the method details above and the data files.

Next measurement: January 2027, on a new page. This one stays as published, so the two can be compared.