AI crawler access, 810 US ecommerce domains, measured 7 September 2026
Study: https://itlab.top/research/ai-crawler-access-2026/
Published by ITlab. Aggregate figures only: no domain-level data.

Cite as:
ITlab (2026). Most stores blocking ChatGPT never decided to. Measurement of
810 US ecommerce domains, 7 September 2026.
https://itlab.top/research/ai-crawler-access-2026/

FILES

sample.csv
  domains_total                       domains measured (810)
  excluded_browser_not_served         browser request did not get 200-399 (90):
                                      73 also refused every crawler, 18 did not
                                      answer at all; excluded from all figures
  comparable                          domains used for all percentages (720)
  robots_returned_200                 robots.txt answered with status 200 (692);
                                      the body was not checked to be a robots file
  robots_disallow_all_for_named_ai_agent
                                      robots.txt has "Disallow: /" in a group naming
                                      at least one of 12 AI agents (115): gptbot,
                                      oai-searchbot, chatgpt-user, perplexitybot,
                                      claudebot, google-extended, ccbot, anthropic-ai,
                                      applebot-extended, bytespider, amazonbot,
                                      meta-externalagent

crawlers.csv  (one row per crawler, out of 720 comparable domains)
  refused             browser served (200-399), crawler not served
  refused_declared    refused, and its robots.txt group has "Disallow: /"
  refused_undeclared  refused, no such rule
  served_declared     served, although robots.txt has "Disallow: /" for it
  comparable_total    720
  Crawler keys: gptbot = GPTBot, oai_search = OAI-SearchBot,
  chatgpt_user = ChatGPT-User, perplexity = PerplexityBot, claudebot = ClaudeBot

refusal-codes.csv  (all 1,399 refusals across the five crawlers)
  code     HTTP status, or network_error (no response; counted as a refusal)
  share    share of all refusals

METHOD NOTES
- Identification by User-Agent header only, from a datacenter address, not
  from the crawlers' own IP ranges.
- Status codes only: a bot-challenge page sent with status 200 counts as served.
- robots.txt: only groups naming the agent are read; "User-agent: *" rules and
  partial disallows are not counted.