NeuralCrawl · August 19, 2026

NeuralCrawl

Monitor how companies and governments treat AI crawlers and search engine bots.

Analyze 1273 major US & European companies, governments, social networks and publishers (updated 24/7).

160 of 1196 analysed websites explicitly block at least one AI crawler by name in their robots.txt

9.3% block GPTBot (OpenAI) — 111 companies

2962 robots.txt snapshots in the archive

212 policy changes detected in the last 7 days

1199 robots.txt files under 24/7 monitoring

AI bots rejection over time weekly · US 500 cohort

Block ≥1 AI crawler Block GPTBot

2026-08-07 2026-08-18

12 weekly snapshots with reliable coverage

“13.4% of monitored organisations now explicitly block at least one AI crawler in their robots.txt
OpenAI's GPTBot is named by 111 of them.”
— Elie Berreby, August 19, 2026 · cohort us-500 · trends

Most blocked AI crawlers share of 1196 analysed websites

GPTBot OpenAI 111 (9.3%) CCBot Common Crawl 110 (9.2%) ClaudeBot Anthropic 103 (8.6%) Bytespider ByteDance 101 (8.4%) Google-Extended Google 93 (7.8%) meta-externalagent Meta 91 (7.6%) Applebot-Extended Apple 83 (6.9%) Amazonbot Amazon 74 (6.2%) anthropic-ai Anthropic 68 (5.7%) Diffbot Diffbot 66 (5.5%)

Which sectors block AI the most all monitored sites · share blocking ≥1 AI crawler · min 5 companies

News & Media 55.1% (43/78)

Social & Communities 50.0% (6/12)

Entertainment & Sports 24.2% (8/33)

Telecom 13.0% (3/23)

Retail & E-commerce 11.9% (12/101)

Technology & Software 10.1% (24/237)

Business & Professional Services 9.1% (1/11)

Education & Research 9.0% (6/67)

Energy & Utilities 7.0% (5/71)

Travel & Hospitality 5.0% (1/20)

Finance & Insurance 4.3% (6/139)

Industrials & Manufacturing 4.2% (5/120)

Most AI-restrictive companies AI crawlers blocked by name

#	Company	Sector	Bots blocked
4	USA Today	News & Media	28
60	Toronto Star	News & Media	28
15	Politico	News & Media	27
38	Msn	Other	27
44	Daily Mail	News & Media	27
104	IGN	Other	27
59	The Globe and Mail	News & Media	26
9	CNN	News & Media	25
42	The Telegraph	News & Media	25
56	Nikkei	Finance & Insurance	25

Change activity robots.txt modifications per week

42 24

173 25

202 26

201 27

198 28

238 29

234 30

221 31

219 32

66 33

ISO week numbers. Baseline (first-archive) events excluded.

Latest AI-policy moves from the change feed

Reliance Steel & Aluminum — Blocked Bytespider (ByteDance) entirely 1h ago
Reliance Steel & Aluminum — Blocked CCBot (Common Crawl) entirely 1h ago
Reliance Steel & Aluminum — Blocked Claude-SearchBot (Anthropic) entirely 1h ago
Reliance Steel & Aluminum — Blocked ClaudeBot (Anthropic) entirely 1h ago
Reliance Steel & Aluminum — Blocked GPTBot (OpenAI) entirely 1h ago
Reliance Steel & Aluminum — Blocked meta-externalagent (Meta) entirely 1h ago
Reliance Steel & Aluminum — Blocked omgili (Webz.io) entirely 1h ago
Renault Group — Unblocked ClaudeBot (Anthropic) 13h ago
Renault Group — Removed Allow: /br/ for ClaudeBot 13h ago
Renault Group — Removed Allow: /de/ for ClaudeBot 13h ago
Renault Group — Removed Allow: /es/ for ClaudeBot 13h ago
Renault Group — Removed Allow: /it/ for ClaudeBot 13h ago
Renault Group — Removed Allow: /pt/ for ClaudeBot 13h ago
Renault Group — Removed Allow: /ro/ for ClaudeBot 13h ago
Renault Group — Unblocked GPTBot (OpenAI) 13h ago
Renault Group — Removed Allow: /br/ for GPTBot 13h ago
Renault Group — Removed Allow: /de/ for GPTBot 13h ago
Renault Group — Removed Allow: /es/ for GPTBot 13h ago
Renault Group — Removed Allow: /it/ for GPTBot 13h ago
Renault Group — Removed Allow: /pt/ for GPTBot 13h ago
Renault Group — Removed Allow: /ro/ for GPTBot 13h ago
DOAJ — Unblocked Amazonbot (Amazon) - rules removed 13h ago
DOAJ — Unblocked Applebot-Extended (Apple) - rules removed 13h ago
DOAJ — Unblocked Bytespider (ByteDance) - rules removed 13h ago
DOAJ — Unblocked CCBot (Common Crawl) - rules removed 13h ago
DOAJ — Unblocked ClaudeBot (Anthropic) - rules removed 13h ago
DOAJ — Unblocked Google-Extended (Google) - rules removed 13h ago
DOAJ — Unblocked GPTBot (OpenAI) - rules removed 13h ago
DOAJ — Unblocked meta-externalagent (Meta) - rules removed 13h ago
Indeed — Added Allow: /*&start=0& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=10& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=20& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=30& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=40& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=50& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=60& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=70& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=80& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=90& for AI2Bot 20h ago
Indeed — Added Allow: /*&start=0& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=10& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=20& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=30& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=40& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=50& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=60& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=70& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=80& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=90& for AmazonBot 20h ago
Indeed — Added Allow: /*&start=0& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=10& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=20& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=30& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=40& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=50& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=60& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=70& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=80& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=90& for anthropic-ai 20h ago
Indeed — Added Allow: /*&start=0& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=10& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=20& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=30& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=40& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=50& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=60& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=70& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=80& for Applebot-Extended 20h ago
Indeed — Added Allow: /*&start=90& for Applebot-Extended 20h ago
AstraZeneca — Added rules for AI crawler Amazonbot (Amazon) 1d ago
AstraZeneca — Added rules for AI crawler Applebot-Extended (Apple) 1d ago
AstraZeneca — Added rules for AI crawler CCBot (Common Crawl) 1d ago
AstraZeneca — Added rules for AI crawler ChatGPT-User (OpenAI) 1d ago
AstraZeneca — Added rules for AI crawler Claude-SearchBot (Anthropic) 1d ago
AstraZeneca — Added rules for AI crawler Claude-User (Anthropic) 1d ago
AstraZeneca — Added rules for AI crawler ClaudeBot (Anthropic) 1d ago
AstraZeneca — Added rules for AI crawler DuckAssistBot (DuckDuckGo) 1d ago
AstraZeneca — Added rules for AI crawler Google-Extended (Google) 1d ago
AstraZeneca — Added rules for AI crawler GPTBot (OpenAI) 1d ago
AstraZeneca — Added rules for AI crawler meta-externalagent (Meta) 1d ago
AstraZeneca — Added rules for AI crawler MistralAI-User (Mistral AI) 1d ago
AstraZeneca — Added rules for AI crawler OAI-SearchBot (OpenAI) 1d ago
AstraZeneca — Added rules for AI crawler Perplexity-User (Perplexity) 1d ago
AstraZeneca — Added rules for AI crawler PerplexityBot (Perplexity) 1d ago
AstraZeneca — Added rules for AI crawler YouBot (You.com) 1d ago
Itaú Unibanco — Added rules for AI crawler Amazonbot (Amazon) 1d ago
Itaú Unibanco — Added rules for AI crawler Applebot-Extended (Apple) 1d ago
Itaú Unibanco — Added rules for AI crawler CCBot (Common Crawl) 1d ago
Itaú Unibanco — Added rules for AI crawler Google-Extended (Google) 1d ago
Itaú Unibanco — Added rules for AI crawler meta-externalagent (Meta) 1d ago
Itaú Unibanco — Removed rules for AI crawler ChatGPT-User (OpenAI) 1d ago
Itaú Unibanco — Removed rules for AI crawler Claude-User (Anthropic) 1d ago
Itaú Unibanco — Removed rules for AI crawler Perplexity-User (Perplexity) 1d ago
Itaú Unibanco — Removed rules for AI crawler YouBot (You.com) 1d ago
Itaú Unibanco — Added Allow: /investimentos/ for ClaudeBot 1d ago
Itaú Unibanco — Added Disallow: /api/internal/ for ClaudeBot 1d ago
Itaú Unibanco — Added Disallow: /auth/ for ClaudeBot 1d ago
Itaú Unibanco — Added Allow: /investimentos/ for OAI-SearchBot 1d ago
Itaú Unibanco — Added Disallow: /api/internal/ for OAI-SearchBot 1d ago
Itaú Unibanco — Added Disallow: /login/ for OAI-SearchBot 1d ago
Itaú Unibanco — Added Disallow: /private/ for OAI-SearchBot 1d ago
Itaú Unibanco — Added Allow: /investimentos/ for PerplexityBot 1d ago
Itaú Unibanco — Added Disallow: /api/internal/ for PerplexityBot 1d ago
Itaú Unibanco — Added Disallow: /auth/ for PerplexityBot 1d ago
Peru (Gob.pe) — Added rules for AI crawler GPTBot (OpenAI) 3d ago
Peru (Gob.pe) — Added rules for AI crawler OAI-SearchBot (OpenAI) 3d ago
Figma Community — Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User 4d ago
Figma Community — Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for Claude-SearchBot 4d ago
Figma Community — Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for Claude-User 4d ago
Figma Community — Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for OAI-SearchBot 4d ago
Figma Community — Added Allow: /blog/how-to-move-fast-toward-the-right-thing/$ for PerplexityBot 4d ago

Methodology. robots.txt files are fetched every 6 hours from the primary domains of every monitored cohort (US & European companies, governments and social networks). "Blocks" counts organisations that name an AI crawler in their own User-agent group with Disallow: /. 1196 of 1273 sites are reachable and analysed. Browse cohorts on the datasets page and trends charts. Full crawl health on the status page.