NeuralCrawl · October 18, 2026

NeuralCrawl

Monitor how companies and governments treat AI crawlers and search engine bots.

Analyze 1273 major US & European companies, governments, social networks and publishers (updated 24/7).

157 of 1202 analysed websites explicitly block at least one AI crawler by name in their robots.txt

8.7% block GPTBot (OpenAI) — 105 companies

4800 robots.txt snapshots in the archive

215 policy changes detected in the last 7 days

1205 robots.txt files under 24/7 monitoring

AI bots rejection over time weekly · US 500 cohort

Block ≥1 AI crawler Block GPTBot

2026-10-06 2026-10-17

12 weekly snapshots with reliable coverage

“13.1% of monitored organisations now explicitly block at least one AI crawler in their robots.txt
OpenAI's GPTBot is named by 105 of them.”
— Elie Berreby, October 18, 2026 · cohort us-500 · trends

Most blocked AI crawlers share of 1202 analysed websites

GPTBot OpenAI 105 (8.7%) CCBot Common Crawl 103 (8.6%) Bytespider ByteDance 97 (8.1%) ClaudeBot Anthropic 96 (8.0%) Google-Extended Google 87 (7.2%) meta-externalagent Meta 82 (6.8%) Applebot-Extended Apple 77 (6.4%) anthropic-ai Anthropic 68 (5.7%) Diffbot Diffbot 68 (5.7%) omgili Webz.io 66 (5.5%)

Which sectors block AI the most all monitored sites · share blocking ≥1 AI crawler · min 5 companies

Social & Communities 58.3% (7/12)

News & Media 56.4% (44/78)

Entertainment & Sports 20.6% (7/34)

Telecom 16.7% (4/24)

Retail & E-commerce 13.7% (14/102)

Technology & Software 10.1% (24/238)

Business & Professional Services 9.1% (1/11)

Education & Research 9.0% (6/67)

Energy & Utilities 5.6% (4/71)

Travel & Hospitality 4.8% (1/21)

Finance & Insurance 4.3% (6/139)

Industrials & Manufacturing 4.2% (5/120)

Most AI-restrictive companies AI crawlers blocked by name

#	Company	Sector	Bots blocked
4	USA Today	News & Media	28
60	Toronto Star	News & Media	28
15	Politico	News & Media	27
38	Msn	Other	27
44	Daily Mail	News & Media	27
104	IGN	Other	27
59	The Globe and Mail	News & Media	26
42	The Telegraph	News & Media	25
56	Nikkei	Finance & Insurance	25
983	wikiHow	Education & Research	25

Change activity robots.txt modifications per week

234 30

221 31

219 32

215 33

243 34

201 35

224 36

205 37

225 38

203 39

219 40

191 41

ISO week numbers. Baseline (first-archive) events excluded.

Latest AI-policy moves from the change feed

Coursera — Added Allow: /api/utilities/v1/imageproxy for CCBot 4h ago
Coursera — Added Allow: /llms.txt for CCBot 4h ago
Coursera — Added Disallow: /5JKLVK/ for CCBot 4h ago
Coursera — Added Disallow: /acclaimbadge/ for CCBot 4h ago
Advance Auto Parts — Blocked Applebot-Extended (Apple) entirely 10h ago
Advance Auto Parts — Blocked Bytespider (ByteDance) entirely 10h ago
Advance Auto Parts — Blocked CCBot (Common Crawl) entirely 10h ago
Advance Auto Parts — Blocked ClaudeBot (Anthropic) entirely 10h ago
Advance Auto Parts — Blocked Google-Extended (Google) entirely 10h ago
Advance Auto Parts — Blocked GPTBot (OpenAI) entirely 10h ago
Advance Auto Parts — Blocked meta-externalagent (Meta) entirely 10h ago
Advance Auto Parts — Unblocked Applebot-Extended (Apple) - rules removed 16h ago
Advance Auto Parts — Unblocked Bytespider (ByteDance) - rules removed 16h ago
Advance Auto Parts — Unblocked CCBot (Common Crawl) - rules removed 16h ago
Advance Auto Parts — Unblocked ClaudeBot (Anthropic) - rules removed 16h ago
Advance Auto Parts — Unblocked Google-Extended (Google) - rules removed 16h ago
Advance Auto Parts — Unblocked GPTBot (OpenAI) - rules removed 16h ago
Advance Auto Parts — Unblocked meta-externalagent (Meta) - rules removed 16h ago
Cisco Systems — Removed rules for AI crawler CCBot (Common Crawl) 1d ago
Cisco Systems — Removed rules for AI crawler ClaudeBot (Anthropic) 1d ago
Cisco Systems — Removed rules for AI crawler Google-Extended (Google) 1d ago
Cisco Systems — Removed rules for AI crawler GPTBot (OpenAI) 1d ago
Figma Community — Added Allow: /es-es/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-la/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /fr-fr/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /hi-in/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /id-id/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /it-it/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ja-jp/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ko-kr/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /nl-nl/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /pl-pl/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /pt-br/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/ai-animation-examples/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/ai-character-consistency/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/ai-video-examples/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/ai-video-prompt/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/best-node-based-ai-platform/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/no-code-chatbot-builder/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/tweening/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /resource-library/whiteboard-animation/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /th-th/agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-es/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-la/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /fr-fr/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /hi-in/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /id-id/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /it-it/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ja-jp/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ko-kr/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /nl-nl/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /pl-pl/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /pt-br/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/ai-animation-examples/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/ai-character-consistency/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/ai-video-examples/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/ai-video-prompt/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/best-node-based-ai-platform/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/no-code-chatbot-builder/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/tweening/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /resource-library/whiteboard-animation/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /th-th/agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-es/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /es-la/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /fr-fr/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /hi-in/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /id-id/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /it-it/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /ja-jp/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /ko-kr/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /nl-nl/agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /pl-pl/agent/$ for Claude-User 1d ago
EPFL — Unblocked Amazonbot (Amazon) - rules removed 1d ago
EPFL — Unblocked anthropic-ai (Anthropic) - rules removed 1d ago
EPFL — Unblocked Applebot-Extended (Apple) - rules removed 1d ago
EPFL — Unblocked Bytespider (ByteDance) - rules removed 1d ago
EPFL — Unblocked CCBot (Common Crawl) - rules removed 1d ago
EPFL — Unblocked Claude-Web (Anthropic) - rules removed 1d ago
EPFL — Unblocked ClaudeBot (Anthropic) - rules removed 1d ago
EPFL — Unblocked cohere-ai (Cohere) - rules removed 1d ago
EPFL — Unblocked Diffbot (Diffbot) - rules removed 1d ago
EPFL — Unblocked Google-Extended (Google) - rules removed 1d ago
EPFL — Unblocked GPTBot (OpenAI) - rules removed 1d ago
EPFL — Unblocked meta-externalagent (Meta) - rules removed 1d ago
EPFL — Unblocked omgili (Webz.io) - rules removed 1d ago
Alamy — Added Disallow: /fpanalytics/ for AI2Bot 1d ago
Alamy — Added Disallow: /purchased-assets/ for AI2Bot 1d ago
Alamy — Added Disallow: /search-assistant/ for AI2Bot 1d ago
Alamy — Added Disallow: /fpanalytics/ for Applebot-Extended 1d ago
Alamy — Added Disallow: /purchased-assets/ for Applebot-Extended 1d ago
Alamy — Added Disallow: /search-assistant/ for Applebot-Extended 1d ago
Alamy — Added Disallow: /fpanalytics/ for Bytespider 1d ago
Alamy — Added Disallow: /purchased-assets/ for Bytespider 1d ago
Alamy — Added Disallow: /search-assistant/ for Bytespider 1d ago
Alamy — Added Disallow: /fpanalytics/ for CCBot 1d ago
Alamy — Added Disallow: /purchased-assets/ for CCBot 1d ago
Alamy — Added Disallow: /search-assistant/ for CCBot 1d ago
Alamy — Added Disallow: /fpanalytics/ for ClaudeBot 1d ago
Alamy — Added Disallow: /purchased-assets/ for ClaudeBot 1d ago
Alamy — Added Disallow: /search-assistant/ for ClaudeBot 1d ago
Alamy — Added Disallow: /fpanalytics/ for GPTBot 1d ago
Alamy — Added Disallow: /purchased-assets/ for GPTBot 1d ago
Alamy — Added Disallow: /search-assistant/ for GPTBot 1d ago
Figma Community — Added Allow: /de-de/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /de-de/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-es/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-es/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-la/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /es-la/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /fr-fr/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /fr-fr/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /id-id/solusi/pemeriksa-aksesibilitas-ai/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ja-jp/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ja-jp/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ko-kr/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /ko-kr/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /nl-nl/oplossingen/ai-toegankelijkheid-checker/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /pl-pl/rozwiazania/sprawdzacz-dostepnosci-ai/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /pt-br/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /pt-br/blog/figma-agent-and-design-systems/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /th-th/solutions/ai-accessibility-checker/$ for CHATGPT-User 1d ago
Figma Community — Added Allow: /de-de/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /de-de/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-es/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-es/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-la/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /es-la/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /fr-fr/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /fr-fr/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /id-id/solusi/pemeriksa-aksesibilitas-ai/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ja-jp/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ja-jp/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ko-kr/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /ko-kr/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /nl-nl/oplossingen/ai-toegankelijkheid-checker/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /pl-pl/rozwiazania/sprawdzacz-dostepnosci-ai/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /pt-br/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /pt-br/blog/figma-agent-and-design-systems/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /th-th/solutions/ai-accessibility-checker/$ for Claude-SearchBot 1d ago
Figma Community — Added Allow: /de-de/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /de-de/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /es-es/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /es-es/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /es-la/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /es-la/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /fr-fr/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /fr-fr/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /id-id/solusi/pemeriksa-aksesibilitas-ai/$ for Claude-User 1d ago
Figma Community — Added Allow: /ja-jp/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /ja-jp/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /ko-kr/blog/3-ways-product-designers-use-the-figma-agent/$ for Claude-User 1d ago
Figma Community — Added Allow: /ko-kr/blog/figma-agent-and-design-systems/$ for Claude-User 1d ago
Figma Community — Added Allow: /nl-nl/oplossingen/ai-toegankelijkheid-checker/$ for Claude-User 1d ago

Methodology. robots.txt files are fetched every 6 hours from the primary domains of every monitored cohort (US & European companies, governments and social networks). "Blocks" counts organisations that name an AI crawler in their own User-agent group with Disallow: /. 1202 of 1273 sites are reachable and analysed. Browse cohorts on the datasets page and trends charts. Full crawl health on the status page.