Free Tool

Robots.txt AI Crawler Checker

Check if your robots.txt is blocking AI crawlers from 20 bots across 3 tiers — ChatGPT, Claude, Perplexity, Gemini, Meta AI, Apple Intelligence, xAI, and more.

Why AI Crawler Access Matters

AI search engines use specialized crawlers to discover and index web content. If your robots.txt blocks these crawlers, your site will not appear in AI-generated answers from ChatGPT, ChatGPT Search, Perplexity, Claude, Google AI Overviews, Meta AI, Apple Intelligence, or Amazon Rufus.

Many site owners accidentally block AI crawlers by using generic Disallow: / rules or aggressive bot-blocking plugins. This tool checks your robots.txt for 20 AI crawler blocking patterns.

AI Crawlers We Check (20 bots, 3 tiers)

Tier 1 — Critical (6 bots)

GPTBot

OpenAI / ChatGPT

OAI-SearchBot

OpenAI Search / ChatGPT Search

ClaudeBot

Anthropic / Claude

anthropic-ai

Anthropic (alt crawler)

PerplexityBot

Perplexity AI

Google-Extended

Google AI Overviews / Gemini

Tier 2 — Important (7 bots)

CCBot

Common Crawl (used by many AI models)

Bytespider

ByteDance / TikTok AI

meta-externalagent

Meta AI / Meta Search

Amazonbot

Amazon AI / Rufus

Applebot-Extended

Apple Intelligence / Apple AI

ChatGPT-User

OpenAI (user-initiated)

Claude-SearchBot

Anthropic Search

Tier 3 — Emerging (7 bots)

cohere-ai

Cohere

DuckAssistBot

DuckDuckGo AI

AI2Bot

Allen Institute (AI2)

xAI-Bot

Grok / xAI

Perplexity-User

Perplexity (user-initiated)

YouBot

You.com AI

PetalBot

Huawei AI

Check Your Robots.txt

Enter your URL to run a full audit including robots.txt AI crawler analysis:

Ideal robots.txt for AI Visibility

# Allow AI crawlers — Tier 1: Critical
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# Tier 2: Important
User-agent: CCBot
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Tier 3: Emerging
User-agent: DuckAssistBot
Allow: /

User-agent: xAI-Bot
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /private/

Tip: Only block paths that truly need protection. Don't block AI crawlers from your public content.