Free Tool

Robots.txt AI Crawler Checker

Check if your robots.txt is blocking AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended from accessing your site.

Why AI Crawler Access Matters

AI search engines use specialized crawlers to discover and index web content. If your robots.txt blocks these crawlers, your site will not appear in AI-generated answers from ChatGPT, Perplexity, Claude, or Google AI Overviews.

Many site owners accidentally block AI crawlers by using generic Disallow: / rules or aggressive bot-blocking plugins. This tool checks your robots.txt for common AI crawler blocking patterns.

AI Crawlers We Check For

GPTBot

OpenAI / ChatGPT

ClaudeBot

Anthropic / Claude

PerplexityBot

Perplexity AI

Google-Extended

Google AI Overviews / Gemini

CCBot

Common Crawl (used by many AI models)

Bytespider

ByteDance AI

Check Your Robots.txt

Enter your URL to run a full audit including robots.txt AI crawler analysis:

Ideal robots.txt for AI Visibility

# Allow AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /private/

Tip: Only block paths that truly need protection. Don't block AI crawlers from your public content.