AI Crawler Robots.txt Templates
Updated August 31, 2026 · 8 min read
If AI crawlers cannot fetch your pages, AI search engines cannot cite them. These templates give you a starting point for four common policies: full access, selective access, search-only, and strict protection.
Use the template that matches your business and privacy posture, then verify the live file with the robots.txt AI checker. Do not paste a policy you cannot explain to your legal, marketing, or engineering teams.
1. Full-access policy
Use this when your content is public, non-personal, and you want maximum discoverability in AI search.
User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot Allow: / User-agent: CCBot Allow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Replace example.com with your domain. This is often the right default for blogs, documentation, public research, and marketing pages.
2. Selective-access policy
Use this when you want AI assistants to read public marketing pages, but not internal tools or account areas.
User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /checkout/ User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /checkout/ User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /checkout/ User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /checkout/ User-agent: * Allow: / Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Keep blocked paths honest. If a URL returns 200 but has no public value, exclude it from your sitemap too.
3. Search-but-not-training policy
Use this when you want AI assistants to answer questions with your public content, but you prefer not to contribute to bulk model-training datasets.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This policy is intentionally simple. Commercial AI crawlers evolve, so review their documented agent names before you treat any single file as permanent.
4. Strict-protection policy
Use this only when AI citations are not useful or when the site contains sensitive, confidential, or high-liability content. For most public businesses, it removes too much discoverability.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This does not replace authentication, encryption, access control, or a real security review. It only communicates crawler policy.
Deployment checklist
- 1. Put the file at
https://your-domain.com/robots.txt. - 2. Keep your sitemap URL accurate.
- 3. Do not disallow pages you actually want cited.
- 4. Verify with the AI crawler checker.
- 5. Re-check after launching new tools, member areas, or international versions.
Check your current policy
See which AI crawlers can reach your site and where your robots.txt conflicts with your business goals.
Run Free GEO Audit →