Skip to content
Sign inStart free
FeaturesFree AI checkFix it for mePricingFAQSign in

All fix guides · Can AI read you?

Open robots.txt to AI search crawlers

Your robots.txt tells AI search crawlers to stay out.

About 10 minutes

Why it matters

robots.txt is the first file every crawler reads. A Disallow for OAI-SearchBot or PerplexityBot keeps you out of ChatGPT search and Perplexity answers entirely.

How to fix it

  1. Open yoursite.com/robots.txt.
  2. Remove any Disallow rules for OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Bingbot and Googlebot.
  3. Decide on training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) deliberately: blocking them is allowed, but models then learn less about you.
  4. Add a Sitemap line pointing at your sitemap.

Template

robots.txt
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Sitemap: https://yourcompany.in/sitemap.xml

Check your site again