AI crawlers such as GPTBot, ClaudeBot or PerplexityBot visit your website so AI systems know and can cite your content. Your robots.txt controls who gets access. For most businesses the rule is: allow them, because whoever blocks disappears from AI answers.
What AI crawlers do
Classic search engine crawlers like Googlebot fill the search index. AI crawlers have two additional jobs: they collect content for training language models, and they fetch live information when a user asks a question in ChatGPT or Perplexity. Only sites reachable for these bots can appear as sources in AI answers.
The most important bots at a glance
- GPTBot (OpenAI): collects content for model training.
- OAI-SearchBot and ChatGPT-User (OpenAI): fetch content for web search in ChatGPT and for live requests.
- ClaudeBot (Anthropic): crawls for the Claude models.
- PerplexityBot: indexes for the answer engine Perplexity.
- Google-Extended: controls whether Google may use your content for its AI models such as Gemini. Important: blocking Google-Extended does not remove you from normal Google search or from AI Overviews, which are based on the regular Googlebot.
Block or allow?
This is a business decision, not a reflex. Publishers with exclusive paid content have good reasons to block. For service providers, retailers and SMEs the maths is usually reversed: you want your content to be found, named and recommended. More and more buying decisions start with a question to an AI, and whoever does not appear there does not exist for those customers.
How to check and steer access
The robots.txt sits in the root of your website (yourdomain.ch/robots.txt). A line like "User-agent: GPTBot" followed by "Disallow: /" locks that bot out completely, "Allow: /" permits it. Check what is in there first: some SEO plugins and hosting providers block AI crawlers without asking. As a complement, an llms.txt explains your offer to AI systems in compact form.
Whether your website is optimally set up for AI search engines is exactly what we check in our GEO optimisation, including a free audit.
Want to put this to work for your business? We review your website for free in classic and AI search and show the biggest levers.