If an AI crawler cannot read your site, an AI assistant cannot quote it. It is a small technical thing with a big effect, and it is often blocked by accident: a security plugin, a CDN rule or an old line in robots.txt that nobody remembers.
This guide shows how to check, what the main crawlers are called and how to decide what to allow.
The main AI crawlers
Names and purposes come from the companies’ own documentation, which can change, so check the current pages before you edit anything.
- GPTBot (OpenAI): collects content that may be used to train models.
- OAI-SearchBot (OpenAI): used to show sites in ChatGPT search results.
- ChatGPT-User (OpenAI): used when a person asks ChatGPT to open a page.
- PerplexityBot (Perplexity): indexes sites for Perplexity search.
- ClaudeBot (Anthropic): collects content for Anthropic’s models.
- Google-Extended (Google): a control for whether content is used for Gemini training and grounding. It is not a switch for ordinary Google Search.
Training crawlers and search crawlers are different. You can allow one kind and block the other. If you want to be cited in answers, the search and user crawlers matter most.
Step 1: Read your robots.txt
Open yourdomain.com/robots.txt in a browser. Look for lines that name the crawlers above, or a blanket rule such as User-agent: * followed by Disallow: /. A line like the one below blocks a crawler completely.
User-agent: GPTBot
Disallow: /To allow a crawler, either remove its block or give it an explicit rule:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Step 2: Look beyond robots.txt
A crawler can be allowed in robots.txt and still be refused. Common causes:
- A firewall or CDN setting that blocks unknown bots or specific user agents.
- A security plugin with bot protection switched on.
- A server rule that returns an error for requests without a browser signature.
- A maintenance or coming soon page shown to non-logged-in visitors.
Your server logs show which crawlers visit and what they receive. Look for the names above and check the status codes. A run of 403 or 429 answers means they are being turned away.
Not sure what AI says about your business?
We ask ChatGPT and Perplexity the questions your customers ask and show you the answers, the sources and the gaps. No promises, just what we see.
Step 3: Check the sitemap and the basics
- Your XML sitemap exists, is listed in robots.txt and contains the pages you want quoted.
- Key pages return status 200 and are not set to noindex.
- Pages are fast enough to be fetched reliably. See our WordPress speed optimization service if they are not.
Step 4: Decide what you want
Some owners allow everything. Others allow search and user crawlers but not training crawlers. This is a business decision, and it is yours. Write it down, then make robots.txt and your firewall match it, so one does not contradict the other.
A note on llms.txt
You may also read about llms.txt, a file that gives AI systems a short guide to your site. It is a proposed convention, not an official standard, and AI providers have not promised to follow it. It can help as a tidy summary, but it is not a replacement for readable pages and open crawler access.
Frequently asked questions
Should I block GPTBot?
It depends on whether you mind your content being used for training. Blocking GPTBot does not stop OAI-SearchBot, which is separate.
How do I know if it worked?
Check your logs for the crawler names and status codes, and repeat your customer questions in ChatGPT and Perplexity a few weeks later.
We can check this for you as part of an audit. See our GEO services.
Want a clear plan for AI search?
See what a GEO audit, a fix and a monthly check include, with starting prices. Or tell us about your site and we will say honestly what is realistic.






