Three roles, and why the difference matters
The names change a few times a year. The roles do not. Once you can sort a crawler into its role, you know what a visit from it means.
- Search
- This is the role that decides whether you can be cited. Each engine answers from an index, and the search crawler is what fills it. Block it, or serve it an empty page, and that engine has nothing of yours to quote.
- On request
- These visits are different in kind. Nobody scheduled them. A person asked an engine a question, and the engine went to read your page to answer it. When these show up in a log, a real answer was being written with your page open.
- Training
- Training crawlers collect pages that may shape what a future model knows. Allowing them is a business decision about your content. It is not a visibility switch, because blocking a training crawler does not remove you from answers.
- Control token
- Two names on this page are not crawlers at all. Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to allow or refuse one use of pages that another crawler has already read.
A robots.txt that welcomes citations
Googlebot and Bingbot are allowed by default on most sites, so they are not listed here. This file names the AI crawlers on purpose, which makes the decision visible to whoever edits it next.
# Search crawlers. These build the indexes AI answers are drawn from.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# On request fetchers. A person asked an engine about your page.
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Training crawlers. A business decision, not a visibility one.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://your-microsite.com/sitemap.xml
The training section is yours to change. Switching GPTBot or ClaudeBot to Disallow keeps your pages out of future training sets and leaves your place in answers alone. Switching a search crawler to Disallow does the opposite of what most people intend.
Three things that quietly block AI crawlers
-
1. Pages that need JavaScript to show their content
Most AI crawlers fetch the HTML and do not run scripts. If the page is assembled in the browser, the crawler receives a nearly empty file. Prerendering fixes this by serving finished HTML.
-
2. A firewall or CDN setting that overrides robots.txt
Many hosting and security products now ship with a switch that blocks AI bots. It sits in front of your site, so it wins no matter what robots.txt says. Check it before you check anything else.
-
3. Old blanket rules written for scrapers
A Disallow rule added years ago to keep scrapers out can match the search crawlers you now want. Read the file line by line, with the list above next to it.
How to see who visited
Analytics tools that run in the browser will not show you any of this, because crawlers do not run the script that does the counting. You need the request log from the server or from the layer in front of it.
Match on the crawler name in the user agent, then confirm the source address against the range the company publishes, where it publishes one. A user agent is a claim. Anyone can type GPTBot into a request.
Then sort the visits by role. Search crawler visits tell you a page is in an index. On request fetches tell you it was used in an answer. That split is the first report we set up on every microsite, and it is described in how the work runs.
Questions about AI crawlers
Is Google-Extended a crawler?+
No. Google-Extended is a control token. You will never see it in a server log. It is a name you put in robots.txt to say whether Google may use your pages for Gemini training and grounding, and it has no effect on Google Search.
Does blocking GPTBot remove us from ChatGPT answers?+
No. GPTBot collects pages for training. ChatGPT search is fed by OAI-SearchBot, and live page visits come from ChatGPT-User. You can block the first and still be cited, as long as the other two can reach you.
Which crawler visit is the best sign?+
An on request fetch. ChatGPT-User, Claude-User and Perplexity-User only arrive because a person asked an engine something and the engine went to read your page to answer it.
Does Grok have a crawler?+
xAI has not documented one. Several names are reported in logs, including GrokBot and xAI-SearchBot, but none is confirmed by xAI and there is no published address range to check them against. Treat a Grok visit as unconfirmed until the source address checks out.
Can AI crawlers read a JavaScript site?+
Usually not. Most AI crawlers fetch the HTML and do not run scripts, so a page that is assembled in the browser reaches them nearly empty. Serving prerendered HTML to crawlers solves it without changing what people see.
Do AI crawlers obey robots.txt?+
The documented search and training crawlers from OpenAI, Anthropic, Perplexity, Google and Microsoft say they do. On request fetchers are a special case, because a person starts each visit. OpenAI notes that robots.txt rules may not apply to ChatGPT-User for that reason.
Where this comes from
Roles and names are taken from the documentation each company publishes for site owners. Where a company publishes none, the ledger says so.