The AI crawler field guide

Who is reading your site on behalf of ChatGPT, Claude, Perplexity, Gemini and the rest, what each crawler is for, and which ones can put your link under an answer.

Checked against each company's own documentation on .

Three roles do the work, plus a control token that is not a crawler at all. Only search crawlers and on request fetchers can put your link under an answer.

OpenAI

Feeds ChatGPT

OpenAI crawler overview
  • ChatGPT-UserOn requestVisits a page when a ChatGPT user or a custom GPT asks for it.
  • GPTBotTrainingCollects pages that may be used to train OpenAI models.

Anthropic

Feeds Claude

Anthropic crawler documentation
  • Claude-UserOn requestVisits a page when a Claude user asks a question that needs it.
  • ClaudeBotTrainingCollects pages that may be used to train Claude models.

Perplexity

Feeds Perplexity

  • Perplexity-UserOn requestVisits a page when a Perplexity user asks about it.

Google

Feeds Gemini, AI Overviews and AI Mode

Google crawler overview
  • Google-ExtendedControl tokenDecides whether your pages may be used for Gemini training and grounding. It does not affect Google Search.

xAI

Feeds Grok

xAI publishes no crawler documentation. There is no user agent list, no published address range and no stated robots.txt policy. Names reported in logs, such as GrokBot and xAI-SearchBot, are unverified, so we treat a Grok visit as unconfirmed until the source address checks out.

Microsoft

Feeds Copilot

Bing crawler list

Meta

Feeds Meta AI

  • meta-externalfetcherOn requestFetches a single link when a person using a Meta product asks for it.
  • meta-externalagentTrainingCollects pages for training Meta AI models.

Apple

Feeds Siri, Spotlight and Apple Intelligence

  • Applebot-ExtendedControl tokenDecides whether pages Applebot has read may be used to train Apple models.

Amazon

Feeds Alexa

DuckDuckGo

Feeds DuckAssist

  • DuckAssistBotOn requestFetches pages in real time to source DuckAssist answers, with a link back to the page.

Mistral

Feeds Le Chat

  • MistralAI-UserOn requestVisits a page when a Le Chat user asks for it.

ByteDance

Feeds ByteDance models

  • BytespiderTrainingCollects pages for training ByteDance models.

Common Crawl

Open crawl data

  • CCBotTrainingBuilds a public archive of the web that many model builders train on.

Three roles, and why the difference matters

The names change a few times a year. The roles do not. Once you can sort a crawler into its role, you know what a visit from it means.

On request
These visits are different in kind. Nobody scheduled them. A person asked an engine a question, and the engine went to read your page to answer it. When these show up in a log, a real answer was being written with your page open.
Training
Training crawlers collect pages that may shape what a future model knows. Allowing them is a business decision about your content. It is not a visibility switch, because blocking a training crawler does not remove you from answers.
Control token
Two names on this page are not crawlers at all. Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to allow or refuse one use of pages that another crawler has already read.

A robots.txt that welcomes citations

Googlebot and Bingbot are allowed by default on most sites, so they are not listed here. This file names the AI crawlers on purpose, which makes the decision visible to whoever edits it next.

# Search crawlers. These build the indexes AI answers are drawn from.
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# On request fetchers. A person asked an engine about your page.
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Training crawlers. A business decision, not a visibility one.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://your-microsite.com/sitemap.xml

The training section is yours to change. Switching GPTBot or ClaudeBot to Disallow keeps your pages out of future training sets and leaves your place in answers alone. Switching a search crawler to Disallow does the opposite of what most people intend.

Three things that quietly block AI crawlers

  1. 1. Pages that need JavaScript to show their content

    Most AI crawlers fetch the HTML and do not run scripts. If the page is assembled in the browser, the crawler receives a nearly empty file. Prerendering fixes this by serving finished HTML.

  2. 2. A firewall or CDN setting that overrides robots.txt

    Many hosting and security products now ship with a switch that blocks AI bots. It sits in front of your site, so it wins no matter what robots.txt says. Check it before you check anything else.

  3. 3. Old blanket rules written for scrapers

    A Disallow rule added years ago to keep scrapers out can match the search crawlers you now want. Read the file line by line, with the list above next to it.

How to see who visited

Analytics tools that run in the browser will not show you any of this, because crawlers do not run the script that does the counting. You need the request log from the server or from the layer in front of it.

Match on the crawler name in the user agent, then confirm the source address against the range the company publishes, where it publishes one. A user agent is a claim. Anyone can type GPTBot into a request.

Then sort the visits by role. Search crawler visits tell you a page is in an index. On request fetches tell you it was used in an answer. That split is the first report we set up on every microsite, and it is described in how the work runs.

Questions about AI crawlers

Is Google-Extended a crawler?+

No. Google-Extended is a control token. You will never see it in a server log. It is a name you put in robots.txt to say whether Google may use your pages for Gemini training and grounding, and it has no effect on Google Search.

Does blocking GPTBot remove us from ChatGPT answers?+

No. GPTBot collects pages for training. ChatGPT search is fed by OAI-SearchBot, and live page visits come from ChatGPT-User. You can block the first and still be cited, as long as the other two can reach you.

Which crawler visit is the best sign?+

An on request fetch. ChatGPT-User, Claude-User and Perplexity-User only arrive because a person asked an engine something and the engine went to read your page to answer it.

Does Grok have a crawler?+

xAI has not documented one. Several names are reported in logs, including GrokBot and xAI-SearchBot, but none is confirmed by xAI and there is no published address range to check them against. Treat a Grok visit as unconfirmed until the source address checks out.

Can AI crawlers read a JavaScript site?+

Usually not. Most AI crawlers fetch the HTML and do not run scripts, so a page that is assembled in the browser reaches them nearly empty. Serving prerendered HTML to crawlers solves it without changing what people see.

Do AI crawlers obey robots.txt?+

The documented search and training crawlers from OpenAI, Anthropic, Perplexity, Google and Microsoft say they do. On request fetchers are a special case, because a person starts each visit. OpenAI notes that robots.txt rules may not apply to ChatGPT-User for that reason.

Where this comes from

Roles and names are taken from the documentation each company publishes for site owners. Where a company publishes none, the ledger says so.

See what AI says about your category.

Send us your site and who you compete with. We run ten buyer questions across the major AI engines and send back who gets named, which pages get cited, and whether the crawlers can read your site. Free.