AI Crawlers: What to Allow on Your Site

AI crawlers are a policy decision for any site that publishes on the open web. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Googlebot each have a job, and one block treats a training crawl, a citation and your Google listing as the same event.

The choice is between AI crawlers that may train on a page and AI crawlers that may quote it when someone asks a related question. You can refuse training, allow quoting, and keep the public site in Google Search. The token in robots.txt is the instruction.

The decision fits a company site, a WordPress magazine, a documentation host or a shop. This article is the allow-list: which tokens to name, which paths stay closed, and how to check. Titles, links and indexation are a separate audit, in our technical SEO services.

What AI crawlers actually fetch

AI crawlers are ordinary HTTP clients. They request a URL, send a published token in the User-Agent header, and store the response for the product that token serves.

Five names cover most public sites. GPTBot collects pages that may train OpenAI’s foundation models. ClaudeBot collects pages that may train Claude. PerplexityBot is how Perplexity surfaces and links sites in answers, and Perplexity calls that token search. Google-Extended is a robots.txt control for Gemini. Googlebot fills the Google Search index.

The token is the whole address. A rule for GPTBot says nothing to ClaudeBot, and a rule for Google-Extended says nothing to Googlebot. AI crawlers from one company are often split on purpose, so a training opt-out and a citation opt-in can both be true on the same host.

A fetch, a citation and a training set are three outcomes. Training can shape a later model that answers without linking you. A search index for an answer product can quote or link the page. Each outcome has its own token.

Training copies and quotes in the answer

AI crawlers split into two jobs, and the brand name does not merge them. A training token lets public pages improve a general model, which can later discuss your topic without a visit. A search token lets the answer product retrieve the page and show a link or a quotation.

OpenAI documents the split in its crawler overview. GPTBot is the training crawl, and a disallow keeps the content out of foundation-model training. OAI-SearchBot is what ChatGPT search uses to surface sites in answers. The groups are independent: disallow GPTBot and allow OAI-SearchBot. Search behaviour can take about a day to follow a changed file. ChatGPT-User visits only when a person asks, and robots.txt may not apply.

Anthropic uses the same shape. ClaudeBot is the training collection, and a block keeps future material out of training sets. Claude-SearchBot builds the index behind Claude’s search answers. Each token needs its own group, and each hostname needs its own robots.txt, because a rule on the apex says nothing to a subdomain.

PerplexityBot sits on the quoting side. Perplexity says that crawl indexes pages for search and is separate from foundation-model pre-training. A disallow stops the full text. Perplexity-User fetches after a question, and Perplexity says that agent generally ignores robots.txt.

An allowed search token makes a citation possible. The page still has to answer the question clearly. A disallow on GPTBot or ClaudeBot instructs that operator about future crawls. A copy already inside an older model is outside what a new file can reach.

A robots.txt pattern you can keep

robots.txt lives at the root of one host. The public file says nothing about staging.example.com. With no matching group, the usual default is that the agent may fetch. The file is how you name the exceptions.

Keep it short. Disallow the training tokens by name. Leave every other agent on one shared group, and put private paths only there.

GPTBot and ClaudeBot stay off the whole host. Google-Extended gets the same disallow, the Gemini control in the next section. Googlebot, PerplexityBot, OAI-SearchBot and Claude-SearchBot have no group of their own, so they follow the star: public URLs stay open, and /wp-admin/ and /staging/ stay closed. WordPress still needs admin-ajax.php, because the public site calls it. Google lets the longer Allow win over Disallow: /wp-admin/.

A named group replaces the star group. It does not inherit the star Disallow lines. User-agent: Googlebot with only Allow: / opens every path, including paths the star group was there to close. Repeat private paths inside any named group, or skip that group.

Delete the three named groups if you accept training as well as citations, and keep the star group. To stay out of answer engines as well, disallow OAI-SearchBot, Claude-SearchBot and PerplexityBot in groups of their own. Leave Googlebot off that list unless leaving Google Search is the decision. AI crawlers you have not named keep following the star group.

On WordPress, confirm the live file. Core and some SEO plugins emit a virtual robots.txt, and a real file in the web root replaces it. If Cloudflare Bot Preference Sync is on, the served copy can include managed lines. Request the public /robots.txt after every change.

Googlebot and Google-Extended

Googlebot requests pages for the Google Search index. Disallow it on a public URL and that URL can fall out of Search. For a site that should stay findable, leave Googlebot allowed.

Google-Extended is a product token in robots.txt. Google’s crawler overview says fetching is done by existing Google user agents. The token says whether that content may train future Gemini models, and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI.

Search inclusion and ranking stay with Googlebot. Google places Google-Extended outside that decision. After a disallow, the log still shows Googlebot. A user agent named Google-Extended never appears, because the string is only a rule in the file.

Search Console has a separate Search generative AI control for AI Overviews, AI Mode and generative features in Discover. Google-Extended remains the training switch. noindex, or a Googlebot disallow, takes the URL out of Search as a whole.

Staging and login areas stay closed

Private paths are a hosting rule for every agent. Staging, draft previews, /wp-admin/ and any URL with customer data belong in Disallow on the star group, so the rule covers Googlebot, the search tokens and AI crawlers together.

Authentication keeps the area private. robots.txt asks a polite client to skip a path, and anyone who reads the file can see that path. Use a staging hostname absent from public DNS, or HTTP authentication on the whole host, plus noindex in case a URL leaks.

Thank-you pages, member libraries and anything you would refuse to paste into a chat follow the same rule. A URL a person can open without an account is a URL a user-triggered agent may open. ChatGPT-User and Perplexity-User are documented that way.

How to see which AI crawlers fetched the site

The file is a hypothesis until a log agrees. The origin access log stores each User-Agent. One filter covers the tokens here:

Googlebot and the search tokens you allowed should request public URLs and get ordinary status codes. GPTBot and ClaudeBot should go quiet. Google-Extended will be missing as a user agent. AI crawlers that ignore the file still show up here, on a path the file closed.

On sites we host, Cloudflare sits in front of the origin. AI Crawl Control lists which AI bots came, how often, and whether they violated robots.txt. Allow and Block there are enforcement. Read the public robots.txt, because a managed Cloudflare copy can differ from the file on disk.

A bot-fight mode or an earlier WAF rule can refuse Googlebot while the file allows it. Our hosting and DNS work is where that shows up: the proxy, the zone and the logs are one system.

Mistakes that cost Search or citations

The expensive mistake is a star group of Disallow: /. Googlebot, PerplexityBot and the search tokens then skip the host. The site thins out in Google. AI crawlers were refused, and so was every other agent that honours the file.

A Googlebot group that only says Allow: / replaces the star group, so staging stays open to Google. The mirror image is a pasted list of AI crawlers that includes Googlebot, or a Google-Extended line rewritten into a Googlebot disallow.

Blocking the quoting tokens, then waiting for citations, means the file is doing what it says. OAI-SearchBot governs ChatGPT search answers, PerplexityBot the page text in Perplexity, Claude-SearchBot Claude’s search. Training opt-outs are GPTBot, ClaudeBot and Google-Extended.

OpenAI says search can take about a day to follow a new file, and an index already built moves more slowly. Fetch the live URL on the host visitors use, then read the logs a few days later.

When to revisit the policy

Revisit when the business decision changes, and when a new token appears. A standing order to block all AI crawlers goes stale in the month a citation bot gets its own token and the star group allows it.

Also revisit for a new hostname, a language on its own host, or an app on a subdomain. robots.txt stops at the host boundary. A few weeks on, disallowed training tokens should be quiet, Googlebot should still fetch public URLs, and allowed search tokens should appear there.

How we work

1. Discovery. Which hosts are public, which paths are private, and whether the goal is citations, a training opt-out, or both.
2. Inventory. The live robots.txt on every hostname, any plugin copy behind it, and Cloudflare rules that can override the file.
3. Policy. A short token list a non-developer can approve, with the trade-off in one paragraph.
4. Implementation. The file at each public root, private paths in the star group, and AI Crawl Control aligned with that file.
5. Check. Logs and Search Console afterwards. Googlebot still fetches, training tokens drop, closed paths stay closed.
6. Review. A date to look again, or a trigger when a new token or hostname appears.

Why CodeWiz

• The file and the edge are one change. We build on WordPress, PHP and Node.js, with Cloudflare in front of the origin, so the same people edit robots.txt and the bot rule.
• Search stays in the conversation. The policy should leave Googlebot on the pages you want found. The wider practice is on our services page.
• AI work is already on the bench. We ship retrieval over a site’s own content, separate from a public training crawl.
• The file matches the site. A publisher who wants citations and a firm keeping manuals out of training approve different groups.
• A small studio. CodeWiz was founded in 2023 by Dragan Plavšić in Banja Luka. We work locally and abroad, remotely, and we explain the trade-off in plain language.

Frequently asked questions

Should we block all AI crawlers?

Block the training tokens when pages should stay out of future training sets. Leave OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot allowed when you want to be found and quoted. AI crawlers are a list of decisions.

Does disallowing GPTBot remove us from ChatGPT answers?

ChatGPT search answers follow OAI-SearchBot. GPTBot is the training crawl, and the groups are independent, so you can disallow GPTBot and allow OAI-SearchBot together. ChatGPT-User can still open a URL a person names.

If we block Google-Extended, do we leave Google Search?

Google Search stays. Google-Extended governs Gemini training and grounding in Gemini Apps and Vertex AI. Googlebot governs the Search index. Search Console’s generative AI control covers AI Overviews and AI Mode.

Is robots.txt enough to keep staging private?

Authentication, or a host kept off public DNS, keeps staging private. The disallow asks polite agents to skip the path and it also lists the path. Keep the line anyway, including for AI crawlers, so a brief leak is skipped by agents that honour the file.

How do we know crawlers are following the policy?

Fetch the live robots.txt, then read the access log or Cloudflare AI Crawl Control a few days later. Disallowed training tokens should be quiet, and Googlebot should still request public URLs. A token that keeps fetching a closed path needs a block at the edge.

Set a policy for AI crawlers

Send the hostnames and the outcome you want: quoted in answers, kept out of training, or both, with staging and logins closed. We read the live file, the Cloudflare rules and a slice of the log, and leave a robots.txt that matches the choice. Contact us to start.