Does blocking GPTBot keep your site out of ChatGPT's answers?

No. GPTBot is the training crawler. Whether ChatGPT can cite you is decided by a different token, OAI-SearchBot, and OpenAI documents the two as separate decisions. We read the robots.txt of 24 sites on 4 November 2026. Of the five publishers that blocked GPTBot outright, three left OAI-SearchBot open, so their text is out of the training set and still eligible for the answers. Two closed both.

Which OpenAI crawler decides whether ChatGPT can cite you?

OpenAI gives each token one sentence. "OAI-SearchBot is for search," and it "is used to surface websites in search results in ChatGPT's search features." Sites opted out of it "will not be shown in ChatGPT search answers," although they "can still appear as navigational links." GPTBot is the other decision entirely: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." ChatGPT-User is the fetch that fires when a person asks for a page, and OpenAI states that it "is not used to determine whether content may appear in Search." Because a person asked for it, "robots.txt rules may not apply." Two of the three lines have nothing to do with being cited. The mechanism is sitting in the token names: retrieval and training arrive as different crawlers, so one file holds two switches. The same page puts propagation at about 24 hours after a robots.txt edit. Source: OpenAI, Overview of OpenAI crawlers, checked 4 November 2026.

Where is the same switch on Claude, Perplexity, Google and Copilot?

In five different places, and only three of them are robots.txt.

Engine	What keeps you eligible for the answer	What the training opt-out is
ChatGPT search	OAI-SearchBot allowed in robots.txt	GPTBot
Claude	Claude-SearchBot and Claude-User allowed	ClaudeBot
Perplexity	PerplexityBot allowed	none published
Google AI Overviews and AI Mode	indexed and snippet-eligible for Googlebot, plus the Search Console setting	Google-Extended
Microsoft Copilot	no NOARCHIVE meta tag on the page	not published as a crawler token

Anthropic splits its three agents the same way OpenAI does: ClaudeBot collects web content for the models, while "Claude-SearchBot navigates the web to improve search result quality for users." Perplexity has no second crawler to block. PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models." A publisher who writes PerplexityBot into a blocklist to protect its text from training has protected nothing. It has closed the citation.

Where did Google move the control?

Out of the file. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search," per Google's crawler documentation. To be a link under an AI Overview or in AI Mode, a page has to be "indexed and eligible to be shown in Google Search with a snippet," per Google's AI features page, which names robots.txt directives for Googlebot as the access control and points to snippet tags for anything finer.

The citation switch is now a Search Console setting. Google announced it on 3 June 2026 for a subset of UK site owners and added a footnote dated 31 August 2026 saying the features had rolled out "to all websites worldwide." Its effect, in Google's words: "Sites that opt out will not receive traffic or impressions from our generative AI features," and the control "will not be used as a ranking signal for search results outside of these generative AI Search features." Microsoft's equivalent is older and lives in a meta tag. "Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers," while a page carrying it "will still appear in our search results." Source: Bing Webmaster Blog, September 2023.

What do real robots.txt files do with these tokens?

We requested /robots.txt from 37 domains on 4 November 2026 and got a readable file from 24. Thirteen refused our fetcher, most of them large newspapers, so read what follows as a sample of sites that tolerate an automated request. Ten of the 24 are publisher or reference sites. Fourteen are software company sites, the kind a buyer asks an engine to recommend.

Site	GPTBot	OAI-SearchBot	ClaudeBot	Claude-SearchBot	PerplexityBot
forbes.com	Disallow /	Allow /	Disallow /	Allow /	Disallow /
techcrunch.com	Disallow /	not named	Disallow /	not named	not named
axios.com	Disallow /	not named	Disallow /	not named	Disallow /
cnn.com	Disallow /	Disallow /	Disallow /	Disallow /	Disallow /
cnbc.com	Disallow /	Disallow /	Disallow /	Disallow /	Disallow /
washingtonpost.com	not named	not named	Disallow /	not named	Disallow /
en.wikipedia.org	path rules	path rules	path rules	not named	path rules
nature.com	not named	not named	not named	not named	not named
github.com	not named	not named	not named	not named	not named
engadget.com	not named	not named	not named	not named	not named

Forbes made the distinction on purpose. GPTBot and ClaudeBot get Disallow: /, and OAI-SearchBot and Claude-SearchBot get an explicit Allow: / a few lines later. Then the same file blocks PerplexityBot, which is a search crawler and the only one Perplexity has. In these ten files Perplexity's crawler is shut out five times and OpenAI's search crawler twice.

The software sites went the other way. Twelve of the fourteen name no AI crawler at all, which leaves every retrieval crawler allowed by default. monday.com names six AI tokens and gives them path-level rules instead of a blanket block. cloudflare.com names four and allows them. Not one of the fourteen blocks a search crawler. The companies with the most to gain from being recommended have the right setting because nobody went near the file.

A larger count points the same way. An October 2026 dataset of 285 major websites published by AI Ranked read 261 robots.txt files and found 35% blocking at least one training crawler against 21% blocking at least one search crawler, with OAI-SearchBot the least blocked token at 11.5%. It is a vendor dataset over a hand-built list rather than a random sample, and its own page says so.

Why does naming a crawler sometimes unblock it?

Group matching. RFC 9309 section 2.2.1 says that "if no matching group exists, crawlers MUST obey the group with a user-agent line with the * value, if present." A group naming GPTBot therefore replaces the wildcard group for GPTBot rather than adding to it. Write four restrictive lines under User-agent: * and a two-line GPTBot group, and GPTBot now has more access than an unnamed bot does.

The reverse trap catches more sites. A file with User-agent: * and Disallow: / and no AI token anywhere has already blocked every search crawler in the table above, because each one falls back to the wildcard. Nobody had to type OAI-SearchBot to remove it. The file says nothing about AI and settles the question anyway.

What if your CDN writes the file for you?

Then your AI visibility is somebody else's default. Cloudflare's managed robots.txt covers "over 3.8 million domains" and, since 24 September 2025, writes Content-Signal: search=yes, ai-train=no into them. The signal that matters for AI answers is left blank on purpose: "We will not serve an ai-input signal for our managed robots.txt customers. We don't know their preference with respect to that signal, and we don't want to guess." Cloudflare also states the limit plainly, which is more than most of the blocklist guides do: "content signals express preferences; they are not technical countermeasures against scraping."

Who does this arrangement suit? The publisher with a licensing deal, a legal team, a syndication contract and somebody whose job includes reading crawler release notes. It costs the five-person vendor whose robots.txt arrived with the CMS and whose eligibility in three engines now rests on a default picked by a CDN. The bill arrives as a citation that never happened.

What should you check before Monday?

Open your own /robots.txt in a browser. Read it for the four search tokens rather than the training ones, and work out which group each crawler lands in, because that is where the surprises are. Then the Search Console setting, which has been live worldwide since August and may already be set. Last, grep the templates for NOARCHIVE, usually left behind by a paywall configuration that nobody revisited.

Our own robots.txt names 18 AI crawler tokens and allows all of them, and you can read the whole decision in one file at unity-horizon.com/robots.txt. Two of those names, anthropic-ai and Claude-Web, are not among the three agents Anthropic's current page documents. Part of our list is addressed to crawlers that stopped answering to it. We read files here and did not test whether each engine honours what its own page says, which is the next measurement and a harder one.

Closing one engine is not a third of the loss. When we asked eight provider questions three times each across three engines for Do ChatGPT, Google AI Mode and Perplexity recommend the same companies?, 85% of the 164 businesses named came from one engine only. The engine you blocked was keeping its own list. Our AI visibility check reads a site and asks the engines what they say about it.

Which of the five switches above is set on your domain today, and who set it?