
Can you prove which AI crawler read your page?
Unity Horizon7 min read
Only by source address. The user-agent is typed by the client, and OpenAI's list for the crawler that decides ChatGPT citations was built 2 January 2026.
Only by its source address, and for one of them you cannot do it well. The user-agent string is typed by the client, so a log line reading OAI-SearchBot/1.4 is a claim rather than an identity. We fetched every address list that OpenAI, Anthropic, Perplexity, Google and Microsoft publish for their crawlers on 18 November 2026. Google's files were rebuilt the day before. OpenAI's file for the crawler that decides whether ChatGPT can cite you carries a creation date of 2 January 2026.
What does an AI crawler's user-agent prove?
Nothing that survives a check. We sent one GET to twelve well-known home pages with OAI-SearchBot/1.4 in the user-agent, from a GitHub Actions address that appears in no vendor file, then repeated each request with a desktop Chrome string. The status code came back identical in all twelve pairs: six 200s, three 403s, three redirects. Six sites served our fake crawler the page. The three that refused us refused the browser too, so that was an address being turned away and not a name.
The vendors are direct about this. OpenAI's crawler documentation tells site owners to allow "requests from our published IP ranges". Perplexity's bot page recommends you "combine both User-Agent string matching and IP address verification for enhanced security". Anthropic's support article puts the whole check in one sentence: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic."
The name is typed. The address is routed.
Which address lists do the answer engines publish?
All five publish something, and the files are not comparable. We read each one on 18 November 2026 and counted the prefixes in it, the IPv4 addresses those prefixes cover, and the creationTime the file states about itself.
| Crawler | What its fetch decides | Prefixes | IPv4 addresses | File built | Days old |
|---|---|---|---|---|---|
| OAI-SearchBot | whether ChatGPT search can show you | 39 | 2,560 | 2026-01-02 | 320 |
| ChatGPT-User | nothing about search inclusion | 237 | 69,312 | 2026-11-06 | 11 |
| GPTBot | training only | 18 | 2,704 | 2026-09-22 | 57 |
| All Anthropic bots | training, search and user fetches together | 14 | 1,037 | 2026-10-20 | 29 |
| PerplexityBot | whether Perplexity can link you | 13 | 13 | 2026-10-28 | 20 |
| Perplexity-User | one answer, for one person | 10 | 10 | 2026-10-28 | 20 |
| Googlebot | Search, AI Overviews and AI Mode | 317 | 5,088 | 2026-11-17 | 1 |
| bingbot | the Bing index | 29 | 4,752 | 2024-01-03 | 1,050 |
Google is the exception twice over. It rebuilds its four crawler files daily, and it documents reverse DNS verification as well, where the lookup has to land on googlebot.com, google.com or googleusercontent.com and then resolve forward to the same address. Nobody else offers a second method.
The address column also hides something. It counts IPv4 only, because Google's file is the only one with IPv6 in it, 147 prefixes worth. The other four publish none. A request that arrives over IPv6 wearing the name of any of those crawlers matches nothing you can check, and in 2026 that is no longer an exotic case.
Why can you not tell Anthropic's training crawler from its search crawler?
Because they share a file. Anthropic documents three agents with three different jobs: ClaudeBot collects content "that could contribute to model training", Claude-SearchBot "navigates the web to analyze content and improve the relevance and accuracy of search responses", and Claude-User fetches a page when a person asks Claude something. One address file covers all three, 14 prefixes and 1,037 addresses on the day we read it.
Now take the arrangement a publisher usually asks for: refuse the training crawler, keep the search crawler. One signal separates those two, and the client writes it. The same article warns that "alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly", and sends you back to robots.txt, which is enforced on the name as well. So the address tells you the company and stops there.
Why is the list for OpenAI's search crawler the oldest one?
We cannot tell you, and the file does not say. What the two files show side by side is an odd division of maintenance. OAI-SearchBot, the token that decides whether ChatGPT's search can surface a site, is verified against 39 prefixes built on 2 January 2026. ChatGPT-User, which OpenAI's own page says "is not used to determine whether content may appear in Search" and for which robots.txt "may not apply", gets 237 prefixes covering 27 times the address space, rebuilt 11 days before we looked.
The cost of the stale file lands as a false negative, and it is quiet. Allowlist the January addresses at your edge, or rate-limit everything outside them, and any capacity OpenAI has added since gets throttled while your log shows a crawler politely backing off. We covered the name half of this problem in Does blocking GPTBot keep your site out of ChatGPT's answers?, where twelve of fourteen software company sites named no AI crawler in robots.txt at all. The address half is worse. The right setting on your side now depends on a JSON file somebody else has to keep current.
Does a signature fix AI crawler verification?
It will, and today it covers the wrong bots. Web Bot Auth asks a crawler to sign its requests with an Ed25519 private key and publish the public keys at /.well-known/http-message-signatures-directory. That takes the address list out of the problem. Cloudflare verifies those signatures at its edge now, with documentation last updated on 8 October 2026, and the three headers it requires are Signature, Signature-Input and Signature-Agent. The protocol itself is an IETF Internet-Draft, revision 01 of 29 October 2026, written by one author at Cloudflare and one at Google, expiring on 2 May 2027.
Then read who signs. Google's Web Bot Auth page calls the implementation "currently experimental", states that "a subset of requests made by the Google-Agent are signed", and adds that "not all Google user agents are using Web Bot Auth". Its directory at agent.bot.goog held five keys when we fetched it. chatgpt.com serves a directory too, one Ed25519 key with "purpose":"ai" and an expiry of 25 November 2026, a week after our check, and OpenAI's crawler page does not mention signatures anywhere. Signed traffic today is the agent clicking on behalf of a person. The crawler that files you in the index still arrives anonymous.
Who does counting user-agents suit?
It suits whoever sells the count. A dashboard of AI crawler hits is three lines of log parsing, the number only goes up as more bots appear, and no engine publishes fetch counts of its own that could contradict it. It suits the large group whose CDN verifies bots at the edge, because that check arrives as a product feature and somebody else maintains the lists behind it.
It costs the firm of five running its own nginx. PerplexityBot's entire indexing fleet is thirteen single addresses, so one per-address rate limit applied to a shared range slows the refresh of Perplexity's index of your site, and the symptom is a stale answer about your pricing four weeks later. It costs anyone who allowlisted bingbot from a file built on 3 January 2024 and has not opened it since. Both of them pay in crawls that never happened, and no log on earth records those.
What should you check before Monday?
Grep last month's access log for four tokens, not one: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. For each hit, match the source address against the vendor file above, and for Google run the reverse and forward DNS pair instead. Then keep the two counts that matter, matched and unmatched. An unmatched hit is either a stale vendor file or somebody wearing the name, and you cannot tell which from the string alone.
We cannot settle the OpenAI question for you, because we do not run a site with enough AI crawler traffic to count how many OAI-SearchBot fetches arrive from addresses outside that January file. We found no published count either, and that is the number that would say whether 320 days is a problem or a file that needed no change. If you have the logs, we would take the count. Our own AI visibility check works from the other side of the exchange: it reads a site, writes the questions a customer would ask, asks ChatGPT, Google AI Mode and Perplexity, then scores visibility, accuracy and website readiness out of 100. It reads answers rather than logs.
Which address fetched the page an answer engine last quoted back to you, and does it appear in any file the vendor publishes?
Common questions
How do you verify that an AI crawler is really from OpenAI, Anthropic, Perplexity or Google?
By matching the request's source IP address against the list the vendor publishes, because the user-agent string is set by the client and anyone can send it. The files are openai.com/searchbot.json, claude.com/crawling/bots.json, perplexity.ai/perplexitybot.json and Google's four files under developers.google.com/static/crawling/ipranges/. Google additionally documents reverse DNS verification against googlebot.com, google.com or googleusercontent.com, and it is the only one of the four that does.
Can you tell ClaudeBot from Claude-SearchBot by IP address?
No. Anthropic publishes one file at claude.com/crawling/bots.json covering all of its bots, with 14 prefixes and 1,037 IPv4 addresses when we checked it on 18 November 2026. Its training crawler, its search crawler and its user-triggered fetcher all arrive from the same address space, so the only thing separating them is the user-agent string, which is the part you cannot verify.
Does OpenAI publish IP ranges for OAI-SearchBot?
Yes, at openai.com/searchbot.json, and OpenAI's crawler documentation tells site owners to allow requests from those ranges. On 18 November 2026 the file held 39 IPv4 prefixes covering 2,560 addresses, and its own creationTime field read 2 January 2026, 320 days earlier. OpenAI's file for ChatGPT-User, the fetcher that cannot affect search inclusion, had been regenerated 11 days before we looked.
Does Web Bot Auth solve AI crawler verification?
Not yet for the crawlers that decide citation. Web Bot Auth signs requests with Ed25519 keys published at /.well-known/http-message-signatures-directory, Cloudflare verifies those signatures at its edge today, and the IETF protocol document is still an Internet-Draft, revision 01 of 29 October 2026. Google calls its own implementation experimental and signs a subset of Google-Agent requests. No search or indexing crawler is documented as signing.


