How to verify AI crawlers and catch bots that spoof them
Any scraper can put GPTBot in its user-agent string. Here is how to verify an AI crawler's real identity with DNS checks, and what to do when you can't confirm it.
Every major AI answer engine now sends a crawler to read your site: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Google. Sites use that identity to decide who gets in, blocking generic scrapers while allowing the crawlers that might cite them later. The problem is that a user-agent string is just a header. Anyone can set it to say GPTBot, whether the request comes from OpenAI's infrastructure or a scraper on a rented server.
Why crawler identity checks matter
If your robots.txt or firewall treats requests differently based on the user-agent string alone, that rule is only as strong as the string is honest. A scraper that wants your content without permission can copy a known AI bot's user-agent and walk straight through a rule meant to welcome only vetted crawlers. A site that blocks by user-agent name can also end up blocking a real crawler that changed its string slightly, while a spoofed one slips through untouched. Either mistake changes who actually gets to read and cite your content.
How to verify a crawler's real identity
Two checks catch most spoofed traffic, and they are the same ones sites have used to verify Googlebot for years.
- Run a reverse DNS lookup on the request's IP address.
- Check that the hostname it returns belongs to the vendor's domain, for example a googlebot.com or an openai.com subdomain.
- Run a forward DNS lookup on that hostname and confirm it resolves back to the same IP address you started with.
- If either step fails, the request did not come from the vendor it claims to be.
This is called forward-confirmed reverse DNS, and it holds up better than trusting a published IP list on its own, because IP ranges change over time. Several AI vendors publish the ranges their crawlers use, and checking a request's IP against that list is a fast first pass. Treat the list as a starting point, not a permanent record: pull the current version before you rely on it instead of hardcoding a range you found once.
What the major AI crawlers publish
- OpenAI: GPTBot, ChatGPT-User, and OAI-SearchBot each have their own published IP ranges and reverse DNS pattern.
- Google: Google-Extended follows the same verification path long used for Googlebot.
- Anthropic: ClaudeBot identifies itself by user-agent, which makes reverse DNS the more dependable check.
- Perplexity: PerplexityBot has been reported crawling from IPs outside its declared ranges and ignoring robots.txt, so verify before you trust the label.
A user-agent string is a claim, not a credential. Only DNS and IP checks confirm it.
What to do when you can't confirm a bot
Not every site can run a DNS check on every request in real time. If you're relying on log review instead, watch behavior rather than the name in the log line. A real AI crawler tends to respect robots.txt, spread requests out, and follow a predictable path through your site. A spoofed one often ignores disallow rules, hits the same pages repeatedly in a short window, or requests pages a legitimate crawler has no reason to fetch, like admin routes or odd query-string combinations.
- Rate-limit by IP regardless of the user-agent a request claims.
- Flag requests that claim to be a known bot but originate from a generic hosting provider instead of the vendor's own ranges.
- Review your logs on a schedule instead of trusting a rule you set once and forgot about.
You don't need to verify every request by hand. Set up forward-confirmed DNS checks, or at least a maintained IP allowlist, for the AI crawlers you actually want to let in. Log the rest, and revisit the list whenever a vendor changes its ranges. That's the difference between a rule that controls who reads your content and one that only controls what a visitor calls itself.