AI Bot Tracking: Detect GPTBot, ClaudeBot, PerplexityBot

Contents

Key facts

all features

► Ingest · Bot identification

Know which AI is reading your content.

AI bot tracking for 25+ LLM crawlers across 19 providers.

A short list of regex signatures, one User-Agent header per request, one classification per hit. No third-party services in the path.

25

Bot signatures

19

AI providers

9

AI assistants (referral)

< 1ms

Overhead per request

Bot signatures

25 signatures · 19 providers

OpenAI

3

ChatGPT-UserGPTBotOAI-SearchBot

Anthropic

3

ClaudeBotclaude-webanthropic-ai

Perplexity

2

PerplexityBotPerplexity-User

Google

1

Google-Extended

ByteDance

1

Bytespider

CommonCrawl

1

CCBot

Meta

2

Meta-ExternalAgentFacebookBot

Amazon

1

Amazonbot

Cohere

1

cohere-ai

Diffbot

1

Diffbot

Apple

1

Applebot-Extended

You.com

1

YouBot

xAI

1

GrokBot

Mistral

1

MistralAI-User

Microsoft

1

Copilot bots

AI2

1

ai2bot

Phind

1

PhindBot

DuckDuckGo

1

DuckAssistBot

Kagi

1

KagiBot

How a request gets classified

Your installer sends one POST per request. We classify the UA, increment the daily aggregate, and return a count. No queues, no batching by default.

01

UA classifier

Regex pattern table, scanned in declaration order. More specific patterns (ChatGPT-User) win over general ones (GPTBot).

02

Daily aggregate

Per-day, per-bot, per-source row. One UPDATE per hit. No raw event log retained.

03

Referral classifier

On non-bot requests, the Referer header is matched against AI-assistant hosts (ChatGPT, Claude, Perplexity, Gemini, Copilot, etc.) and a separate aggregate is incremented.

AI assistants (referral tracking)

Different signal from crawlers. These show up in the Referer header when a real user clicks a citation in an AI answer.

ChatGPT, Claude, Perplexity, Gemini, Copilot, You.com, Le Chat, Phind, Doubao

Bot tiers

Every detected bot is classified into one of three tiers. The dashboard shows a By-tier traffic table, and tier chips follow each bot through every other view.

Training & data crawlers

GPTBot, ClaudeBot, CCBot, Bytespider

Bulk collection for model training. High volume, no immediate intent. This is how your content ends up inside next year's models.

Search & index crawlers

OAI-SearchBot, PerplexityBot, Google-Extended

Indexing for AI search products. A hit here can turn into a citation in an AI answer days later.

Live user agents

ChatGPT-User, Perplexity-User, MistralAI-User

A human is on the other end right now, asking their assistant about your site. The highest-intent traffic in the table.

The live-user tier also powers an alert: when those agents keep hitting action-bearing pages on a site with no WebMCP tools enabled, the dashboard flags it as a missed transaction opportunity.

Visibility scorecard

Three numbers built from your own crawl set and your own logs. Most "AI share of voice" tools sample a vendor-chosen list of prompts you can't inspect; we think that's unauditable, so the scorecard uses a denominator you can verify.

Every number on the scorecard expands to the raw rows behind it, and the whole thing exports to CSV.

// the detection layer

See every AI crawler the instant it lands — named, classified, and counted before it leaves the page.

Related features

Frequently Asked Questions

How do I know if GPTBot is crawling my site?

Three options: (1) Search your server access logs for User-Agent strings containing "GPTBot" — works for any site with raw logs; (2) Check robots.txt traffic in your CDN dashboard if you use Cloudflare or Fastly; (3) Install Crawlytics, which auto-classifies every GPTBot hit (and 24 other AI crawlers) the moment it arrives. Option 3 also catches ChatGPT-User (the live-fetch bot, separate from GPTBot) and OAI-SearchBot.

How do I block ChatGPT from crawling my website?

Add User-agent: GPTBot\nDisallow: / and User-agent: ChatGPT-User\nDisallow: / to your robots.txt. To block at the CDN layer (more reliable), add a bot rule in Cloudflare/Fastly/Vercel. Crawlytics shows you exactly what GPTBot is reading before you decide — many sites prefer to allow it and benefit from being cited in ChatGPT answers instead.

What is the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's training crawler — it scrapes your site a few times per week to collect content for future model training. ChatGPT-User is the live-fetch agent that runs when a user asks ChatGPT a question and ChatGPT needs to read a specific page right now. OAI-SearchBot is OpenAI's SearchGPT crawler. Crawlytics classifies all three separately.

How is a bot classified?

Each installer captures the request's User-Agent header and POSTs it to Crawlytics. The ingest endpoint runs a single regex check against the in-process bot table — under one millisecond. No third-party UA service in the path. New bot patterns are shipped as they emerge in the wild.

What gets stored per hit?

Per-day per-bot per-source aggregate counts. We never store IP addresses, cookies, session IDs, or any visitor identifier. There is no raw event log of individual requests.

Does tracking slow down my site?

No. Every installer fires the tracking event asynchronously after the response is sent. Your visitor gets the response before the event leaves your server.

Cite this page

Related on this site

This page is part of Crawlytics.app. View all pages: llms.txt · llms-full.txt

Site index for AI agents: llms.txt · sitemap