Google Analytics filters out bot traffic and can't see what AI crawlers do on your site. Here is where Crawlytics complements GA, and where each tool wins.
You should run both. Google Analytics measures human behavior; Crawlytics measures AI behavior. They answer different questions and they complement, not compete.
Where it gets interesting is the overlap zone — AI assistants driving real human visits to your site. GA misses most of this (it ends up in "(direct)"). Crawlytics catches it. If you only run GA, you have a large and growing blind spot in your acquisition data.
Google Analytics 4 is excellent at:
If you're optimizing your funnel for human conversion, GA is the right tool. None of what's below is a criticism of that use case.
Three concrete gaps:
GA4 has a setting called "exclude all hits from known bots and spiders" and it's on by default. Even if you turn it off, GA's JavaScript tag only fires in real browsers. AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) don't execute JavaScript. The tag never loads. The bot visit never lands in GA.
Net result: Google Analytics shows you ~0% of your AI crawler traffic, because the data pipeline can't see it. The full detection playbook — which UAs to grep for, the live-fetch vs training-crawler distinction, and benchmarks for what a healthy bot-to-human ratio looks like — is in our piece on how to track AI citations.
Even when an AI assistant drives a real human visitor to your site, GA usually fails to attribute it correctly. Why: ChatGPT, Claude, and Perplexity's mobile and in-app browsers strip the Referer header on outbound clicks. GA sees a visit with no source and buckets it as (direct) / (none).
This is the most under-discussed measurement problem in marketing right now. A typical mid-size SaaS site might have 5-15% of its "direct traffic" actually originating from AI assistants. Full write-up here.
Even if GA could see AI traffic, it doesn't have the right schema for it. GA knows about sessions, page views, conversions, and channels but not:
/llms.txt is being fetched and by whomThese questions don't map to GA's data model at all. You can't build a custom report or property to answer them, because the underlying events never enter GA's pipeline.
| Question | GA4 | Crawlytics |
|---|---|---|
| How many sessions did real human visitors have last week? | ✓ Excellent | — |
| Which content is converting? | ✓ Excellent | — |
| What % of visits came from Google organic vs paid vs social? | ✓ Excellent | — |
| Which AI bots are crawling my site right now? | ✗ Not captured | ✓ Real-time, per-bot |
| How often is GPTBot fetching my pricing page? | ✗ Not captured | ✓ Per-page time series |
| Did Perplexity drive any real human visitors this month? | ~ Shows in "direct" — mis-attributed | ✓ Per-LLM UTM attribution |
| Which AI assistants are citing me most? | ✗ Not captured | ✓ Per-source breakdown |
| Is my /llms.txt being fetched? | ✗ Not captured | ✓ Per-bot fetch log |
| How does my AI bot traffic trend compare to last month? | ✗ Not captured | ✓ Date-range compare |
| Are AI agents transacting (checkout, leads, bookings)? | ~ Sees the conversion, can't attribute to agent | ✓ WebMCP-level conversion attribution |
The cleanest setup uses both:
chatgpt / ai_referral, claude / ai_referral, etc. — recoverable in any GA report that respects UTM params.This means you don't lose anything by adding Crawlytics. Your GA reports get more accurate (because previously-"direct" AI traffic now shows its real source), and you gain a whole new analytics surface for AI-specific behavior that GA was never going to provide.
Cloudflare Radar shows aggregated, industry-wide AI bot traffic distribution. It's a great public reference. But it's not per-customer — you can't see which pages on your site GPTBot is reading, how often, or whether the trend is up or down.
For per-customer analytics with the same depth that GA provides for human traffic, you need a dedicated tool. Full Cloudflare comparison here.
| GA4 | Crawlytics Visibility | Crawlytics Commerce | |
|---|---|---|---|
| Monthly price | Free (with 360 upgrade ~$150k/yr enterprise) | $29.99 | $49.99 |
| Human session analytics | ✓ | — | — |
| AI bot tracking | — | ✓ | ✓ |
| AI referral attribution | ~ broken on in-app | ✓ | ✓ |
| llms.txt generation | — | ✓ | ✓ |
| WebMCP agent commerce | — | — | ✓ |
You probably need both if any of these are true:
If none of those apply, GA alone is probably fine for another quarter or two but the trend lines say "another quarter or two" is roughly the lifespan of that statement.
Written by Crawlytics Team. Crawlytics tracks AI bots, generates llms.txt, and powers WebMCP commerce, all from one snippet on any stack. See how it works →
Cloudflare Radar shows aggregated, industry-wide AI bot traffic distribution. It's a great public reference. But it's not per-customer — you can't see which pages on your site GPTBot is reading, how often, or whether the trend is up or down. For per-customer analytics with the same depth that GA provides for human traffic, you need a dedicated tool. Full Cloudflare comparison here.
No. Crawlytics doesn't track human page views, sessions, conversions, or any of the things GA4 is good at. It tracks AI-specific events GA4 can't see. Use both.
Yes, in a good way. Crawlytics' UTM injection means previously-"direct" AI traffic starts showing up in GA with proper source attribution (chatgpt, claude, etc.). Your channels report gets more accurate. Nothing else about your GA setup changes.
No. The exclude-bots setting is unrelated. GA can't see AI bots because they don't run JavaScript, so the GA tag never fires. Toggling the setting won't help.
Not directly today. The export-to-CSV/JSON endpoint that would feed Looker is on the roadmap. For now, the in-product dashboard is the main reporting surface.
This page is part of Crawlytics.app. View all pages: llms.txt · llms-full.txt