Know which AI is reading your site
Crawlytics tracks every time GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers access your content. Real-time analytics for the age of AI search.
Last 24 hours
Overview · acme.com
Bot requests
0
+0% vs prior
Pages crawled
0
+12% vs prior
Avg latency
0.4ms
−8% vs prior
Unique bots
14
+2 vs prior
Request volume
Top bots
share of requests
GPTBot
OpenAIClaudeBot
AnthropicPerplexityBot
PerplexityGoogle-Extended
GoogleLive activity
Tracking bots from
How it works
Three steps to AI visibility
No code changes to your site. Up and running in under 5 minutes.
Point to your site
Provide your URL and sitemap. Crawlytics crawls your pages and converts them to clean markdown — llms.txt, llms-full.txt, and per-page .md files.
Choose your integration
Drop in a one-line middleware, edge function, or tracking snippet. Works with Cloudflare, Vercel, Netlify, WordPress, Express, and more — no nginx required.
See who is reading
Every LLM bot request is logged with bot name, company, timestamp, and the page they accessed. Full analytics dashboard in real time.
Features
Everything you need to understand AI traffic
From content generation to real-time analytics, Crawlytics gives you full visibility.
LLM Bot Identification
Automatically detect 20+ AI crawlers including GPTBot, ClaudeBot, PerplexityBot, and more.
Learn moreMarkdown Generation
Auto-generate llms.txt, llms-full.txt, and per-page .md files from any website via sitemap.
Learn moreAnalytics Dashboard
Real-time traffic charts, bot breakdowns, top pages, and provider share — all in one view.
Learn moreMulti-Site Support
Monitor all your websites from a single dashboard. Each site connects independently.
Learn moreDate Range Comparison
Compare any two time periods to track growth. See % change across all metrics.
Learn moreSecurity First
Encrypted API keys, SSRF protection, rate limiting, IP hashing, and localhost binding.
Learn moreIntegrations
Works with your stack
One-line integration for any platform. No server access required.
0+
LLM bots tracked
0
Tracking backends
0+
Pages per site
<0ms
Tracking overhead
Start tracking AI traffic today
Set up in under 5 minutes. No code changes to your site. Works with any CMS or platform.
FAQ
Frequently asked questions
Crawlytics is an analytics platform that tracks which AI models and LLM crawlers are reading your website, and which AI assistants are sending real users to your pages via citations. You drop in one installer snippet (or upload access logs) and the dashboard shows you who is reading, what pages they care about, and how AI search referrals are trending.
Crawlytics identifies 25+ LLM crawlers including GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, Bytespider (ByteDance), CCBot (Common Crawl), Meta-ExternalAgent, Amazonbot, Applebot-Extended, YouBot, GrokBot (xAI), and more. New bot patterns are shipped as they emerge.
Yes. We ship installer snippets for Cloudflare Workers, Vercel Edge / Next.js middleware, Express, WordPress, nginx, and Apache — plus a log-import path that works on any web server. WordPress, Shopify, Squarespace, custom builds, static sites are all supported.
No reverse proxy or DNS changes required. Add the site to the dashboard to get a unique tracking ID, then paste the snippet for your stack into your site. Events stream in real time. If you can't add code, upload an nginx or Apache access log and Crawlytics parses it and ingests the bot hits.
Yes. Crawlytics never stores visitor IPs (not raw, not hashed), drops no cookies, and embeds no third-party trackers. Hit data is rolled up into per-day per-bot aggregates the moment it arrives — there is no raw event log. Login passwords use bcrypt; all traffic is TLS.
JSON and CSV export endpoints are on the immediate roadmap. Once shipped you can point Looker Studio (via the Community Connector) or any BI tool at the same data the dashboard renders. Native BigQuery / Google Sheets streaming is a longer-term integration.