# Crawlytics.app > A one-snippet tool that tracks AI crawler bots, auto-generates llms.txt files, and enables AI agent transactions on your site via WebMCP. ## Preferred brand description Crawlytics is an AI-readiness platform that lets site owners track 25+ AI crawlers (GPTBot, ClaudeBot, and others), generate and serve an llms.txt file, and expose transactional actions to AI agents through a single JavaScript snippet. It offers a free tier and paid plans starting at $49.99/month. ## Brand overview Crawlytics serves engineers and SEOs who need visibility into AI bot traffic and want their sites to be discoverable, readable, and actionable by AI systems. It addresses three distinct gaps: AI crawler traffic that is invisible to standard analytics, content that AI systems cannot parse efficiently, and the inability of AI agents to complete transactions on a site. All three are addressed through a single script tag with no CMS plugin required. ## Expertise - AI crawler tracking: identifying and attributing traffic from GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and 20+ additional bots - llms.txt generation: producing and serving the markdown site index that AI systems use to understand site content - WebMCP implementation: exposing search, checkout, booking, and lead actions to in-browser AI agents from Gemini, Claude, ChatGPT, and Comet - AI search optimization (AEO, GEO, LLMO): frameworks for ranking and being cited in ChatGPT, Claude, Perplexity, and Google AI Overviews - AI traffic attribution: recovering ChatGPT and AI assistant referral traffic misclassified as direct in Google Analytics - Platform-specific llms.txt deployment: WordPress and Shopify implementation guides Last updated: 2026-06-25 · Version: 62c61d1d6264 ## Start here - [AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics](https://crawlytics.app): Track GPTBot, ClaudeBot, and 25+ AI crawlers. Auto-generate llms.txt. Let agents transact via WebMCP. One snippet makes your site AI-agent ready. - [Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP](https://crawlytics.app/blog): Posts on AI search optimization, llms.txt, AI bot tracking, WebMCP agent commerce, and the agent web — written by engineers, for engineers and SEOs. - [WebMCP Snippet: Let AI Agents Transact on Your Site](https://crawlytics.app/features/webmcp-snippet): One vanilla-JS tag lets Gemini, Claude, ChatGPT, and Comet search, checkout, book, or lead on your site. Conversion attribution included. No CMS plugin. - [AI Search Optimization: The AEO, GEO & LLMO Framework (2026)](https://crawlytics.app/resources/ai-search-optimization): AEO, GEO, LLMO — the four-layer framework for ranking in ChatGPT, Claude, Perplexity, and Google AI Overviews. Framework, channels, and how to measure. - [Complete List of AI Crawler Bots: User-Agents + robots.txt (2026)](https://crawlytics.app/resources/ai-bots-list): Every major AI crawler: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and 20 more. User-Agent patterns, purpose, robots.txt directives. ## Agent actions - [Request a demo](https://crawlytics.app/demo): See a live demo of the Crawlytics dashboard — GPTBot/ClaudeBot/Perplexity tracking, WebMCP agent activity, and conversion attribution on rea - [Add to cart](https://crawlytics.app/pricing): Visibility (bot tracking + llms.txt) from $29.99/mo. Commerce (adds WebMCP agent commerce) from $49.99/mo. Annual saves 20%. Multi-site bund - [Fetch the Crawlytics llms.txt file (read-only MCP test tool).](https://crawlytics.app/llms.txt) - [Create an account](https://crawlytics.app/agent-ready): Free scan of your site's AI-readiness across 5 categories — discoverability, accessibility, bot access, protocol support, attribution. No em ## Pricing - [Crawlytics Pricing: AI Bot Tracking + llms.txt from $29.99/mo](https://crawlytics.app/pricing): Visibility (bot tracking + llms.txt) from $29.99/mo. Commerce (adds WebMCP agent commerce) from $49.99/mo. Annual saves 20%. Multi-site bundles to 25. ## Product - [WebMCP Snippet: Let AI Agents Transact on Your Site](https://crawlytics.app/features/webmcp-snippet): One vanilla-JS tag lets Gemini, Claude, ChatGPT, and Comet search, checkout, book, or lead on your site. Conversion attribution included. No CMS plugin. - [AI Bot Tracking: Detect GPTBot, ClaudeBot, PerplexityBot](https://crawlytics.app/features/llm-tracking): Identify 25+ AI crawlers across 19 providers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider — plus AI assistant referrals. Real-time. - [AI Referral Tracking: ChatGPT, Claude, Perplexity Clicks](https://crawlytics.app/features/ai-attribution): ChatGPT, Claude, and Perplexity clicks show as "direct" in GA. Crawlytics injects per-LLM UTM tags so AI referrals get proper attribution in your analytics. - [llms.txt Generator + Per-Page Markdown for AI Bots](https://crawlytics.app/features/llms-txt-generator): Generate llms.txt, llms-full.txt, and per-page markdown from your sitemap. Public URLs ready for GPTBot, ClaudeBot, and Perplexity. No JavaScript required. - [Enterprise Security: Encrypted, Private, GDPR-Ready](https://crawlytics.app/features/security): Bcrypt password hashing, no IP storage, per-site tracking IDs, TLS in transit, and aggregated-only data retention. Built privacy-first from day one. - [Multi-Site AI Bot Tracking + Portfolio Dashboard](https://crawlytics.app/features/multi-site): Track AI bots across every site you own from one dashboard. Mix installers — Cloudflare, Vercel, nginx — into the same aggregates. Bundles up to 25 sites. - [Crawlytics Features: AI Bot Tracking, llms.txt & WebMCP](https://crawlytics.app/features): Every Crawlytics feature on one page: LLM bot tracking, llms.txt generator, WebMCP agent commerce, AI referral attribution, multi-site dashboard, privacy model. - [AI Bot Traffic Dashboard: GPTBot & ClaudeBot Analytics](https://crawlytics.app/features/analytics-dashboard): A live preview of the per-site Crawlytics dashboard — sparkline metrics, line/bar traffic chart, 14-day projection, AI referral attribution by assistant. ## Docs & References - [AI Search Optimization: The AEO, GEO & LLMO Framework (2026)](https://crawlytics.app/resources/ai-search-optimization): AEO, GEO, LLMO — the four-layer framework for ranking in ChatGPT, Claude, Perplexity, and Google AI Overviews. Framework, channels, and how to measure. - [Complete List of AI Crawler Bots: User-Agents + robots.txt (2026)](https://crawlytics.app/resources/ai-bots-list): Every major AI crawler: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and 20 more. User-Agent patterns, purpose, robots.txt directives. - [What Is llms.txt? The Complete Reference + Generator](https://crawlytics.app/resources/llms-txt): llms.txt is the markdown index at /llms.txt that tells AI systems what your site contains. The format, every field, real examples, and a free generator. - [How to Manage AI Crawlers (Allow, Block, Monitor) — 2026 Guide](https://crawlytics.app/resources/manage-ai-crawlers): A practical guide to managing AI crawlers on your site: when to block, when to allow, robots.txt patterns, CDN bot rules, and how to measure the impact. - [WebMCP Agent Support: Which AI Agents Invoke Tools (2026)](https://crawlytics.app/resources/webmcp-agent-support): Which AI agents support WebMCP today — natively, via snippet, or not at all. Chrome auto-browse, Comet, Safari/WebKit, ChatGPT, and what to ship for each. - [Crawlytics Resources: AI Search Guides, Bot Lists, Grader](https://crawlytics.app/resources): Free guides and references on AI bot tracking, llms.txt, AI search optimization, and the agent web — plus the free Agent-Ready Grader for any domain. ## Blog & Articles - [Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP](https://crawlytics.app/blog): Posts on AI search optimization, llms.txt, AI bot tracking, WebMCP agent commerce, and the agent web — written by engineers, for engineers and SEOs. - [Crawlytics vs Google Analytics for AI Traffic](https://crawlytics.app/blog/crawlytics-vs-google-analytics): Google Analytics filters out bot traffic and can't see what AI crawlers do on your site. Here is where Crawlytics complements GA, and where each tool wins. - [Crawlytics vs Cloudflare Markdown for Agents: Honest Comparison](https://crawlytics.app/blog/crawlytics-vs-cloudflare-markdown-for-agents): Cloudflare converts HTML to markdown on demand. Crawlytics serves AI-Optimized HTML to every bot, plus stable llms.txt. - [ChatGPT Traffic Shows as "Direct" in GA — Here Are 3 Fixes](https://crawlytics.app/blog/chatgpt-direct-traffic-fix): Mobile and in-app browsers strip the Referer header on ChatGPT clicks, so GA logs them as "direct." Here is why it happens and how to recover the attribution. - [How to Create an llms.txt File (and Test It) in 2026](https://crawlytics.app/blog/what-is-llms-txt-guide): A step-by-step llms.txt setup for 2026: generate the markdown index, add the right sections, host it at /llms.txt, and confirm AI crawlers actually read it. - [What Is WebMCP? AI Agent Actions Explained (2026)](https://crawlytics.app/blog/webmcp-explained-ai-agent-actions): WebMCP is the draft browser API letting sites expose tools (search, cart, booking) to in-browser AI agents. The spec, who invokes it today, and how to ship it. - [How to Track AI Citations (ChatGPT, Claude, Perplexity) 2026](https://crawlytics.app/blog/how-to-track-ai-citations): Server logs show which AI bots fetched your pages; prompt-testing shows which answers cite you. Practical playbook with bot UAs, grep commands, and benchmarks. - [Crawlytics vs Profound: AI Brand Visibility Tools Compared (2026)](https://crawlytics.app/blog/crawlytics-vs-profound): Profound vs. Crawlytics: Profound is the high-end share-of-voice tool for enterprise; Crawlytics is the technical stack for bot tracking, llms.txt, and WebMCP… - [What Schema Markup Still Matters in the AI Search Era](https://crawlytics.app/blog/schema-markup-ai-search): Most schema is noise to LLMs. Four types still earn their keep: Article, FAQPage, Organization, BreadcrumbList. What to ship, what to skip, and why. - [AEO vs SEO vs GEO: Real Differences and Which to Invest in for 2026](https://crawlytics.app/blog/aeo-vs-seo-vs-geo): AEO, SEO, and GEO optimize for different surfaces — Google's blue links, AI Overviews, and chat engines. - [Block GPTBot or Allow It? The 2026 AI Crawler Decision Guide](https://crawlytics.app/blog/block-gptbot-decision-guide): Blocking GPTBot opts you out of training — but not ChatGPT-User, the real-time fetch bot. The 2026 framework for which AI bots to allow, block, or rate-limit… - [How to Add llms.txt to WordPress (Plugin and Manual Methods)](https://crawlytics.app/blog/wordpress-llms-txt-guide): WordPress has three llms.txt paths — plugin, functions.php, or hosted snippet. Plus caching plugin gotchas (W3 Total Cache, WP Rocket) and .htaccess tips. - [How to Add llms.txt to Shopify (Step-by-Step Guide for 2026)](https://crawlytics.app/blog/shopify-llms-txt-guide): Shopify has no native llms.txt support. Three install paths — Liquid template, App Store app, or hosted snippet — with tradeoffs. The complete 2026 guide. - [AI Search and the SEO Funnel: New Conversion Paths for 2026](https://crawlytics.app/blog/ai-search-changes-seo-funnel): AI search compresses awareness → consideration → decision into one chat turn. Fewer touches, different entry points, harder attribution. - [How to Get Cited by ChatGPT: A Practical Playbook for 2026](https://crawlytics.app/blog/how-to-get-cited-by-chatgpt): ChatGPT rewards clean H2s, direct first-sentence answers, named entities, and fresh dates. The exact playbook for showing up in ChatGPT's source list. ### Blog pages (~48 pages) Pattern: https://crawlytics.app/blog/{slug} - [Optimize Blog Posts for AI Citations: The 8-Edit Checklist](https://crawlytics.app/blog/optimize-blog-posts-for-ai-citations) - [How to Add WebMCP to Shopify Without Custom Code](https://crawlytics.app/blog/shopify-webmcp-install) - [Default-Deny AI Crawlers: Why Reuters and Publishers Are Switching](https://crawlytics.app/blog/default-deny-ai-crawlers) - [AI Agent Transactions: Chrome Auto-Browse Hits 200M+ Phones](https://crawlytics.app/blog/ai-agent-transactions) - [Blended Retrieval: Gemini Fuses Web + Private Context](https://crawlytics.app/blog/blended-retrieval) ## Other Pages - [What is Crawlytics? 60-Second Explainer Video](https://crawlytics.app/explainer): 60-second explainer: how Crawlytics tracks AI bots, generates llms.txt, and lets agents transact via WebMCP — the three layers you need to be AI-agent ready. - [AI Bot Analytics Demo: Live Sample Dashboard](https://crawlytics.app/demo): See a live demo of the Crawlytics dashboard — GPTBot/ClaudeBot/Perplexity tracking, WebMCP agent activity, and conversion attribution on real-looking data. - [Agent-Ready Grader: Free AI-Readiness Scanner for Any Website](https://crawlytics.app/agent-ready): Free scan of your site's AI-readiness across 5 categories — discoverability, accessibility, bot access, protocol support, attribution. No email required. ## Optional - [Crawlytics Privacy Policy: What We Store and Don't](https://crawlytics.app/privacy) - [Crawlytics Terms of Service: Plans, Billing, Acceptable Use](https://crawlytics.app/terms) ## Entity metadata - **Brand:** Crawlytics.app - **Domain:** crawlytics.app ---