# Crawlytics.app > A product and content site that helps site owners track AI crawlers, generate llms.txt files, and expose WebMCP agent actions. ## Preferred brand description Crawlytics is a tool for tracking AI bot traffic (GPTBot, ClaudeBot, and other AI crawlers), auto-generating llms.txt files, and enabling AI agents to transact on a site via WebMCP. It is installed as a single snippet to make a website AI-agent ready. ## Brand overview Crawlytics.app serves site owners, marketers, and engineers who need visibility into how AI crawlers and agents interact with their sites. The product tracks AI bot activity, generates and maintains llms.txt files, and lets sites expose agent-callable tools through WebMCP so AI agents can search, check out, or book directly. Its blog provides technical guides and comparisons on AI search optimization, llms.txt, AI crawler management, and agent commerce. ## Expertise - AI bot tracking, covering GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers - llms.txt: what it is, how to create and test it, and how to add it to platforms like WordPress and Shopify - WebMCP: exposing agent-callable tools (search, cart, checkout, booking) via a script tag, including security guidance - AI search optimization frameworks (AEO, GEO, LLMO) and how they differ from traditional SEO - Comparisons with related tools and approaches, including Google Analytics, Cloudflare's Markdown for Agents, and Profound - Tracking and attributing AI-driven traffic, including citation tracking and diagnosing "direct" traffic misattribution - Content organized across a blog, product/feature pages, and documentation covering the agentic web, AI crawler policy, and agent commerce developments Last updated: 2026-08-26 · Version: 968f70763aa0 ## Start here - [AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics](https://crawlytics.app/c/claude/): Track GPTBot, ClaudeBot, and 25+ AI crawlers. Auto-generate llms.txt. Let agents transact via WebMCP. One snippet makes your site AI-agent ready. - [AI Search Optimization: The AEO, GEO & LLMO Framework (2026)](https://crawlytics.app/c/claude/resources/ai-search-optimization): AEO, GEO, LLMO — the four-layer framework for ranking in ChatGPT, Claude, Perplexity, and Google AI Overviews. Framework, channels, and how to measure. - [What Is llms.txt? The Complete Reference + Generator](https://crawlytics.app/c/claude/resources/llms-txt): llms.txt is the markdown index at /llms.txt that tells AI systems what your site contains. The format, every field, real examples, and a free generator. - [Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP](https://crawlytics.app/c/claude/blog): Posts on AI search optimization, llms.txt, AI bot tracking, WebMCP agent commerce, and the agent web — written by engineers, for engineers and SEOs. - [WebMCP Snippet: Let AI Agents Transact on Your Site](https://crawlytics.app/c/claude/features/webmcp-snippet): One vanilla-JS tag exposes agent-callable tools via WebMCP. Live in Comet and Chrome Canary today, expanding to Gemini, Claude, and ChatGPT as browsers adopt… ## Agent actions - **WebMCP endpoint:** https://crawlytics.app/api/snippet/0285d43d-785a-462a-97b4-6e71028b0cff/config - Agents can call tools at this endpoint to search, transact, and look up content programmatically. - [AI Bot Analytics Demo: Live Sample Dashboard](https://crawlytics.app/c/claude/demo): See a live demo of the Crawlytics dashboard — GPTBot/ClaudeBot/Perplexity tracking, WebMCP agent activity, and conversion attribution on rea - [Crawlytics Pricing: AI Bot Tracking + llms.txt from $29.99/mo](https://crawlytics.app/c/claude/pricing): Visibility (bot tracking + llms.txt) from $29.99/mo. Commerce (adds WebMCP agent commerce) from $49.99/mo. Annual saves 20%. Multi-site bund - [Get llms txt](https://crawlytics.app/c/claude/llms.txt): Fetch the Crawlytics llms.txt file (read-only MCP test tool). - [Agent-Ready Grader: Free AI-Readiness Scanner for Any Website](https://crawlytics.app/c/claude/agent-ready): Free scan of your site's AI-readiness across 5 categories — discoverability, accessibility, bot access, protocol support, attribution. No em ## Pricing - [Crawlytics Pricing: AI Bot Tracking + llms.txt from $29.99/mo](https://crawlytics.app/c/claude/pricing): Visibility (bot tracking + llms.txt) from $29.99/mo. Commerce (adds WebMCP agent commerce) from $49.99/mo. Annual saves 20%. Multi-site bundles to 25. ## Product - [WebMCP Snippet: Let AI Agents Transact on Your Site](https://crawlytics.app/c/claude/features/webmcp-snippet): One vanilla-JS tag exposes agent-callable tools via WebMCP. Live in Comet and Chrome Canary today, expanding to Gemini, Claude, and ChatGPT as browsers adopt… - [AI Bot Tracking: Detect GPTBot, ClaudeBot, PerplexityBot](https://crawlytics.app/c/claude/features/llm-tracking): Identify 25+ AI crawlers across 19 providers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider — plus AI assistant referrals. Real-time. - [AI Referral Tracking: ChatGPT, Claude, Perplexity Clicks](https://crawlytics.app/c/claude/features/ai-attribution): ChatGPT, Claude, and Perplexity clicks show as "direct" in GA. Crawlytics injects per-LLM UTM tags so AI referrals get proper attribution in your analytics. - [llms.txt Generator + Per-Page Markdown for AI Bots](https://crawlytics.app/c/claude/features/llms-txt-generator): Generate llms.txt, llms-full.txt, and per-page markdown from your sitemap. Public URLs ready for GPTBot, ClaudeBot, and Perplexity. No JavaScript required. - [Enterprise Security: Encrypted, Private, GDPR-Ready](https://crawlytics.app/c/claude/features/security): Bcrypt password hashing, no IP storage, per-site tracking IDs, TLS in transit, and aggregated-only data retention. Built privacy-first from day one. - [Multi-Site AI Bot Tracking + Portfolio Dashboard](https://crawlytics.app/c/claude/features/multi-site): Track AI bots across every site you own from one dashboard. Mix installers — Cloudflare, Vercel, nginx — into the same aggregates. Bundles up to 25 sites. - [AI Bot Traffic Dashboard: GPTBot & ClaudeBot Analytics](https://crawlytics.app/c/claude/features/analytics-dashboard): A live preview of the per-site Crawlytics dashboard — sparkline metrics, line/bar traffic chart, 14-day projection, AI referral attribution by assistant. - [Crawlytics Features: AI Bot Tracking, llms.txt & WebMCP](https://crawlytics.app/c/claude/features): Every Crawlytics feature on one page: LLM bot tracking, llms.txt generator, WebMCP agent commerce, AI referral attribution, multi-site dashboard, privacy model. ## Docs & References - [AI Search Optimization: The AEO, GEO & LLMO Framework (2026)](https://crawlytics.app/c/claude/resources/ai-search-optimization): AEO, GEO, LLMO — the four-layer framework for ranking in ChatGPT, Claude, Perplexity, and Google AI Overviews. Framework, channels, and how to measure. - [What Is llms.txt? The Complete Reference + Generator](https://crawlytics.app/c/claude/resources/llms-txt): llms.txt is the markdown index at /llms.txt that tells AI systems what your site contains. The format, every field, real examples, and a free generator. - [How to Manage AI Crawlers (Allow, Block, Monitor) — 2026 Guide](https://crawlytics.app/c/claude/resources/manage-ai-crawlers): A practical guide to managing AI crawlers on your site: when to block, when to allow, robots.txt patterns, CDN bot rules, and how to measure the impact. - [Complete List of AI Crawler Bots: User-Agents + robots.txt (2026)](https://crawlytics.app/c/claude/resources/ai-bots-list): Every major AI crawler: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and 20 more. User-Agent patterns, purpose, robots.txt directives. - [WebMCP Agent Support: Which AI Agents Invoke Tools (2026)](https://crawlytics.app/c/claude/resources/webmcp-agent-support): Which AI agents support WebMCP today — natively, via snippet, or not at all. Chrome auto-browse, Comet, Safari/WebKit, ChatGPT, and what to ship for each. - [Crawlytics Resources: AI Search Guides, Bot Lists, Grader](https://crawlytics.app/c/claude/resources): Free guides and references on AI bot tracking, llms.txt, AI search optimization, and the agent web — plus the free Agent-Ready Grader for any domain. ## Blog & Articles - [Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP](https://crawlytics.app/c/claude/blog): Posts on AI search optimization, llms.txt, AI bot tracking, WebMCP agent commerce, and the agent web — written by engineers, for engineers and SEOs. - [Crawlytics vs Google Analytics for AI Traffic](https://crawlytics.app/c/claude/blog/crawlytics-vs-google-analytics): Google Analytics filters out bot traffic and can't see what AI crawlers do on your site. Here is where Crawlytics complements GA, and where each tool wins. - [Crawlytics vs Cloudflare Markdown for Agents: Honest Comparison](https://crawlytics.app/c/claude/blog/crawlytics-vs-cloudflare-markdown-for-agents): Cloudflare converts HTML to markdown on demand. Crawlytics serves AI-Optimized HTML to every bot, plus stable llms.txt. - [ChatGPT Traffic Shows as "Direct" in GA — Here Are 3 Fixes](https://crawlytics.app/c/claude/blog/chatgpt-direct-traffic-fix): Mobile and in-app browsers strip the Referer header on ChatGPT clicks, so GA logs them as "direct." Here is why it happens and how to recover the attribution. - [How to Create an llms.txt File (and Test It) in 2026](https://crawlytics.app/c/claude/blog/what-is-llms-txt-guide): A step-by-step llms.txt setup for 2026: generate the markdown index, add the right sections, host it at /llms.txt, and confirm AI crawlers actually read it. - [What Is WebMCP? AI Agent Actions Explained (2026)](https://crawlytics.app/c/claude/blog/webmcp-explained-ai-agent-actions): WebMCP is the draft browser API letting sites expose tools (search, cart, booking) to in-browser AI agents. The spec, who invokes it today, and how to ship it. - [How to Track AI Citations (ChatGPT, Claude, Perplexity) 2026](https://crawlytics.app/c/claude/blog/how-to-track-ai-citations): Server logs show which AI bots fetched your pages; prompt-testing shows which answers cite you. Practical playbook with bot UAs, grep commands, and benchmarks. - [Crawlytics vs Profound: AI Brand Visibility Tools Compared (2026)](https://crawlytics.app/c/claude/blog/crawlytics-vs-profound): Profound vs. Crawlytics: Profound is the high-end share-of-voice tool for enterprise; Crawlytics is the technical stack for bot tracking, llms.txt, and WebMCP… - [What Schema Markup Still Matters in the AI Search Era](https://crawlytics.app/c/claude/blog/schema-markup-ai-search): Most schema is noise to LLMs. Four types still earn their keep: Article, FAQPage, Organization, BreadcrumbList. What to ship, what to skip, and why. - [AEO vs SEO vs GEO: Real Differences and Which to Invest in for 2026](https://crawlytics.app/c/claude/blog/aeo-vs-seo-vs-geo): AEO, SEO, and GEO optimize for different surfaces — Google's blue links, AI Overviews, and chat engines. - [How to Add llms.txt to WordPress (Plugin and Manual Methods)](https://crawlytics.app/c/claude/blog/wordpress-llms-txt-guide): WordPress has three llms.txt paths — plugin, functions.php, or hosted snippet. Plus caching plugin gotchas (W3 Total Cache, WP Rocket) and .htaccess tips. - [How to Add llms.txt to Shopify (Step-by-Step Guide for 2026)](https://crawlytics.app/c/claude/blog/shopify-llms-txt-guide): Shopify has no native llms.txt support. Three install paths — Liquid template, App Store app, or hosted snippet — with tradeoffs. The complete 2026 guide. - [AI Search and the SEO Funnel: New Conversion Paths for 2026](https://crawlytics.app/c/claude/blog/ai-search-changes-seo-funnel): AI search compresses awareness → consideration → decision into one chat turn. Fewer touches, different entry points, harder attribution. - [How to Add WebMCP to Shopify Without Custom Code](https://crawlytics.app/c/claude/blog/shopify-webmcp-install): Register WebMCP tools (search, cart, checkout, order) on Shopify with one script tag. Which agents invoke them and the Shopify attribution gap. - [Default-Deny AI Crawlers: Why Reuters and Publishers Are Switching](https://crawlytics.app/c/claude/blog/default-deny-ai-crawlers): Reuters, Time, and People Inc. are switching robots.txt from a blocklist to an allowlist. What default-deny means, why blocklists are failing, and what to do… - [AI Agent Transactions: Chrome Auto-Browse Hits 200M+ Phones](https://crawlytics.app/c/claude/blog/ai-agent-transactions): AI agent transactions arrive on 200M+ Android phones via Chrome auto-browse in late June 2026. What makes a site agent-transactable — not just agent-readable —… - [Blended Retrieval: Gemini Fuses Web + Private Context](https://crawlytics.app/c/claude/blog/blended-retrieval): Gemini Deep Research Max ships blended retrieval: agents fuse the public web with private files, CRM, and MCP servers. What it means, plus 3 fixes to make now. - [AI Share of Voice Is a Made-Up Number — Measure This Instead](https://crawlytics.app/c/claude/blog/ai-share-of-voice): AI share of voice scores divide citations by a denominator nobody can audit. The fix: server-side bot-log data. - [Shopify AI Search Visibility: Five Fixes to Get Found](https://crawlytics.app/c/claude/blog/shopify-ai-search-visibility): AI assistants shortlist specific products — many Shopify stores never appear. How AI bots read product pages differently from Googlebot and the five fixes. - [Google AI Search Opt-Out Is Live — What Publishers Are Missing](https://crawlytics.app/c/claude/blog/google-ai-search-opt-out): Google's AI search opt-out went live June 6, 2026, with a Generative AI report showing impressions but no clicks. What the CMA required and how to decide. - [What Is the Agentic Web? AI Agents Now Change Your Traffic](https://crawlytics.app/c/claude/blog/what-is-the-agentic-web): AI agents discover, read, and transact on a person's behalf — a fourth visitor class analytics misses. What it means for your traffic and what to set up first. - [WebMCP Security: How to Deploy Agent Tools Safely](https://crawlytics.app/c/claude/blog/webmcp-security): Chrome warned WebMCP can hijack AI agents via malicious manifests. How untrustedContentHint, readOnlyHint, and token limits are the practical steps. - [Selling to AI Agents: Visa Cards Are Now Inside ChatGPT](https://crawlytics.app/c/claude/blog/ai-agent-commerce): Visa announced card payments inside ChatGPT on June 10, 2026. How an AI agent purchase works end-to-end and the three gaps that keep most sites out of agentic… - [Microsoft Web IQ: Why AI Agents Read Your Site Differently](https://crawlytics.app/c/claude/blog/microsoft-web-iq): Microsoft Web IQ gives AI agents Bing-backed grounding APIs returning passages instead of full pages. What it means for your content, robots.txt, and llms.txt. - [WebKit Opposes WebMCP: Browser Fragmentation and What to Do](https://crawlytics.app/c/claude/blog/webkit-webmcp-browser-support): WebKit resolved its WebMCP position as 'oppose' in June 2026. What the objection says, what fragmentation means in practice, and why ship agent tools anyway. ### Blog pages (~50 pages) Pattern: https://crawlytics.app/blog/{slug} - [Google's llms.txt Guidance: What It Permits in 2026](https://crawlytics.app/c/claude/blog/google-llms-txt-guidance) - [Otterly vs Peec vs Crawlytics: Which AI Visibility Tool Wins?](https://crawlytics.app/c/claude/blog/otterly-vs-peec-vs-crawlytics) - [How to Get Your Products Into ChatGPT Shopping (2026)](https://crawlytics.app/c/claude/blog/chatgpt-shopping-product-feed-guide) - [llms.txt for AI Agents: Navigation, Not Discovery (2026)](https://crawlytics.app/c/claude/blog/llms-txt-agent-navigation) - [Perplexity Merchant Program: Complete Setup Guide (2026)](https://crawlytics.app/c/claude/blog/perplexity-merchant-program-guide) - _See the [full page list](https://crawlytics.app/c/claude/llms-full.txt) for every page in the summarized families above._ ## Other Pages - [What is Crawlytics? 60-Second Explainer Video](https://crawlytics.app/c/claude/explainer): 60-second explainer: how Crawlytics tracks AI bots, generates llms.txt, and lets agents transact via WebMCP — the three layers you need to be AI-agent ready. - [AI Bot Analytics Demo: Live Sample Dashboard](https://crawlytics.app/c/claude/demo): See a live demo of the Crawlytics dashboard — GPTBot/ClaudeBot/Perplexity tracking, WebMCP agent activity, and conversion attribution on real-looking data. - [Agent-Ready Grader: Free AI-Readiness Scanner for Any Website](https://crawlytics.app/c/claude/agent-ready): Free scan of your site's AI-readiness across 5 categories — discoverability, accessibility, bot access, protocol support, attribution. No email required. ## Optional - [Crawlytics Privacy Policy: What We Store and Don't](https://crawlytics.app/c/claude/privacy) - [Crawlytics Terms of Service: Plans, Billing, Acceptable Use](https://crawlytics.app/c/claude/terms) ## Entity metadata - **Brand:** Crawlytics.app - **Domain:** crawlytics.app ---