--- site: "Crawlytics.app" url: https://crawlytics.app/ publisher: "Crawlytics" author: "Crawlytics Team" lastUpdated: 2026-06-21 pagesIncluded: 35 pagesTotal: 84 generatedAt: 2026-06-25T01:56:16.135Z --- # Crawlytics.app > Full markdown bundle of the top 35 of 84 pages on https://crawlytics.app/, ranked by content quality, freshness, and importance. ## About this site **Publisher:** Crawlytics **Author:** Crawlytics Team **Last updated:** 2026-06-21 **Total pages indexed:** 84 ## Pages in this bundle 1. [AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics](https://crawlytics.app) 2. [Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP](https://crawlytics.app/blog) 3. [WebMCP Snippet: Let AI Agents Transact on Your Site](https://crawlytics.app/features/webmcp-snippet) 4. [AI Search Optimization: The AEO, GEO & LLMO Framework (2026)](https://crawlytics.app/resources/ai-search-optimization) 5. [Complete List of AI Crawler Bots: User-Agents + robots.txt (2026)](https://crawlytics.app/resources/ai-bots-list) 6. [What Is llms.txt? The Complete Reference + Generator](https://crawlytics.app/resources/llms-txt) 7. [How to Manage AI Crawlers (Allow, Block, Monitor) — 2026 Guide](https://crawlytics.app/resources/manage-ai-crawlers) 8. [Crawlytics vs Google Analytics for AI Traffic](https://crawlytics.app/blog/crawlytics-vs-google-analytics) 9. [Crawlytics vs Cloudflare Markdown for Agents: Honest Comparison](https://crawlytics.app/blog/crawlytics-vs-cloudflare-markdown-for-agents) 10. [ChatGPT Traffic Shows as "Direct" in GA — Here Are 3 Fixes](https://crawlytics.app/blog/chatgpt-direct-traffic-fix) 11. [How to Create an llms.txt File (and Test It) in 2026](https://crawlytics.app/blog/what-is-llms-txt-guide) 12. [What Is WebMCP? AI Agent Actions Explained (2026)](https://crawlytics.app/blog/webmcp-explained-ai-agent-actions) 13. [How to Track AI Citations (ChatGPT, Claude, Perplexity) 2026](https://crawlytics.app/blog/how-to-track-ai-citations) 14. [Crawlytics vs Profound: AI Brand Visibility Tools Compared (2026)](https://crawlytics.app/blog/crawlytics-vs-profound) 15. [What Schema Markup Still Matters in the AI Search Era](https://crawlytics.app/blog/schema-markup-ai-search) 16. [AEO vs SEO vs GEO: Real Differences and Which to Invest in for 2026](https://crawlytics.app/blog/aeo-vs-seo-vs-geo) 17. [Block GPTBot or Allow It? The 2026 AI Crawler Decision Guide](https://crawlytics.app/blog/block-gptbot-decision-guide) 18. [How to Add llms.txt to WordPress (Plugin and Manual Methods)](https://crawlytics.app/blog/wordpress-llms-txt-guide) 19. [How to Add llms.txt to Shopify (Step-by-Step Guide for 2026)](https://crawlytics.app/blog/shopify-llms-txt-guide) 20. [AI Search and the SEO Funnel: New Conversion Paths for 2026](https://crawlytics.app/blog/ai-search-changes-seo-funnel) 21. [How to Get Cited by ChatGPT: A Practical Playbook for 2026](https://crawlytics.app/blog/how-to-get-cited-by-chatgpt) 22. [Optimize Blog Posts for AI Citations: The 8-Edit Checklist](https://crawlytics.app/blog/optimize-blog-posts-for-ai-citations) 23. [How to Add WebMCP to Shopify Without Custom Code](https://crawlytics.app/blog/shopify-webmcp-install) 24. [Default-Deny AI Crawlers: Why Reuters and Publishers Are Switching](https://crawlytics.app/blog/default-deny-ai-crawlers) 25. [AI Agent Transactions: Chrome Auto-Browse Hits 200M+ Phones](https://crawlytics.app/blog/ai-agent-transactions) 26. [Blended Retrieval: Gemini Fuses Web + Private Context](https://crawlytics.app/blog/blended-retrieval) 27. [AI Share of Voice Is a Made-Up Number — Measure This Instead](https://crawlytics.app/blog/ai-share-of-voice) 28. [Shopify AI Search Visibility: Five Fixes to Get Found](https://crawlytics.app/blog/shopify-ai-search-visibility) 29. [Google AI Search Opt-Out Is Live — What Publishers Are Missing](https://crawlytics.app/blog/google-ai-search-opt-out) 30. [What Is the Agentic Web? AI Agents Now Change Your Traffic](https://crawlytics.app/blog/what-is-the-agentic-web) 31. [WebMCP Security: How to Deploy Agent Tools Safely](https://crawlytics.app/blog/webmcp-security) 32. [Selling to AI Agents: Visa Cards Are Now Inside ChatGPT](https://crawlytics.app/blog/ai-agent-commerce) 33. [Microsoft Web IQ: Why AI Agents Read Your Site Differently](https://crawlytics.app/blog/microsoft-web-iq) 34. [WebKit Opposes WebMCP: Browser Fragmentation and What to Do](https://crawlytics.app/blog/webkit-webmcp-browser-support) 35. [Google's llms.txt Guidance: What It Permits in 2026](https://crawlytics.app/blog/google-llms-txt-guidance) > **Note:** Truncated to the top 35 pages (84 total). Per-page markdown still available at https://crawlytics.app/md/. --- title: "AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics" type: [Organization, FAQPage, WebSite] canonical: https://crawlytics.app category: homepage wordCount: 680 readingTime: 3 min crawledAt: 2026-06-21 16:40:32 lastVerified: 2026-06-21 16:40:32 site: https://crawlytics.app/ --- # AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics ## Key facts - When ChatGPT or Google's agent lands on your site, it arrives with a job: book the appointment, buy the product, request the quote. - No code changes to your site, no DNS, no reverse proxy. - no reverse proxy · no DNS · one `. That's it. The loader registers your configured tools with navigator.modelContext on browsers that support WebMCP, and silently no-ops on browsers that don't. No CMS plugin, no build step. ### Which AI agents support WebMCP? WebMCP is the draft web spec exposing navigator.modelContext. Currently supported in Chrome 146+ Canary (which means Gemini Live, in-browser Claude artifacts, ChatGPT browser-mode, and Perplexity's Comet browser can invoke tools). Safari and Firefox have not shipped support yet. Crawlytics feature-detects before doing anything — zero risk to non-supporting browsers. ### Does WebMCP work in Safari? Not yet. WebMCP is a draft web spec and Safari has not announced support. The Crawlytics snippet feature-detects navigator.modelContext before doing anything, so Safari visitors see no behavior change. The conversion-attribution half of the snippet does run in every browser (it watches Stripe's ?session_id= on redirect-back), so you still get attribution from Safari-routed purchases. ### What is WebMCP? WebMCP is a draft web spec — currently in Chrome 146+ Canary preview — that exposes navigator.modelContext, letting a page register tools an in-browser AI agent can invoke. The snippet is your one-step way to register tools without writing browser-API code yourself. ### Does it require Chrome 146 Canary to work? The agent-action half does. On every other browser the snippet silently no-ops — it feature-detects navigator.modelContext before doing anything, so there is zero risk to real visitors. The conversion-attribution half runs in every browser (it just watches the success URL on Stripe redirect-back). ### Do I need to change my checkout? No. Conversion attribution works by detecting Stripe's ?session_id=cs_… on your success page — same page your customers already land on. Zero customer setup, no webhook, no API key. For cryptographically verified amounts you can optionally add a Stripe webhook later. ### What about CMS plugins? There aren't any and there won't be. The snippet is one script tag that drops into any HTTPS page — Shopify, Wix, Squarespace, custom Next.js, WordPress. No CMS-specific code anywhere. ### Where do API secrets live? On your server, never in the DB or browser. The snippet config stores the NAME of an env var (e.g. SITE_42_SHOPIFY_TOKEN, where 42 is the site id); Crawlytics resolves the value at invocation time via process.env. Names must match the per-site SITE__ pattern so a user can't name a server-internal env var as their "auth ref" and exfiltrate the value. The dashboard shows a green/red dot so you can confirm the var is wired without ever seeing the value. ### Can agents enter card details? No. PCI compliance and Stripe's sandboxed iframes make this impossible — by design. The agent collects intent, your endpoint creates a Stripe Checkout session, the agent hands the URL to the user. The user completes payment on Stripe's hosted page. No card data ever touches Crawlytics or the snippet. --- title: "AI Search Optimization: The AEO, GEO & LLMO Framework (2026)" type: [Organization, TechArticle, BreadcrumbList, FAQPage, WebSite] author: Crawlytics publisher: Crawlytics datePublished: 2026-06-03 dateModified: 2026-06-03 canonical: https://crawlytics.app/resources/ai-search-optimization category: docs wordCount: 1525 readingTime: 8 min crawledAt: 2026-06-21 16:40:31 lastVerified: 2026-06-21 16:40:31 site: https://crawlytics.app/ --- # AI Search Optimization: The AEO, GEO & LLMO Framework (2026) ## Summary AEO, GEO, LLMO — the four-layer framework for ranking in ChatGPT, Claude, Perplexity, and Google AI Overviews. Framework, channels, and how to measure. ## Key facts - AI search optimization is the practice of making your content discoverable, fetchable, and citable by AI assistants — ChatGPT, Claude, Perplexity, Gemini, Copilot, You. - Traditional search optimization (Google SEO) and AI search optimization overlap, but the inputs are different enough that you'd be wrong to treat them - AI search optimization breaks into four layers, in order of increasing impact: - If AI crawlers can't find your content, none of the rest matters. - AI crawlers can technically read HTML, but they spend most of their token budget on noise — nav, footer, cookie banner, JavaScript, ads. ## What is "AI search optimization"? AI search optimization is the practice of making your content discoverable, fetchable, and citable by AI assistants — ChatGPT, Claude, Perplexity, Gemini, Copilot, You.com, AI Overviews — instead of (or in addition to) traditional search engines. You'll see the same idea called by different names depending on who's writing: - **AEO** — Answer Engine Optimization. Emphasizes optimizing for answers, not blue links. - **GEO** — Generative Engine Optimization. Emphasizes generative AI as the surface. - **LLMO** — Large Language Model Optimization. Emphasizes the technology. - **"AI SEO"** — informal catch-all. All four terms describe the same underlying problem: traditional SEO assumes a human types a query, sees ten links, picks one. AI search assumes a human asks a question and gets a synthesized answer with two or three citations. The optimization work is to be in the citation list. ## Why this is its own discipline Traditional search optimization (Google SEO) and AI search optimization overlap, but the inputs are different enough that you'd be wrong to treat them as the same job: | | Google SEO | AI Search Optimization | | --- | --- | --- | | Primary signal | Inbound links + on-page content | Content depth + citability + structured signals | | Discovery surface | SERP — 10 blue links | Answer panel — 2-5 cited sources | | Click incentive | "Learn more" — click to read | "Get the source" — click to verify or go deeper | | Content format that wins | ~1,500-word topic-cluster posts optimized for keywords | Direct, factual, structured content that answers a specific question completely in 1-3 paragraphs | | Crawler that matters | Googlebot | GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and ~20 others | | Metric to optimize | Organic clicks, ranking position | Citation share, AI referral traffic, mention frequency | | Format AI prefers | HTML (with schema) | Markdown + llms.txt | The good news: most AI-optimized content also performs well in traditional SEO. The bad news: most traditionally-optimized content does _not_ perform well in AI search. You can do both, but you'll have to be deliberate. ## The four-layer framework AI search optimization breaks into four layers, in order of increasing impact: 1. **Discoverability** — can AI crawlers find your content? 2. **Readability** — can AI crawlers parse your content? 3. **Citability** — is your content the kind that gets cited in answers? 4. **Attribution** — do you see the resulting traffic, and can you measure what's working? Work them in order. Each layer compounds the value of the layers above. ## Layer 1: Discoverability If AI crawlers can't find your content, none of the rest matters. The checklist: - **Your robots.txt allows AI crawlers** (or you've made an explicit decision to block some — see [how to manage AI crawlers](https://crawlytics.app/resources/manage-ai-crawlers)) - **Your sitemap.xml is current and submitted** — most AI crawlers use sitemaps as their seed list - **You have an [`/llms.txt`](https://crawlytics.app/resources/llms-txt) file** the emerging standard for AI-readable site indexes - **No JavaScript-only rendering** — most AI crawlers don't execute JS; if your content only appears after JS runs, AI sees an empty page - **Mobile + desktop both serve the same content** — divergent mobile/desktop content confuses AI summaries Quick self-check: `curl -A "GPTBot" -L https://yoursite.com/` should return your real homepage content, not a JavaScript stub or a 403. ## Layer 2: Readability AI crawlers can technically read HTML, but they spend most of their token budget on noise — nav, footer, cookie banner, JavaScript, ads. The amount of your actual content that survives parsing is small, and it's the part that determines whether you get cited. Three approaches, increasing in technical effort: ### Easiest: clean HTML with semantic structure Wrap your real content in `
` and `
` tags. Use proper heading hierarchy (one H1, then H2s, then H3s). Put nav and footer in `