Notes on AI search, llms.txt, AI bot tracking, WebMCP commerce.https://crawlytics.app/https://crawlytics.app/blog/silent-funnel-ai-agent-traffic/https://crawlytics.app/blog/silent-funnel-ai-agent-traffic/AI agents already visit your site and fail silently: no bounce, no 404, nothing in analytics. What the silent funnel is and how to see real AI agent traffic.Sun, 21 Jun 2026 00:00:00 GMTAI AgentsBotsAnalyticshttps://crawlytics.app/blog/ai-catalog-json/https://crawlytics.app/blog/ai-catalog-json/ai-catalog.json, from the Google- and Microsoft-backed Agentic Resource Discovery spec, is a domain file that lets AI agents discover your tools and APIs at runtime.Sun, 21 Jun 2026 00:00:00 GMTAI Agentsllms.txtAI Searchhttps://crawlytics.app/blog/best-geo-tools/https://crawlytics.app/blog/best-geo-tools/The best GEO tools for 2026, organized by job: monitoring, optimizing, and serving. An honest generative engine optimization roundup with prices.Fri, 19 Jun 2026 00:00:00 GMTGEOToolsAI Searchhttps://crawlytics.app/blog/best-llms-txt-generators/https://crawlytics.app/blog/best-llms-txt-generators/An honest roundup of ways to generate llms.txt: hosted auto-crawlers, CMS plugins, free tools, validators, and DIY, and which fits your site.Fri, 19 Jun 2026 00:00:00 GMTllms.txtToolsComparisonhttps://crawlytics.app/blog/best-ai-bot-tracking-tools/https://crawlytics.app/blog/best-ai-bot-tracking-tools/Two ways to track AI bots, compared honestly: server-log and edge trackers vs CDN dashboards, intent tiering, bot ROI, and why GA misses crawlers.Fri, 19 Jun 2026 00:00:00 GMTBotsToolsAnalyticshttps://crawlytics.app/blog/best-agent-readiness-tools/https://crawlytics.app/blog/best-agent-readiness-tools/An honest roundup of AI readiness checkers, compared. Composite graders, free bot-access tools, and the score-to-fix gap most checkers leave open.Fri, 19 Jun 2026 00:00:00 GMTAgent ReadinessToolsAI Searchhttps://crawlytics.app/blog/ai-search-visibility-audit/https://crawlytics.app/blog/ai-search-visibility-audit/Audit your AI search visibility in five dimensions by hand (robots.txt, server HTML, bot access, schema, attribution), then score them with a free grader.Fri, 19 Jun 2026 00:00:00 GMTAI SearchAuditHow-tohttps://crawlytics.app/blog/best-profound-alternatives/https://crawlytics.app/blog/best-profound-alternatives/Cheaper Profound alternatives compared on price, engines, and free options: Otterly, Peec, Knowatoa, AthenaHQ, Scrunch, plus suite add-ons.Fri, 19 Jun 2026 00:00:00 GMTComparisonProfoundAI Visibilityhttps://crawlytics.app/blog/best-ai-brand-monitoring-tools/https://crawlytics.app/blog/best-ai-brand-monitoring-tools/The 8 best AI brand monitoring tools compared honestly: Profound, Otterly, Peec, AthenaHQ, Scrunch, Knowatoa, Ahrefs, Semrush, on engines and price.Fri, 19 Jun 2026 00:00:00 GMTAI VisibilityBrandToolshttps://crawlytics.app/blog/best-chatgpt-brand-monitoring-tools/https://crawlytics.app/blog/best-chatgpt-brand-monitoring-tools/Which monitors actually cover ChatGPT, the accuracy caveat nobody mentions, and how to fix what ChatGPT says about your brand and track ChatGPT-User.Fri, 19 Jun 2026 00:00:00 GMTChatGPTBrandToolshttps://crawlytics.app/blog/crawlytics-vs-scrunch-ai/https://crawlytics.app/blog/crawlytics-vs-scrunch-ai/Scrunch (now a Sitecore product) does share-of-voice and edge serving; Crawlytics adds llms.txt and WebMCP commerce from $29.99/mo. Where each wins.Fri, 19 Jun 2026 00:00:00 GMTComparisonScrunch AIhttps://crawlytics.app/blog/crawlytics-vs-knowatoa/https://crawlytics.app/blog/crawlytics-vs-knowatoa/Knowatoa monitors AI share-of-voice and probes whether 24 AI bots can reach your site; Crawlytics reads real bot traffic and serves llms.txt.Fri, 19 Jun 2026 00:00:00 GMTComparisonKnowatoahttps://crawlytics.app/blog/crawlytics-vs-ahrefs-bot-analytics/https://crawlytics.app/blog/crawlytics-vs-ahrefs-bot-analytics/Ahrefs Bot Analytics is a Cloudflare-only beta that logs AI-bot hits inside the suite. Crawlytics tracks bots on any stack, plus llms.txt and WebMCP.Fri, 19 Jun 2026 00:00:00 GMTComparisonAhrefshttps://crawlytics.app/blog/do-you-need-ahrefs-semrush-for-ai-visibility/https://crawlytics.app/blog/do-you-need-ahrefs-semrush-for-ai-visibility/You don't need a full Ahrefs or Semrush plan for AI visibility; both sell it as add-ons. When that pays off, and when a dedicated tool wins.Fri, 19 Jun 2026 00:00:00 GMTAI VisibilityAhrefsSemrushhttps://crawlytics.app/blog/ai-visibility-tracking-cost/https://crawlytics.app/blog/ai-visibility-tracking-cost/AI visibility tools cost $0 to $2,000+/mo as of June 2026. A full price table from free graders to enterprise, plus what you need at each budget.Fri, 19 Jun 2026 00:00:00 GMTAI VisibilityPricingToolshttps://crawlytics.app/blog/is-llms-txt-worth-it/https://crawlytics.app/blog/is-llms-txt-worth-it/Skeptics are right that llms.txt won't rank you, but that misses its real job: agent navigation and coding tools. An honest, by-site-type decision.Fri, 19 Jun 2026 00:00:00 GMTllms.txtStrategyAI Searchhttps://crawlytics.app/blog/ai-visibility-tools-for-agencies/https://crawlytics.app/blog/ai-visibility-tools-for-agencies/AI visibility tools for agencies: which monitors have agency tiers, the honest white-label reality, and how to deliver monitoring plus fixing per client.Fri, 19 Jun 2026 00:00:00 GMTAI VisibilityAgenciesToolshttps://crawlytics.app/blog/white-label-ai-search-reports/https://crawlytics.app/blog/white-label-ai-search-reports/Build and deliver white-label AI search reports to clients: what to include, the data sources, which tools export today, and a reusable template.Fri, 19 Jun 2026 00:00:00 GMTAgenciesReportingAI Searchhttps://crawlytics.app/blog/prove-geo-roi-to-clients/https://crawlytics.app/blog/prove-geo-roi-to-clients/The first-party metrics that hold up when AI attribution is hard: bot-traffic growth, recovered AI referrals, readiness deltas, and a reusable framework.Fri, 19 Jun 2026 00:00:00 GMTAgenciesGEOROIhttps://crawlytics.app/blog/how-to-track-ai-bots-crawling-your-site/https://crawlytics.app/blog/how-to-track-ai-bots-crawling-your-site/A step-by-step guide to tracking AI bots: capture requests from nginx, Vercel, Cloudflare, or WordPress, identify crawlers by user-agent, and tier by intent.Fri, 19 Jun 2026 00:00:00 GMTBotsHow-toAnalyticshttps://crawlytics.app/blog/agentic-commerce-for-saas/https://crawlytics.app/blog/agentic-commerce-for-saas/Stripe Projects launched April 30, 2026, letting AI agents create accounts, purchase plans, and provision infrastructure autonomously. The detect/serve/sell checklist for SaaS founders who want to be legible when the agent arrives.Thu, 18 Jun 2026 00:00:00 GMTAI AgentsAgentic CommerceSaaSStripeWebMCPhttps://crawlytics.app/blog/retrieval-vs-citation/https://crawlytics.app/blog/retrieval-vs-citation/AI retrieval is when a crawler fetches your page to build an answer. Citation is when AI names you as a source in the answer users see. The gap between them is where most AI visibility strategy goes wrong.Thu, 18 Jun 2026 00:00:00 GMTAI SearchAEOStrategyhttps://crawlytics.app/blog/open-knowledge-format/https://crawlytics.app/blog/open-knowledge-format/Google's OKF is a markdown+YAML spec for packaging AI-readable knowledge. How it relates to llms.txt, AGENTS.md, CLAUDE.md, and the context-layer stack forming around AI agents.Thu, 18 Jun 2026 00:00:00 GMTllms.txtAI AgentsAI Searchhttps://crawlytics.app/blog/google-llms-txt-guidance/https://crawlytics.app/blog/google-llms-txt-guidance/Google calls llms.txt 'completely fine' — but that's not the same as Google using it. What the guidance actually says, and why non-Google AI surfaces are the real audience.Thu, 18 Jun 2026 00:00:00 GMTllms.txtAI SearchSEOhttps://crawlytics.app/blog/llms-txt-no-traffic-data/https://crawlytics.app/blog/llms-txt-no-traffic-data/Ahrefs found 97% of llms.txt files across 137,000 domains got zero AI bot requests. The stat is real, but the scoreboard it uses misses coding agents, the file's primary audience.Thu, 18 Jun 2026 00:00:00 GMTllms.txtBotsStrategyhttps://crawlytics.app/blog/ai-bot-traffic-cost/https://crawlytics.app/blog/ai-bot-traffic-cost/AI bot traffic cost is now a real line item: crawlers hammer carts, checkout, and search pages that never cache. The ROI framework for which AI crawlers to allow, rate-limit, or monetize.Sat, 13 Jun 2026 00:00:00 GMTBotsCostStrategyhttps://crawlytics.app/blog/fix-what-chatgpt-says-about-your-brand/https://crawlytics.app/blog/fix-what-chatgpt-says-about-your-brand/AI assistants confidently state wrong facts about your product — missing features, invented ones, stale pricing. The repeatable loop to audit, trace, fix the source, and verify.Thu, 11 Jun 2026 00:00:00 GMTAI SearchBrandAEOhttps://crawlytics.app/blog/chatgpt-agent-cant-access-website/https://crawlytics.app/blog/chatgpt-agent-cant-access-website/A ChatGPT agent blocked from your website is usually accidental. The 6 common causes, from Cloudflare bot rules to JS-only rendering, and how to confirm and fix each.Thu, 11 Jun 2026 00:00:00 GMTAI AgentsBotsTroubleshootinghttps://crawlytics.app/blog/best-ai-visibility-tools-under-50/https://crawlytics.app/blog/best-ai-visibility-tools-under-50/The best AI visibility tools under $50/mo for small businesses in 2026. Honest picks, prompt tracking vs crawler tracking, the diagnose-but-don't-fix trap, and pricing.Thu, 11 Jun 2026 00:00:00 GMTAI VisibilityToolsComparisonhttps://crawlytics.app/blog/otterly-vs-peec-vs-crawlytics/https://crawlytics.app/blog/otterly-vs-peec-vs-crawlytics/Otterly and Peec sample what LLMs say about you; Crawlytics reads your server logs for real crawl evidence. Pricing, what each measures, and which fits a solo, agency, or ecommerce budget.Thu, 11 Jun 2026 00:00:00 GMTComparisonAI VisibilityToolshttps://crawlytics.app/blog/agentic-checkout-readiness-checklist/https://crawlytics.app/blog/agentic-checkout-readiness-checklist/AI shopping agents have started finishing purchases. The practical find/read/buy checklist for store owners, scored honestly, with tiered fixes at $0, $50, and $500/mo.Thu, 11 Jun 2026 00:00:00 GMTAgentic CommerceCommerceAI Agentshttps://crawlytics.app/blog/cloudflare-agent-readiness-score/https://crawlytics.app/blog/cloudflare-agent-readiness-score/Cloudflare's free isitagentready.com checker scans your llms.txt, MCP, and markdown signals. What each check means, what a one-time scan can't see, and how to improve it.Thu, 11 Jun 2026 00:00:00 GMTAgent ReadinessAI AgentsCloudflarehttps://crawlytics.app/blog/woocommerce-chatgpt-shopping/https://crawlytics.app/blog/woocommerce-chatgpt-shopping/WooCommerce has no native ChatGPT integration like Shopify. The manual playbook: build a product feed, wire Stripe for AI agents, and make pages agent-readable.Thu, 11 Jun 2026 00:00:00 GMTWooCommerceAgentic CommerceCommercehttps://crawlytics.app/blog/perplexity-merchant-program-guide/https://crawlytics.app/blog/perplexity-merchant-program-guide/How the Perplexity Merchant Program works in 2026: free feed-based application, product feed and GTIN requirements, the schema it reads, and how to measure results.Thu, 11 Jun 2026 00:00:00 GMTPerplexityAgentic CommerceCommercehttps://crawlytics.app/blog/chatgpt-shopping-product-feed-guide/https://crawlytics.app/blog/chatgpt-shopping-product-feed-guide/Get products into ChatGPT Shopping by submitting a structured product feed. The fields, the SFTP submission, validation, refresh cadence, and how to verify agents reach you.Thu, 11 Jun 2026 00:00:00 GMTChatGPT ShoppingAgentic CommerceCommercehttps://crawlytics.app/blog/webmcp-security/https://crawlytics.app/blog/webmcp-security/Chrome warned WebMCP can hijack AI agents via malicious manifests and contaminated outputs. The mitigations, mapped to practical manifest settings.Thu, 11 Jun 2026 00:00:00 GMTWebMCPSecurityAI Agentshttps://crawlytics.app/blog/google-ai-search-opt-out/https://crawlytics.app/blog/google-ai-search-opt-out/Google's Search Console AI opt-out toggle is live, but the new Generative AI report shows impressions only. How to decide without click data.Thu, 11 Jun 2026 00:00:00 GMTAI OverviewsSearch ConsolePublishershttps://crawlytics.app/blog/ai-agent-commerce/https://crawlytics.app/blog/ai-agent-commerce/Visa just plugged its payment network into ChatGPT. How agent purchases work end-to-end, and the three gaps keeping most sites out of agentic commerce.Thu, 11 Jun 2026 00:00:00 GMTAI AgentsAgentic CommercePaymentshttps://crawlytics.app/blog/webkit-webmcp-browser-support/https://crawlytics.app/blog/webkit-webmcp-browser-support/WebKit resolved its WebMCP position as 'oppose' in June 2026. What the objection says, and why you can still ship agent tools confidently today.Thu, 11 Jun 2026 00:00:00 GMTWebMCPAI AgentsBrowser Standardshttps://crawlytics.app/blog/what-is-the-agentic-web/https://crawlytics.app/blog/what-is-the-agentic-web/AI agents are a fourth visitor class your analytics can't count, and they now convert 42% better than human traffic. What the agentic web means for your site.Thu, 11 Jun 2026 00:00:00 GMTAgentic WebAI AgentsAI Searchhttps://crawlytics.app/blog/microsoft-web-iq/https://crawlytics.app/blog/microsoft-web-iq/Microsoft Web IQ gives AI agents Bing-backed grounding APIs returning passages, not pages. What it means for your content, robots.txt, and llms.txt.Thu, 11 Jun 2026 00:00:00 GMTAI AgentsAI SearchGroundinghttps://crawlytics.app/blog/blended-retrieval/https://crawlytics.app/blog/blended-retrieval/Gemini Deep Research Max fuses the public web with users' private files and MCP data in one pass. How signal share works now, and three fixes to ship.Thu, 11 Jun 2026 00:00:00 GMTAI AgentsAI SearchStructured Datahttps://crawlytics.app/blog/ai-share-of-voice/https://crawlytics.app/blog/ai-share-of-voice/AI share of voice divides citations by a denominator nobody can audit. The fix is server-side bot-log data: three AI visibility metrics you can defend.Thu, 11 Jun 2026 00:00:00 GMTAI VisibilityAnalyticsMeasurementhttps://crawlytics.app/blog/shopify-ai-search-visibility/https://crawlytics.app/blog/shopify-ai-search-visibility/AI assistants recommend specific products, and many Shopify stores never make the shortlist. Why AI bots miss your store and the five fixes that restore it.Thu, 11 Jun 2026 00:00:00 GMTShopifyAI SearchCommercehttps://crawlytics.app/blog/ai-agent-transactions/https://crawlytics.app/blog/ai-agent-transactions/Chrome auto-browse arrives on 200M+ Android phones in late June 2026. What makes a site agent-transactable (not just agent-readable), and how to audit yours with a 30-second test.Wed, 10 Jun 2026 00:00:00 GMTAI AgentsAgentic WebCommercehttps://crawlytics.app/blog/default-deny-ai-crawlers/https://crawlytics.app/blog/default-deny-ai-crawlers/Reuters, Time, and People Inc. are switching robots.txt from a blocklist to an allowlist. What default-deny means, why blocklists are failing, and what smaller sites should do instead.Wed, 10 Jun 2026 00:00:00 GMTBotsStrategyPublishershttps://crawlytics.app/blog/aeo-vs-seo-vs-geo/https://crawlytics.app/blog/aeo-vs-seo-vs-geo/AEO, SEO, and GEO sound interchangeable but optimize for different surfaces — Google's blue links, AI Overviews, and chat-based engines. What each one means and which to invest in for 2026.Fri, 05 Jun 2026 00:00:00 GMTAEOAI SearchGuidehttps://crawlytics.app/blog/ai-search-changes-seo-funnel/https://crawlytics.app/blog/ai-search-changes-seo-funnel/AI search compresses awareness → consideration → decision into a single chat turn. The new funnel has different entry points, fewer touches, and different attribution problems.Fri, 05 Jun 2026 00:00:00 GMTAI SearchStrategyConversionhttps://crawlytics.app/blog/how-to-get-cited-by-chatgpt/https://crawlytics.app/blog/how-to-get-cited-by-chatgpt/ChatGPT's citation algorithm rewards specific structural patterns — clean H2s, direct first-sentence answers, named entities, fresh dates. The exact playbook for showing up in the source list.Fri, 05 Jun 2026 00:00:00 GMTChatGPTAEOPlaybookhttps://crawlytics.app/blog/optimize-blog-posts-for-ai-citations/https://crawlytics.app/blog/optimize-blog-posts-for-ai-citations/Most sites have a 50+ post archive and don't know how to upgrade it for AI search. The 8-edit retrofit checklist that adds AI-citability without rewriting from scratch.Fri, 05 Jun 2026 00:00:00 GMTAEOContentPlaybookhttps://crawlytics.app/blog/schema-markup-ai-search/https://crawlytics.app/blog/schema-markup-ai-search/Most schema markup is noise to LLMs. A short list still moves the needle for AI citations — Article, FAQPage, Organization, BreadcrumbList. What to ship, what to skip, and why.Fri, 05 Jun 2026 00:00:00 GMTSchemaAI SearchHow-tohttps://crawlytics.app/blog/squarespace-llms-txt-guide/https://crawlytics.app/blog/squarespace-llms-txt-guide/Squarespace has no native llms.txt support, but a URL Mappings 301 redirect makes it work. The exact upload-and-redirect steps, plus an honest read on what it does.Thu, 11 Jun 2026 00:00:00 GMTllms.txtSquarespaceHow-tohttps://crawlytics.app/blog/shopify-llms-txt-guide/https://crawlytics.app/blog/shopify-llms-txt-guide/Shopify has no native llms.txt support. Three install paths — Liquid template, custom Shopify app, or a hosted snippet — each with their tradeoffs.Fri, 05 Jun 2026 00:00:00 GMTllms.txtShopifyHow-tohttps://crawlytics.app/blog/wordpress-llms-txt-guide/https://crawlytics.app/blog/wordpress-llms-txt-guide/WordPress has three install paths for llms.txt — plugin, manual functions.php edit, or hosted snippet. Plus the WordPress-specific gotchas around caching plugins and .htaccess.Fri, 05 Jun 2026 00:00:00 GMTllms.txtWordPressHow-tohttps://crawlytics.app/blog/block-gptbot-decision-guide/https://crawlytics.app/blog/block-gptbot-decision-guide/Blocking GPTBot opts you out of training the next ChatGPT — but it doesn't block ChatGPT-User (the real-time fetch). The framework for which AI bots to allow, block, or rate-limit.Fri, 05 Jun 2026 00:00:00 GMTBotsStrategyGPTBothttps://crawlytics.app/blog/shopify-webmcp-install/https://crawlytics.app/blog/shopify-webmcp-install/Register WebMCP tools on your Shopify store with one script tag. The honest read on which agents invoke them today (Comet, extensions — not yet ChatGPT/Claude), and the install.Fri, 05 Jun 2026 00:00:00 GMTWebMCPShopifyCommercehttps://crawlytics.app/blog/crawlytics-vs-profound/https://crawlytics.app/blog/crawlytics-vs-profound/Profound and Crawlytics both track AI brand visibility but optimize for different jobs. Profound is the high-end share-of-voice tool; Crawlytics is the technical stack for bot tracking + llms.txt + WebMCP commerce.Fri, 05 Jun 2026 00:00:00 GMTComparisonProfoundhttps://crawlytics.app/blog/what-is-llms-txt-guide/https://crawlytics.app/blog/what-is-llms-txt-guide/llms.txt is a markdown index at /llms.txt that tells AI systems what your site contains. Format, examples, three setup paths, and whether it helps your SEO.Fri, 05 Jun 2026 00:00:00 GMTllms.txtGuideAI Searchhttps://crawlytics.app/blog/how-to-track-ai-citations/https://crawlytics.app/blog/how-to-track-ai-citations/Two questions, two methods: server logs tell you which AI bots fetched your pages; prompt-testing tells you which answers cite you. The full playbook.Fri, 05 Jun 2026 00:00:00 GMTAI SearchAttributionAnalyticshttps://crawlytics.app/blog/llms-txt-agent-navigation/https://crawlytics.app/blog/llms-txt-agent-navigation/John Mueller is right that llms.txt can't get you discovered by AI — that was never its job. Here's what the file actually does (on-site agent navigation, coding agents, WebMCP handoff) and why it still belongs on your site.Thu, 18 Jun 2026 00:00:00 GMTllms.txtAI AgentsWebMCPhttps://crawlytics.app/blog/webmcp-explained-ai-agent-actions/https://crawlytics.app/blog/webmcp-explained-ai-agent-actions/WebMCP exposes navigator.modelContext so in-browser agents can invoke tools on your site. The spec, who invokes it today (Comet, extensions — not yet ChatGPT/Claude), and how to ship it as a forward investment.Fri, 05 Jun 2026 00:00:00 GMTWebMCPAI AgentsCommercehttps://crawlytics.app/blog/crawlytics-vs-cloudflare-markdown-for-agents/https://crawlytics.app/blog/crawlytics-vs-cloudflare-markdown-for-agents/Cloudflare ships Markdown for Agents free on Pro+. Crawlytics costs $29.99/mo. When does each win?Wed, 03 Jun 2026 00:00:00 GMTComparisonCloudflarehttps://crawlytics.app/blog/crawlytics-vs-google-analytics/https://crawlytics.app/blog/crawlytics-vs-google-analytics/GA filters out bots and misses AI assistant referrals. Where Crawlytics complements GA, and where each wins.Wed, 03 Jun 2026 00:00:00 GMTComparisonGoogle Analyticshttps://crawlytics.app/blog/chatgpt-direct-traffic-fix/https://crawlytics.app/blog/chatgpt-direct-traffic-fix/In-app browsers strip the Referer header on ChatGPT clicks, so GA logs them as direct. How to recover the attribution.Thu, 28 May 2026 00:00:00 GMTAnalyticsAttribution