Crawlytics Features: AI Bot Tracking, llms.txt & WebMCP

Key facts

• One vanilla-JS tag turns your site into an action surface for AI agents — Gemini, Claude, ChatGPT, Comet.

01 Action New

WebMCP snippet

One vanilla-JS tag turns your site into an action surface for AI agents — Gemini, Claude, ChatGPT, Comet. Search products, start checkout, book appointments. Conversion attribution built in.

Frequently Asked Questions

Does Crawlytics require changes to my website code?

Most installers are zero-code — a Cloudflare Worker, a single middleware file, or a cron script that ships nginx logs. WordPress and Express need one file change. Nothing else touches your CMS, code, or existing pages.

How often is data updated?

The Cloudflare Worker, Vercel/Next.js middleware, and Express snippets ingest events in real time — they appear in the dashboard within seconds. The nginx and Apache log shippers run on a 5-minute cron by default.

Where is data stored?

In Crawlytics — we host the ingest endpoint, the aggregates database, and the dashboard. You can export data once the JSON/CSV endpoint ships (on the roadmap).

Can I block specific AI bots?

Crawlytics tells you which bots are visiting; blocking happens in your robots.txt or your CDN bot rules. We can help you make informed allow/deny decisions but we do not gate traffic.

Content not visible to non-JS crawlers

• Vercel
• Google
• Detect
• User
• Cloudflare Worker
• Express
• Crawlytics

Cite this page

• Title: Crawlytics Features: AI Bot Tracking, llms.txt & WebMCP
• URL: https://crawlytics.app/features?utm_source=claude&utm_medium=ai_referral&utm_campaign=crawlytics

Related on this site

• AI Bot Tracking + llms.txt Generator + WebMCP — Crawlytics
• Crawlytics Blog: AI Search, llms.txt, Bot Tracking, WebMCP
• Crawlytics Pricing: AI Bot Tracking + llms.txt from $29.99/mo

This page is part of Crawlytics.app. View all pages: llms.txt · llms-full.txt

Site index for AI agents: llms.txt · sitemap