Cloudflare converts HTML to markdown on demand. Crawlytics serves AI-Optimized HTML to every bot, plus stable llms.txt. Different format bets — and ChatGPT cannot read markdown. An honest decision guide.
Quick answer
Cloudflare's Markdown for Agents is a free edge feature that converts HTML to markdown on demand — but only when an AI agent sends Accept: text/markdown, which most don't. It's a clean primitive, but markdown isn't universal: ChatGPT-User, the fetcher that fires when someone pastes a link into ChatGPT, discards text/markdown as unreadable. Crawlytics instead serves AI-Optimized HTML — clean, chrome-free, with JSON-LD — to every AI bot by routing on the User-Agent, so there's nothing to negotiate and every fetcher can read it. It also publishes stable /llms.txt markdown, per-customer bot analytics, ChatGPT/Claude/Perplexity referral attribution, and WebMCP agent commerce — none of which Cloudflare ships. Use Cloudflare if you're on Pro+ and only need on-demand markdown for the agents that ask for it. Use Crawlytics if you want universal HTML coverage, analytics, attribution, llms.txt, or you're not on Cloudflare. Run both to layer Cloudflare's edge markdown under Crawlytics' measurement.
Cloudflare quietly shipped Markdown for Agents in beta a few months ago. If you're on Cloudflare Pro, Business, Enterprise, or SSL for SaaS, it's free. It converts your HTML to clean markdown on the fly, at the edge, whenever an AI agent asks for it.
That overlaps with what Crawlytics does — both make your site readable by AI bots. So the obvious question: if Cloudflare is free, why would anyone pay Crawlytics $29.99/mo for the Visibility tier?
This post is the honest answer. Cloudflare's offering is real, well-built, and free, and there are absolutely sites that should just use it. But the two products make a different bet on format, and that bet decides how many AI clients can actually read your content. Cloudflare converts your HTML to markdown, on demand, when an agent negotiates for it. Crawlytics serves AI-Optimized HTML — clean semantic HTML with JSON-LD, no nav or chrome — to every AI bot by routing on the User-Agent, and still publishes stable /llms.txt markdown alongside it. The difference matters more than the price tag, because of one inconvenient fact: the most-used live fetcher can't read markdown at all.
I'm going to compare them feature-by-feature without weasel words. Then I'll give you a clean decision tree at the end.
Three things, all at the network edge:
Accept: text/markdown in the request header, Cloudflare's edge fetches the HTML from your origin, converts it to markdown on the fly, and serves it back. The agent sees the same URL but a different format.x-markdown-tokens header showing the estimated token count of the markdown payload. Useful for agents budgeting their context window.That's it. It's a clean, well-executed primitive. If you're a Cloudflare customer on Pro or above, you get it free with one toggle.
/llms.txt or /llms-full.txt at stable URLs. Conversion only happens when the agent specifically requests markdown via the Accept header.None of those are criticisms. Cloudflare scoped the feature deliberately. They built the smallest useful thing and shipped it free to existing customers. Smart.
Crawlytics' Visibility tier ($29.99/mo) overlaps with Cloudflare's Markdown for Agents in exactly one area: making your site's content readable by AI bots. Same goal, different format bet.
Both strip nav, footer, scripts, and boilerplate, and both return clean structured content an LLM context window can ingest cheaply. The split is in what they hand back and who has to ask for it. Cloudflare returns markdown, but only to agents that send Accept: text/markdown. Crawlytics returns AI-Optimized HTML — clean semantic HTML with JSON-LD — to every AI bot, routed automatically on the User-Agent, and also publishes the same content as markdown at stable /llms.txt URLs (with raw markdown still available per page via ?format=md).
| Aspect | Cloudflare Markdown for Agents | Crawlytics |
|---|---|---|
| Format served to bots at the page URL | Markdown | AI-Optimized HTML (clean semantic HTML + JSON-LD) |
| How a bot gets the AI version | Must send Accept: text/markdown — content negotiation | Crawlytics detects the bot's User-Agent and routes it automatically — no header required |
| Coverage of AI fetchers | Only agents that negotiate and can read markdown | Every fetcher reads HTML — universal, including ChatGPT-User |
| Stable markdown URLs | None — conversion is per-request only | /llms.txt, /llms-full.txt, and per-page /md (?format=md for raw markdown) |
| Token count returned | x-markdown-tokens response header | Not yet (on the roadmap — see below) |
| Hosting requirement | Must be on Cloudflare Pro+ | Any host — Vercel, Netlify, WordPress, nginx, Apache, raw HTML |
Two of these decide whether your content is actually reachable.
Here's the wedge, and it's the whole reason format strategy matters. ChatGPT-User — the fetcher that fires the moment someone pastes your link into ChatGPT — discards text/markdown as unreadable. It expects HTML. So even on a site where Cloudflare's feature is enabled and a client negotiates for markdown, the single most important live fetcher gets content it throws away. Markdown isn't dead — Claude's fetcher, for example, tolerates it fine — but it isn't universal, and the gap lands on exactly the traffic you most want to win.
AI-Optimized HTML sidesteps the whole problem: every AI fetcher reads HTML. Crawlytics serves it by detecting the bot's User-Agent and returning a clean, chrome-free version of the page (with JSON-LD) at the same URL a human would visit — humans still get your full site. Nothing has to be negotiated, and nothing gets discarded. Cloudflare's markdown is a clean primitive, but it only reaches agents that both ask for markdown and can read it; Crawlytics reaches all of them.
The llms.txt standard (llmstxt.org, or our full setup guide here) emerged because LLM crawlers needed a predictable place to look. The convention is: put a markdown file at /llms.txt, mention your top pages, and AI systems will discover it. Tools like ChatGPT, Claude, Perplexity, and AI Overviews increasingly fetch this file directly — and here markdown is the right format, because the clients that consume /llms.txt expect it.
Cloudflare's Markdown for Agents has no stable URL at all; conversion only happens per request when an agent negotiates for it. So there's no single place an AI system can go to discover your site's structure. Crawlytics pre-generates /llms.txt and /llms-full.txt and keeps them current with a daily re-crawl. Between AI-Optimized HTML at every page URL and markdown at a stable /llms.txt, you cover both the fetchers that crawl pages and the clients that look for the index. Cloudflare covers neither without the agent asking first.
Cloudflare's Markdown for Agents is free, but only if you're already a Pro+ customer — that's $25/mo to start. If you're on Vercel, Netlify, plain GitHub Pages, WordPress.com, Squarespace, or any of the other ~70% of sites that aren't proxied through Cloudflare, the feature doesn't exist for you.
Crawlytics is host-agnostic. The snippet runs as a Cloudflare Worker, a Vercel middleware, a WordPress plugin, an Express middleware, or a static nginx/Apache log shipper. Pick what matches your stack.
I'm not going to pretend Cloudflare's offering is weak. Three things they do better:
When an agent does negotiate for markdown, Cloudflare converts your live origin at the edge during the request — no cron, no re-crawl lag. Crawlytics' AI-Optimized HTML at every page URL is rendered per request from the most-recently-crawled version of that page — the same crawled content as /llms.txt, just rendered to HTML instead of markdown — so it carries the same crawl lag: publish at 9:00 a.m. and the page may not reflect it until the next crawl. Cloudflare reads the current origin every time, so it's genuinely fresher for just-published pages, and that edge applies across the board, not just on a markdown path. It's a real freshness tradeoff — Crawlytics gives up a little currency in exchange for UA-routed universal HTML, per-page analytics, and attribution — but on raw freshness, Cloudflare wins here.
If you're already paying $25/mo for Cloudflare Pro, Markdown for Agents is bundled at no additional cost. You toggle it on and you're done. Crawlytics' Visibility tier is $29.99/mo on top of whatever else you're paying for hosting.
x-markdown-tokens headerThis is a small but real quality-of-life feature. AI agents that fetch your markdown can budget their context window without having to count tokens themselves. It's the kind of signal that becomes a de facto standard once enough hosts return it. Crawlytics doesn't return this header today — it should, and it will (more below).
Cloudflare Radar shows aggregated industry-wide bot traffic. It's a great public reference, but it doesn't tell you which pages on your site GPTBot is reading, when, or which paths it's ignoring. Crawlytics gives you that as a real-time dashboard — per-bot, per-page, per-day, with time-series charts and a 14-day projection.
If you're trying to optimize for AI search (figuring out which content is getting cited, which is being ignored, which deserves a refresh), you need per-customer data. Cloudflare's free tier doesn't provide it.
This is the biggest functional gap. ChatGPT, Claude, and Perplexity's in-app browsers strip the Referer header on outbound clicks. So when a user taps a citation in ChatGPT mobile, your Google Analytics logs the visit as "Direct / None." Cloudflare can't fix this — they don't touch your outbound links.
Crawlytics solves this by injecting per-LLM UTM tags (utm_source=chatgpt, utm_medium=ai_referral) into the links inside the AI-Optimized HTML each bot fetches. When ChatGPT cites your URL, the UTMs travel with it. Your analytics see chatgpt as the source, not (direct). Full write-up here.
This is the kind of feature that exists because someone went looking for the problem. Nobody at Cloudflare has shipped it. Probably nobody will until the problem gets loud enough.
WebMCP is the draft web spec (currently in Chrome 146+ Canary) that exposes navigator.modelContext, letting a page register tools an in-browser AI agent can invoke. Crawlytics' Commerce tier ($49.99/mo) ships a one-tag snippet that registers your tools (search, checkout, book, lead-capture) and attributes conversions back to the agent that drove them.
Cloudflare has nothing in this category. They convert HTML to markdown. They don't expose action surfaces, they don't handle conversion attribution, they don't integrate with Stripe/Paddle/Lemon Squeezy for revenue tracking.
If you're running an ecommerce site and you want AI agents to actually buy things on your site, Cloudflare can't help. Crawlytics can.
Already covered above but worth repeating: Crawlytics works on every stack. Cloudflare's feature only works for Cloudflare-proxied traffic.
Crawlytics scores each page on six signals (sitemap priority, URL depth, category, word count, recency, has-meta-description) and surfaces an agent-readiness score so you know which content is winning and which needs work. Cloudflare just converts whatever you have.
I'll be transparent: Cloudflare's feature has patterns Crawlytics should match, and the team is working on them:
x-markdown-tokens header. When Crawlytics serves raw markdown (via /llms.txt or ?format=md), it should include the token-count header so agents can budget their context window without counting themselves. Cheap to add, useful, and on its way to becoming a de facto standard.Accept: text/markdown on the page URL too. Crawlytics already routes AI bots to AI-Optimized HTML by User-Agent, which covers the fetchers that can't read markdown. For the agents that explicitly prefer markdown and say so via the Accept header (Claude's fetcher, for instance), it would be a nice touch to return markdown inline rather than making them hit ?format=md. The UA routing is the safe default; the Accept header is the courtesy.None of those affect what Crawlytics charges for — they're table stakes the whole industry is moving toward.
Here's the clean answer. Pick the path that matches you:
Accept: text/markdown and can read it — accepting that ChatGPT-User and other HTML-only fetchers fall through/llms.txt URLs any AI client can discover, not just per-request conversionFor most sites I talk to, the answer is "use Crawlytics" because they're not on Cloudflare Pro and they want the dashboard. For Cloudflare-native shops that don't care about per-customer attribution and are comfortable reaching only markdown-capable agents, just turn on Cloudflare's free feature and skip the bill.
Short answer: no. Cloudflare commoditized one layer (on-demand HTML → markdown conversion) for a subset of the market (their existing Pro+ customers) and a subset of agents (the ones that negotiate for markdown and can read it). That's not what Crawlytics charges for. Crawlytics charges for the layer above: universal AI-Optimized HTML that every fetcher can read, stable /llms.txt URLs, per-customer analytics, per-LLM attribution, and WebMCP commerce. Cloudflare doesn't compete in any of those.
If anything, Cloudflare's launch validates the category. Two years ago, "AI bot tracking" wasn't a phrase anyone used. Now Cloudflare ships a feature with that pitch, attached to one of the biggest infrastructure brands on the internet. That's a signal that the market is real, growing, and worth investing in.
If you want to see what a per-customer dashboard looks like, walk through the live demo. If you just need free HTML→markdown and you're on Cloudflare, go enable their feature — you don't need us for that.
Written by Crawlytics Team. Crawlytics tracks AI bots, generates llms.txt, and powers WebMCP commerce, all from one snippet on any stack. See how it works →
Short answer: no. Cloudflare commoditized one layer (on-demand HTML → markdown conversion) for a subset of the market (their existing Pro+ customers) and a subset of agents (the ones that negotiate for markdown and can read it). That's not what Crawlytics charges for. Crawlytics charges for the layer above: universal AI-Optimized HTML that every fetcher can read, stable /llms.txt URLs, per-customer analytics, per-LLM attribution, and WebMCP commerce. Cloudflare doesn't compete in any of those. If anything, Cloudflare's launch validates the category. Two years ago, "AI bot tracking" wasn't a phrase anyone used. Now Cloudflare ships a feature with that pitch, attached to one of the biggest infrastructure brands on the internet. That's a signal that the market is real, growing, and worth investing in. If you want to see what a per-customer dashboard looks like, walk through the live demo. If you just need free HTML→markdown and you're on Cloudflare, go enable their feature — you don't need us for that.
This page is part of Crawlytics.app. View all pages: llms.txt · llms-full.txt