Cloudflare Just Made It Real: How to Charge AI Agents for Your Content with x402

An AI crawler can hit your site ten thousand times and send back a single visitor. Sometimes zero. For most of the web's history that trade still worked, because the crawler was Googlebot and the visit it sent was worth real money. That deal is breaking, and Cloudflare has now shipped the plumbing for what replaces it: charge the agents directly.

Two announcements landed close together. The Monetization Gateway brings a payment protocol called x402 to the network edge, so a site can put a price on the requests an AI bot makes. Alongside it, Making AI Search Smarter lays out Pay Per Use, the successor to Pay Per Crawl, plus a research program on freshness signals. Read together, they describe one shift: the attention economy is turning into a payment economy, and site owners need a way to take part.

This post explains x402 in plain terms, with no crypto sermon. Then it shows the three layers you actually need to charge agents, and where Cloudflare's edge tollbooth ends and Crawlytics begins. The two are complementary, not rivals.

The deal that ran the web for 30 years just broke

The bargain was simple and it held for three decades. You let a search engine crawl your pages for free. In return it sent you visitors, and those visitors saw ads, subscribed, or bought something. Crawl in, referral out, revenue at the end. Cloudflare calls this the attention bargain, and its framing is worth crediting, because Cloudflare can see the raw numbers across a large slice of the web.

AI broke the return half of that trade. When an assistant reads your page and answers the question in its own words, the user never clicks. Pew Research found that when an AI summary is shown, people follow a link to the source only 8% of the time, a figure Cloudflare cites in its Making AI Search Smarter post. The crawl still happens. The referral mostly does not.

The exchange rate makes the problem concrete. Cloudflare reports that AI crawlers already send somewhere between 100 and 10,000 requests for every single visitor they refer back. A training crawler at the high end of that range takes thousands of times more than it returns. You are subsidizing the model and getting a rounding error of traffic in exchange.

Blocking every bot is one answer, and for pure training scrapers it is often the right one. Blunt blocking also shuts out the agents you might want, like an assistant fetching your product page to help someone buy. For a lot of content the better answer is a price. That is what x402 makes possible.

What x402 actually is, in plain English

x402 is a way to charge for a web request using the web's own plumbing. The name comes from HTTP status code 402, labeled "Payment Required," which the people who wrote the early web reserved for exactly this purpose and then left unused for decades. x402 finally puts it to work.

The flow is short. An AI agent requests your page. Instead of returning the content, your server, or Cloudflare at the edge, answers with a 402 and a price, say a fraction of a cent. The agent's wallet pays automatically and repeats the request with proof of payment. The server returns the page. There is no account to create, no invoice to email, no human clicking approve. It happens in the time it takes to load a page.

Here is the part that trips people up, so let me be direct. The money settles in stablecoins under the hood, but you never touch that layer. Cloudflare's Monetization Gateway handles settlement, so from your seat it looks like a price on a request and money arriving in an account. It is closer to a metered API bill than to anything you would call trading. You do not need to understand blockchains to charge an agent, any more than you need to understand card networks to accept a credit card.

That is the whole idea. A machine-readable price, paid by a machine, for a single request. No jargon required.

Cloudflare's Monetization Gateway: the tollbooth at the edge

The Monetization Gateway is Cloudflare's implementation of that flow, running on its network. Turn it on and requests from AI bots can be met with a 402 and a price before they ever reach your origin server. Cloudflare collects, settles, and hands you the revenue. You never write payment code.

Making AI Search Smarter adds the pricing model. Pay Per Crawl, the first version, charged a bot each time it crawled. Pay Per Use is the evolution: it aims to charge based on how an agent uses your content, not just the act of fetching it. A search engine that reads one page to answer a live question is worth more to you than a scraper that vacuums the whole site once for training, and Pay Per Use is built to price that gap. The freshness-signal research in the same post is Cloudflare working out how to value recency, since fresh content is often what agents will pay a premium for.

This is Cloudflare productizing the shift, not inventing it. For a site owner the takeaway is that the tollbooth now exists and it is close to one toggle. What the toggle does not give you is knowledge: which agents are showing up, how often, on which pages, and whether any of them are worth charging at all. That is a different layer.

Charging agents is a three-layer job: detect, serve, sell

Putting a price on agent traffic is not one action. It is three, and they run in order. You have to see the traffic before you can value it, describe what you are selling before an agent can want it, and only then collect. Skip the first two and you are pricing blind.

Detect. Crawlytics runs from one snippet and shows which bots hit which pages, per bot, per page, per day, with a 14-day projection. You can see that ClaudeBot reads your docs nightly while GPTBot ignores them, and understand the real cost that traffic adds to your bandwidth bill. Pricing starts with this picture. Our guide to tracking AI bots walks through the setup.

Serve. Crawlytics generates a stable llms.txt and serves AI-Optimized HTML to every bot by routing on the User-Agent. Think of llms.txt as the menu: it tells an agent what content exists and how current it is, which is exactly the signal a Pay Per Use price attaches to. An agent cannot decide your page is worth paying for if it cannot cleanly read what is there.

Sell. This is where the two products meet. Cloudflare's gateway charges for the request at the edge. Crawlytics adds WebMCP, a browser standard that lets a page expose actions an agent can call, so an agent can search, book, or complete a purchase on your site rather than only paying to read it. One monetizes access, the other monetizes action. The mechanics are covered in how WebMCP works.

Where Cloudflare ends and Crawlytics begins

Short version: Cloudflare enforces payment, Crawlytics gives you sight and reach. Neither replaces the other. If you already run Cloudflare, its gateway is the natural place to collect. What it does not do is tell you who is at the gate.

The longer comparison of Crawlytics and Cloudflare covers the content-format side of this in depth. The pattern holds for monetization too. Cloudflare owns the edge, Crawlytics owns the per-site picture and the on-page action surface, and they layer cleanly.

What to do this week

Pick the row that matches you.

• You are on Cloudflare and getting hammered by crawlers. Turn on the Monetization Gateway to start charging at the edge, and add Crawlytics so you can see which agents are paying, which are worth charging more, and which you should leave free.
• You are not on Cloudflare, or not on a paid plan. Start with Crawlytics for detection and llms.txt. You get the visibility and the menu now, and you can add edge payment later without redoing the first two layers.
• You sell something an agent could buy. Add WebMCP through the Crawlytics Commerce tier so agents can transact on your pages, not only pay to read them. Charging for access is defense; charging for a completed sale is offense.
• You just want the data before deciding anything. Run the free scan, watch a week of bot traffic, then choose. Do not price content you have not measured.

The web's old deal traded content for attention. The new one trades content for payment, and the rails now exist to make that automatic. Cloudflare built the tollbooth. Crawlytics tells you who is driving up to it and gives agents a way to buy once they are through. See your own bot traffic in the live demo and decide what is worth charging for.

Frequently asked questions

What is the x402 protocol?

x402 is a payment method built into the web's own plumbing. It uses HTTP status code 402, Payment Required, which sat unused for decades. When an AI agent requests a page, the server can answer with a 402 and a price instead of the content. The agent's wallet pays automatically, usually a fraction of a cent, and the server then returns the page. There is no account to create, no invoice, and no human approval step. Cloudflare's Monetization Gateway implements x402 at the network edge so site owners do not have to build the payment logic themselves.

How much can I charge an AI agent to crawl my site?

You set the price. x402 and Cloudflare's Pay Per Use model let you quote any amount per request, from a fraction of a cent up. The right number depends on how valuable your content is to the agent and how often it fetches. Crawlytics helps you decide by showing which bots hit which pages and how often, so you can price high-demand content higher and leave low-value paths free.

Do I need cryptocurrency to use x402?

Not in the way most people fear. The payments settle in stablecoins in the background, but Cloudflare's Monetization Gateway handles that layer for you. From your seat it looks like a price on a request and money arriving in an account. You are not trading tokens or watching charts. The mechanism is closer to a metered API bill than to speculative crypto, much as accepting a credit card does not require you to understand card networks.

Is Crawlytics a competitor to Cloudflare's Monetization Gateway?

No. They do different jobs and work well together. Cloudflare enforces payment at the edge, acting as the tollbooth. Crawlytics gives you per-site visibility into which agents are visiting, generates the llms.txt that advertises what content is available, and adds WebMCP so agents can transact on your pages. Cloudflare collects the toll; Crawlytics tells you who is at the gate and what they want.

What is Pay Per Use and how is it different from Pay Per Crawl?

Pay Per Crawl was Cloudflare's first version, charging a bot each time it crawled a page. Pay Per Use is the evolution announced alongside the freshness-signal research: it charges based on how an agent actually uses your content, not just the act of fetching it. The shift matters because an AI search engine that reads one page to answer a live question is worth more to you than a training crawler that scrapes everything once. Pay Per Use is built to price that difference.