# Pay per crawl: charging AI crawlers with HTTP 402
Table of Contents
Anyone who publishes content online has long faced a stark choice when the AI “crawlers” show up. A crawler is an automated program that downloads web pages: the ones run by AI companies do this to train their models or to gather material for answering user questions. Until recently, site owners had only two options: let them through for free, or shut them out. In early July 2025, Cloudflare - a large network infrastructure company that sits between a huge share of web traffic and the sites behind it - proposed a third path: make them pay. The mechanism is called “pay per crawl”, and it revives an almost-never-used HTTP code: 402, “Payment Required”.
The problem: block everything, or give it all away
To see why this matters to people who produce content, it helps to start with the context. A site that hosts articles, photos or data costs money to produce and to keep online. When an AI company’s crawler downloads that material, it effectively takes it and leaves nothing behind: no human visit, no ad seen, no subscription.
The traditional tool for managing crawlers is the robots.txt file, a simple list of rules where a site declares “you may download this, not that”. It has two limits, though. First, it’s only a polite request: it stops no one who decides to ignore it. Second, it speaks a binary language, yes or no. There’s no “yes, but for a fee” box.
The result is an uncomfortable ultimatum for publishers. Blocking crawlers entirely means giving up any return and, sometimes, vanishing from AI-generated answers. Leaving them free means giving your work away. Cloudflare says it heard this frustration from news organizations, publishers and large platforms: many would like a third option, allowing access but being compensated for it.
What pay per crawl is
Pay per crawl is exactly that third option. For each crawler, a site owner can choose among three behaviors:
- Allow - free, open access, as before.
- Charge - access is granted, but at a set price.
- Block - access denied, with no option to pay.
The core idea is simple: turn crawler access into a small economic transaction, without having to negotiate case by case with every AI company. For now the price is a single figure that applies across the whole domain: you set an amount per successful request, and it covers every page.
How it works: the return of HTTP 402
Here’s the technical part, kept simple. Every time a browser or a program asks a server for a page, the server replies with a “status code”: a number that sums up how it went. 200 means “all good, here’s the content”; 404 means “not found”; 403 means “forbidden”. Among these there’s one, 402 “Payment Required”, defined since the earliest versions of the HTTP standard but kept on the shelf for years, waiting for a concrete use. Pay per crawl is one of those uses.
The exchange, simplified, works like this. A crawler asks for a paid page without offering anything: the server replies with a 402 and adds a header (a line of metadata that travels with the response) called crawler-price, the amount being asked. At that point the crawler, if it accepts, repeats the request stating how much it’s willing to pay through the crawler-exact-price header. If everything checks out, the server responds with a 200 and the content, adding crawler-charged to confirm the amount billed.
There’s also a more direct route: the crawler can declare a spending cap in its very first request using the crawler-max-price header. If the site’s price falls within that cap, the content arrives immediately, with the charge confirmed. In practice the 402 isn’t an error to avoid but a way of saying “you can have this, but it costs X”: a kind of price tag returned on the fly by the server.
Telling a paying crawler apart
A scheme like this only holds up if the server knows for certain who it’s dealing with; otherwise anyone could pose as an authorized crawler. That’s why pay per crawl relies on cryptographic signatures. Each participating crawler holds a key pair (a private one, kept secret, and a public one, shared) and signs its requests with the private key; the server verifies the signature with the public key. It’s the same principle that makes many secure communications on the web trustworthy: the signature proves identity and guarantees the request wasn’t tampered with along the way. This ties the charge to a recognized sender, not an anonymous one.
Who collects and who pays: Cloudflare as the middleman
One practical problem remained: no publisher wants to handle micro-payments from dozens of AI companies, and no AI company wants to open an account with every single site. Cloudflare steps in as the “merchant of record”, the party that appears as the official seller in the transaction. In practice it collects the charges as they accrue, bills the crawler operators and passes the proceeds to publishers. For the site, this means not having to build a payment system from scratch; for the crawler, a single counterpart instead of a thousand.
It should be said plainly: to use pay per crawl you have to be a Cloudflare customer and route your site through its network. It’s a commitment to one provider, not an open standard anyone can adopt on their own.
What changes for content creators
The value of the proposal isn’t the individual payment, which on its own will often be tiny, but the fact that it finally puts a price on something you could previously only give away or deny. For a news outlet, or for anyone running a rich, constantly updated archive, being able to say “you’re welcome here, but this work has a cost” is a shift in perspective, if only as bargaining leverage.
Expectations should stay sober, though. The mechanism only works if AI companies choose to take part and pay, rather than simply not downloading the site. A single price for the whole domain is still inflexible. And the entire structure lives inside one vendor’s infrastructure. These are the things that make it an interesting experiment more than a mature solution.
It’s a beta, and it’s eyeing “agents”
Precision matters here: at the time of the announcement, pay per crawl is in private beta. It isn’t an open service or a settled standard; to try it you have to sign up and go through restricted access. The headers, prices and rules described here are those of the early phase and may change as the experiment progresses.
There is, however, a broader direction that explains why the news drew attention. Cloudflare frames it within a larger scenario, that of “agents” - AI programs acting on a person’s behalf. The idea, openly forward-looking, is that one day you might hand an assistant a budget and let it independently buy the best content for a task. Whether that future arrives or not, today’s concrete signal is more modest but not trivial: taking a forgotten piece of the HTTP standard, the 402 code, and trying to use it to put a price back on automated access to content.
