Web Bot Auth: identifying bots with cryptographic signatures

Anyone running a website gets a lot of daily visits that don’t come from people, but from automated programs. Some are useful - a search engine crawler, a feed aggregator, an uptime monitor - and some aren’t, like the ones that copy content or probe for holes. On top of these there are now AI agents: assistants that browse the web on a user’s behalf to answer a question or carry out a task. The problem is old but increasingly pressing: how do you tell a legitimate bot from one that’s only pretending to be?

In mid-May 2025 Cloudflare, one of the world’s largest network and CDN operators, put forward a proposal to answer that question. It’s called Web Bot Auth. The core idea is simple. Instead of guessing who’s behind a request from clues that are easy to fake, you ask the bot to sign its requests with a verifiable cryptographic signature.

The problem: recognising bots today is fragile

A “bot” is simply a program that sends web requests without a person clicking in real time. Today, anyone trying to work out whether a bot is trustworthy has only weak tools to lean on.

The User-Agent. This is a line of text every request carries to declare “who I am” (for example the name of the browser or crawler). The trouble is that it’s free text: anyone can write whatever they like in it. Impersonating a search engine crawler is trivial.
IP address ranges. The IP address is the “phone number” a request comes from. Some operators publish the ranges their bots use, so sites can build allowlists (lists of permitted addresses). But it’s a brittle mechanism: IPs change, they’re often shared between different services, and many tools go through proxies or VPNs that hide their origin. Keeping those lists up to date is constant, imperfect work.
Secret tokens. You could hand each bot a password to present to the site. That doesn’t scale: it would need a separate credential for every site to visit, and a prior agreement with each one.

The upshot is that, today, blocking bad bots without hitting the good ones is an exercise in approximation.

What Web Bot Auth is

Web Bot Auth flips the approach: rather than guessing a bot’s identity, it asks the bot to prove it with cryptography.

A quick reminder of what a digital signature is. Whoever signs holds two linked keys: a private one, kept secret, and a public one, shared with everyone. The private key “seals” a message; the public key lets anyone verify that the seal is authentic and that the message hasn’t been altered. The important part is that only the holder of the private key can produce a valid signature: you can’t forge one by looking at the public key, the same key-pair principle that lets WireGuard do away with passwords.

Web Bot Auth applies this principle to HTTP requests. A legitimate bot generates a key pair, publishes the public one at a known location, and signs every request with the private one. The receiving site fetches the public key and verifies the signature. If it checks out, the site knows for certain which operator the request came from.

Underneath sits a standard already published by the IETF (the body that maintains Internet’s technical standards): RFC 9421, HTTP Message Signatures, from February 2024. It defines, in general terms, how to sign parts of an HTTP message and how to attach the signature through two headers, Signature-Input and Signature. Web Bot Auth reinvents nothing: it’s a specific way of using that standard for the bot case.

How it works, in practice

Before sending a request, the bot signs some of its elements with its private key. In particular it signs the target authority - the domain of the site it’s writing to - so the signature only holds for that recipient and can’t be reused elsewhere. The request then travels with three extra headers:

Signature-Input - describes what was signed and with which parameters: a validity window (the moments it was created and when it expires), an identifier for the key used, and a tag flagging the purpose, web-bot-auth.
Signature - contains the signature itself.
Signature-Agent - points to where the bot’s public keys can be found, in a standard format, so the site knows where to go and verify.

In simplified form, a signed request looks like this:

GET /article HTTP/1.1
Host: www.example.com
Signature-Agent: signer.example-bot.com
Signature-Input: sig=("@authority" "signature-agent");
  created=1700000000; expires=1700003600;
  keyid="ba3e64=="; tag="web-bot-auth"
Signature: sig=:abc...==:

The validity window matters: the signature expires quickly, so even if someone intercepted it they couldn’t reuse it for long. And because it’s tied to the target domain, it won’t work against a site other than the intended one.

To make adoption easier, Cloudflare released working examples you can already use: an npm package (web-bot-auth) to generate signatures with Ed25519 keys, a Chrome browser extension, a plugin for the Caddy web server that shows server-side verification, and a public test server to experiment against.

Why it’s more reliable than IP allowlists

The practical difference is sharp. An IP list says, at best, “this request comes from an address that usually belongs to a certain operator” - indirect information that goes stale and can be worked around. A cryptographic signature instead says “this specific request, to this specific domain, within this time window, was produced by whoever holds that private key”.

You no longer have to chase addresses that change, or trust a User-Agent anyone can write. Verification doesn’t depend on the network path: it works even if the bot goes through proxies or shared infrastructure. And the identity doesn’t have to be arranged site by site: the bot publishes its key once, and anyone can verify it.

What changes for site operators

For someone running a site, the prospect is being able to decide more finely who gets in. Instead of reasoning by lists of addresses, you can reason by verified identities: grant access to a search engine crawler or to an AI agent you want to make room for, and treat the rest differently. Cloudflare has stated its intention to fold this mechanism into its own bot management and AI traffic analysis tools, in keeping with other recent moves such as self-managed OAuth.

It’s worth being clear that this signature proves who sent the request, not what they’ll then do with it. A signed bot is still subject to the site’s rules: authentication is the premise for applying sensible permissions, not an automatic pass.

Where it stands

Some caution is needed here, because none of this is settled yet. At launch, Web Bot Auth started as a closed beta, that is a limited-access trial phase.

The specifications describing it are IETF Internet-Drafts (for example the draft on the architecture and the one on the format of the key directory): working documents, still evolving and not approved as standards. It’s worth remembering that an Internet-Draft, on its own, has no official standing in the IETF process until it completes the path. The standard underneath it, RFC 9421, is by contrast already published and stable.

Cloudflare also described a more experimental alternative based on mTLS (mutual authentication at the TLS layer, with certificates on both client and server side), but it points to HTTP message signatures as the main route, because they’re simpler to adopt without touching the TLS layer.

In short: it’s a concrete, already testable proposal, not a finished product. The idea of identifying bots with cryptographic proof rather than spoofable clues is sound, and it arrives at the right moment, with AI agents multiplying automated traffic. But its real value will depend on how many bot operators start signing and how many sites learn to verify. For now it’s worth following, understanding how it works, and treating it as a complement to existing defences - not yet a replacement for them.

# Web Bot Auth: identifying bots with cryptographic signatures