Field manual · 19 agents on the register · generated from the detector itself

How they announce themselves

Most automated readers of the web will tell you exactly who they are, if you know where to look. A handful will not. This page lists both, and it is generated directly from the table the detector on this site matches against — so what you read here is what it actually looks for.

The register · by operator

Who sendswhat

OpenAI

OAI-SearchBot
Search index · Respects robots.txt
Indexes pages so ChatGPT search can cite them.
ChatGPT-User
Sent on an errand · Respects robots.txt
Sent when a person asks ChatGPT to go and read a specific page.
GPTBot
Training · Respects robots.txt
Collects public pages that may train future models.

Anthropic

Claude-SearchBot
Search index · Respects robots.txt
Indexes pages for Claude search results.
Claude-User
Sent on an errand · Respects robots.txt
Sent when a person asks Claude to fetch a specific page.
ClaudeBot
Training · Respects robots.txt
Collects public pages that may train future models.

Perplexity

Perplexity-User
Sent on an errand · Respects robots.txt
Live retrieval on behalf of someone asking a question.
PerplexityBot
Search index · Respects robots.txt
Builds the source pool Perplexity answers from.

Meta

Meta-ExternalAgent
Training · Compliance reported as inconsistent
Training collection for Llama and Meta AI. Compliance reported as inconsistent.

ByteDance

Bytespider
Training · Ignores robots.txt
Documented history of ignoring robots.txt.

Amazon

Amazonbot
Mixed · Respects robots.txt
Alexa and Amazon AI products.

Common Crawl

CCBot
Training · Respects robots.txt
Builds the open dataset a great many models are trained on.

Apple

Applebot
Search index · Respects robots.txt
Siri and Spotlight. Apple Intelligence training is governed separately.

Google

Googlebot
Search index · Respects robots.txt
Search index, and the retrieval layer behind AI Overviews.

Microsoft

bingbot
Search index · Respects robots.txt
Bing index and Bing AI features.

Cohere

cohere-ai
Training · Respects robots.txt

Diffbot

Diffbot
Training · Respects robots.txt

Timpi

Timpibot
Training · Respects robots.txt

Webz.io

omgilibot
Training · Respects robots.txt

Three jobs, worth telling apart. A training crawler collects pages that may train a future model. A search index crawler builds the pool an assistant cites from. An errand agent is sent because a specific person asked a specific question about a specific page — there is a human at the other end of it, waiting.

Two names that are not crawlers

The phantoms

These appear in a great many published lists of “AI user agents”. They are not user agents. Nothing ever sends them. They are opt-out tokens you write into robots.txt, and blocking them affects training permission rather than any actual visit — which is why a list that files them alongside GPTBot is a list to stop trusting.

Cryptographic identity · Web Bot Auth

Agents thatsign their name

A user-agent string is a claim, and anyone can type any claim they like. Since 2025 there has been an alternative: an agent holds a keypair, publishes the public half, and signs its requests. The claim becomes checkable.

Header
Signature-Agent — names the domain that vouches for this agent
With
Signature and Signature-Input — HTTP Message Signatures, RFC 9421
Keys at
/.well-known/http-message-signatures-directory on that domain
Bound to
The target authority, so a signature cannot be lifted and replayed against another site
Status
IETF draft, moved to Standards Track in August 2026

This matters more than any detection heuristic on this site. Once an honest agent can prove who it is in one cheap header, an agent that arrives dressed as an ordinary Chrome window and says nothing has made a choice, rather than merely lacked a convention.

Cloudflare’s write-up is the readable introduction.

Undisguised tooling

Not everythingis undercover

A large share of non-human traffic is not hiding at all — it is a script that never thought to dress up. These strings identify themselves plainly, and treating them as adversaries is usually a mistake:

Headless Chrome · python-requests · httpx · aiohttp · Go net/http · node-fetch · axios · Scrapy · libwww-perl · OkHttp · Java HttpURLConnection · curl · Wget · Postman · HTTPie

Practical · if you run a site

What to doabout it

Decide separately about training and about errands. Blocking a training crawler and blocking the agent a reader sent to fetch your page are different decisions with different consequences. Most robots.txt files conflate them.

Do not block on user-agent alone. It is a claim, freely typed. Use it to recognise the honest, not to stop the dishonest.

Prefer signatures where you can check them. An agent presenting a verifiable Signature-Agent has told you something a string never can.

Watch what your own agents do. The traffic leaving your organisation is somebody else’s undercover problem. Set a trap and point one of them at it.

Generated from the same table the detector matches on. Cross-checked against operator documentation and two independent references in August 2026. If something here is wrong or has moved, it is worth telling us — the register is meant to stay correct.