AI crawler reads: the install guide

What it is, and what it is not

An AI crawler is a program that fetches your pages so an answer engine can quote them: GPTBot and ChatGPT-User for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Googlebot for Gemini and AI Overviews, and a dozen more. Every one of them announces itself in the request with a user agent string. The sensor is a few lines that run where those requests arrive, notice that string, and send Oftheard one line per visit: the time, the path, the crawler's name and the status your site answered with.

It is not a script tag. The AI crawlers read pages the way a printer reads a document, without running anything in them, so a tag inside the page would never see one. The sensor has to sit at the edge, in your Cloudflare worker, or on the server, in a WordPress plugin or a few lines of middleware. Nothing about human visitors is sent, and the sensor changes nothing about how your site answers.

Step 1: make a site key for your domain

The reads are part of the Growth plan. On Free or Starter the panel shows what the plan adds and a way to request it.

1. Sign in and open Settings, then the panel AI crawler reads.
2. In Bind to, confirm the domain the key is for. It is filled in with the website on your account, for example example.com. You may bind the key to a subdomain of it instead, such as shop.example.com, if that is where the sensor will run. A key cannot be bound to any other domain: the beacon refuses a report from a host that does not match, so a key can never report somebody else's site.
3. Press Make a key. The key looks like oh_ followed by thirty-two letters and digits. It identifies your site, not you; it sits inside the plugin or the worker, and if it leaks you rotate it.

One key per account. Rotating makes a new one and stops the old one at once; stored reads are kept. Revoking stops reporting until a new key is made.

Step 2: install the sensor

Pick the route that matches your site. Each takes about five minutes, once.

WordPress

1. On the settings panel, press Download the WordPress plugin. Your key is already inside the file, so there is nothing to type.
2. Sign in to your WordPress admin, the address that ends in /wp-admin.
3. In the left menu open Plugins, press Add New Plugin, then Upload Plugin at the top of the page.
4. Choose the zip you downloaded, press Install Now, then Activate.

The plugin hooks the request early, checks the user agent, and posts one line to Oftheard without waiting for a reply, so a crawler is never served a slower page for being counted. It has no settings screen. Its readme names the version.

Cloudflare

First, check whether a worker already serves your site. In Workers & Pages, open each worker's Settings → Domains & Routes. If one lists your domain, two workers cannot share the route, so do not create another. Open that worker, press Edit code, paste the block between the Oftheard: start and Oftheard: end markers into its fetch handler right after the line that gets the response, and press Deploy. The block is self-contained: it carries the key, the beacon address and the crawler pattern.

No worker yet? Then:

1. Sign in to the Cloudflare dashboard and, in the left menu, open Workers & Pages.
2. Press Create, then Create Worker. Give it a name such as oftheard-reads and press Deploy. That publishes a placeholder; the next step replaces it.
3. Press Edit code, select everything in the editor and delete it, paste the worker from the settings panel, and press Deploy again.
4. Open the worker's Settings tab, then Domains & Routes. Press Add, choose Route, pick your domain, and enter example.com/* with your own domain. If your site also answers on www, add www.example.com/* as a second route.

The worker fetches your origin exactly as before and, only when the user agent is an AI crawler, posts a line after the response has gone out. A crawler is never slowed.

Any other server

A Node, PHP, Python or Java server, or an Nginx log shipper, can report a visit in ten lines. Where you handle a request, read the User-Agent header. If it matches an AI crawler, send one POST to the beacon with the body in the reference below. Send it after the response has gone out, or from a background thread. Never send anything for a request that did not match; the beacon drops it, but it should not leave your server.

Step 3: verify it works

On the settings panel or the crawlers page, press Test the sensor. Oftheard fetches your home page the way GPTBot would, marked as a test, and waits up to eight seconds for the sensor to report it. You get one of three answers.

A test visit is stored like any other, under GPTBot, so it also appears on the crawlers page. The button is rate-limited, since it is a request to your site.

Reading the crawlers page

• The status line says whether the sensor is reporting, when it last did, through which sensor, and when the key was last verified. A window switch chooses seven, thirty or ninety days.
• The last 24 hours is one column per hour, split into live fetches, index reads and training reads. Hovering a column says which engines came in that hour and what for; clicking one lists only that hour's reads; the chips above the chart show one engine at a time, and the legend hides a kind. Under it, every read of the day, newest first, with the engine, the program, the page and what the site answered. It is where you see that an engine you just asked sent its indexing crawler rather than opening the page live, which is why the live figure did not move.
• Visitors from AI answers is the same shape for people: three numbers, visits from AI answers reconciled between your sensor and Google Analytics, how many stayed and read, and the share of all visits; then a table per engine with visits, share, stayed, the sensor's live count, landing pages and the last visit. Pick an engine above the chart to see only its weeks and landing pages.
• In the window gives the three kinds of read as numbers, live fetches, index reads and training reads, each with how many different pages it touched, then the same split per engine: ChatGPT, Perplexity, Claude, Google, Bing and the rest, with the pages each opened and when it last came. A live fetch is a page opened at the moment somebody was asking, so it could be quoted in that answer. An index read stocks the copy most citations are later drawn from. A training read feeds a model's memory and is not a citation.
• By crawler is one card each: visits in the window and this week, pages, a bar per day, when it last came, and when it last read the home page.
• Pages read lists every path fetched, how often, by whom, and the status your site answered with on the last visit. A 404 is a page a crawler was sent to that is not there. A 403 is a crawler being turned away.
• Checked pages, and whether they were read joins the site check to the reads: every check that names a page of your site, with the last crawler read of that page. After you verify a fix, a read later than the fix is marked, because a fix only counts once an engine has read the changed page.

The beacon, for developers

One endpoint, one method, one body.

POST https://oftheard.in/api/crawl/beacon
Content-Type: application/json

{
  "key":   "oh_0123456789abcdef0123456789abcdef",
  "host":  "example.com",
  "via":   "server",
  "events": [
    { "at": "2026-09-10T08:44:34Z", "path": "/pricing",
      "ua": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
      "status": 200 }
  ]
}

The reply is JSON.

Bodies are capped at sixty kilobytes. There is no authentication beyond the key, which is why the key is bound to a host and rate-limited: the worst a leaked key can do is add lines about the host it was made for, and rotating it ends that.

The crawlers recognised

Matched by these tokens in the user agent, case-insensitively. Live reads a page at the moment somebody asks a question. Index builds the index an answer is drawn from. Training feeds a model's memory and is not a citation.

ChatGPT comes as three of them, and which one you see says what happened. OAI-SearchBot builds the search index that ChatGPT's cited answers are drawn from; it visits on OpenAI's schedule, not when somebody asks, so a day with no read from it is not a day without ChatGPT citations. ChatGPT-User is the live fetch: it appears only when the model chooses to open your page during a conversation, which is a minority of citations. GPTBot collects training material and never puts a page in an answer.

Google-Extended and Applebot-Extended are not here because they are robots.txt tokens, not user agents; no request ever carries them. The page reports what claimed to be a crawler. Checking claims against the address ranges the vendors publish is planned, and until then a scraper sending GPTBot's string is counted as GPTBot.

Troubleshooting

Questions

Can I paste a script tag on my pages instead?

No. The AI crawlers fetch raw HTML and never run JavaScript, so a tag in the page would never see them. The sensor has to sit where the request arrives.

Does the sensor record my visitors?

No. A line is sent only when the user agent is a known AI crawler, and the server drops anything else. No address of anyone is kept, and records are deleted after ninety days.

How do I know it is working?

Press Test the sensor. Oftheard fetches your home page as GPTBot would and tells you whether the report came back, and the key shows the time it was last verified.

What if my site key leaks?

Rotate it from the settings panel and install the new download or the new worker block. The old key stops being accepted at once. Stored reads are kept.