# AI crawler reads: the sensor install guide

The site check tells you whether the AI crawlers are allowed in. The crawler sensor tells you who actually came, and which pages they read. This is how to install it, prove it works, and read what it shows.

## What it is, and what it is not

An AI crawler is a program that fetches your pages so an answer engine can quote them: GPTBot and ChatGPT-User for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Googlebot for Gemini and AI Overviews, and others. Each announces itself in its user agent. The sensor runs where those requests arrive, notices that string, and sends Oftheard one line per visit: the time, the path, the crawler and the status your site answered with.

It is not a script tag. The AI crawlers read pages without running anything in them, so a tag in the page would never see one. The sensor sits at the edge (a Cloudflare worker) or on the server (a WordPress plugin, or a few lines of your own). Nothing about your human visitors is sent, except a visit that arrived from an AI answer, which is kept as the assistant and the page only.

**Privacy, in short.** A line is kept only when the user agent is a known AI crawler, or the visit came from an AI answer. The server drops anything else, whatever a sensor sends. No address of anyone is kept. Records are deleted after ninety days. Removing the plugin or the worker stops the sending at once, and so does turning the sensor off. The [privacy page](https://oftheard.in/privacy) says the same in the words that bind us.

## Step 1: open the sensor for your business

The sensor is part of a Starter or Growth campaign; it is step 2 of a campaign’s setup, and the business’s **AI crawlers** page shows its reads. One sensor serves the business: every campaign reads from it.

Opening it makes a site key for your domain, which looks like `oh_` and thirty-two letters and digits. The key is bound to your domain or a subdomain of it, and the beacon refuses a report about any other host, so a key can never report somebody else’s site. **Make a new key** stops the old one at once; **Turn the sensor off** stops reporting. Reads already stored are kept.

The key is filled in with the business’s own website, for example `example.com`. If the sensor will run on a subdomain, such as `shop.example.com`, the key can be bound to that instead; it cannot be bound to any other domain. The key identifies the site, not you: it sits inside the plugin or the worker, and if it leaks you make a new one.

## Step 2: install it

Pick where your site runs. Each takes about five minutes, once.

### WordPress

- Press **Download the plugin**. Your key is already inside; there is nothing to type.

- Sign in to your WordPress admin, the address that ends in `/wp-admin`.

- In the left menu open **Plugins**, press **Add New Plugin**, then **Upload Plugin** at the top of the page.

- Choose the zip you downloaded, press **Install Now**, then **Activate**.

The plugin notes the crawler early in the request and posts one line without waiting for a reply, so a crawler is never served a slower page for being counted. It also notes a visitor who arrived from an AI answer, as the assistant and the page only. It has no settings screen, and its readme names the version.

### Cloudflare

First, check whether a worker already serves your site. In **Workers & Pages**, open each worker’s **Settings**, then **Domains & Routes**. If one lists your domain, do not create another (see below). No worker yet? Then:

- Sign in to the Cloudflare dashboard and open **Workers & Pages**.

- Press **Create**, then **Create Worker**. Give it a name such as `oftheard-reads` and press **Deploy**. That publishes a placeholder; the next step replaces it.

- Press **Edit code**, select everything in the editor and delete it, paste the worker Oftheard shows (the key is inside), and press **Deploy** again.

- Open the worker’s **Settings**, then **Domains & Routes**. Press **Add**, choose **Route**, pick your domain, and enter `example.com/*` with your own domain. If your site also answers on `www`, add `www.example.com/*` as a second route.

Already have a worker on this domain? Two workers cannot share a route, so paste only the block between “Oftheard: start” and “Oftheard: end” into its fetch handler instead, right after the line that gets the response, and press **Deploy**. The block is self-contained: it carries the key, the beacon address and the crawler pattern. The worker fetches your site exactly as before and posts the line after the response has gone out, so a crawler is never slowed.

### Anywhere else

Any server can report a visit in a few lines: a Node, PHP, Python or Java server, or middleware in front of it. Where you handle a request, read the `User-Agent` header; if it matches an AI crawler, send one POST to the beacon with the body in the [reference below](#the-beacon-for-developers). Send it after the response has gone out, or from a background thread, so no visitor waits. Never send anything for a request that did not match: the beacon drops it, but it should not leave your server. Oftheard shows the address, the key and the crawler pattern to use.

## Step 3: test it

Press **Test the sensor**. Oftheard opens your home page the way GPTBot would, marked as a test, and waits up to eight seconds for the sensor to report it.

| Answer | What it means | What to do |
| --- | --- | --- |
| Working | The visit reached your site and the sensor reported it. The sensor is marked tested, with the time. | Nothing. Real visits accumulate; the first usually comes within a day. |
| Your site did not answer | The fetch failed, so the sensor was never reached. Usually the site turns AI crawlers away at the edge. | Run the site check; if it reports AI crawlers blocked, fix that first. |
| Reached your site, nothing came back | The plugin or the worker is not live on that address, or carries an old key. | Check the plugin is active, the worker’s route covers the domain, and the key inside is the current one. |

A test can be run once every 30 seconds, since it is a request to your site. A test visit is stored like any other, under GPTBot, so it also appears on the crawlers page.

## Reading the crawlers page

- **The window**: the last 7, 30 or 90 days.

- **The tiles**: AI visits, and the three kinds of read. **Answering someone now** is a live fetch, a page opened while somebody was asking, so it could be quoted in that answer. **Saved for later** is an index read, which stocks the copy most citations are drawn from. **Training only** feeds a model’s memory and is not a citation. And **people sent to you** from AI answers.

- **When AI visited**, day by day, or hour by hour for the last day.

- **Which AI reads you** and **each AI program**, with the pages each opened and when it last came.

- **Pages AI read most**, and **problems AI hit**: a 404 is a page a crawler was sent to that is not there; a 403 is a crawler being turned away.

- **Every page: read or missed**, your sitemap against the reads.

In a campaign, the reads are also counted per week, and each page published by the campaign shows its reads before and after it went live, because a fix only counts once an engine has read the changed page.

The page also says whether the sensor is reporting: when it last reported, through which sensor, and when it was last tested. The lines themselves are kept ninety days; a campaign keeps its own weekly counts after that, so its twelve weeks stay whole.

## The beacon, for developers

One endpoint, one method, one body.

```
POST https://oftheard.in/api/crawl/beacon
Content-Type: application/json

{
  "key":   "oh_0123456789abcdef0123456789abcdef",
  "host":  "example.com",
  "via":   "server",
  "events": [
    { "at": "2026-10-22T08:44:34Z", "path": "/pricing",
      "ua": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
      "status": 200 }
  ]
}
```

| Field | Meaning |
| --- | --- |
| key | The site key. |
| host | The host the visit was to: the one the key is bound to, or a subdomain of it. www. is ignored. Any other host is refused with 403. |
| via | Which sensor sent it: worker, wordpress, or a short word of your own. |
| events | Up to one hundred visits per body. Batch them if you can. |
| events[].at | When, as an ISO 8601 time. Optional: left out, or more than a day from now, it is replaced by the time of arrival. |
| events[].path | The path and query, up to 500 characters. A leading slash is added if it is missing. |
| events[].ua | The request’s user agent. Not a known AI crawler: the event is dropped and counted in dropped. |
| events[].status | The status your site answered with, as a number. |
| events[].ref | For a person, not a crawler: the referrer or utm_source, when it is an AI assistant (chatgpt.com, perplexity.ai, gemini.google.com, copilot, claude.ai and others). Kept as the assistant and the path only. |

Requests that carry the header `x-oftheard-probe: 1` are Oftheard’s own site check testing crawler access, and the sensors do not report them. The sensor test is sent without it, so it is reported.

The reply is JSON.

| Status | Body |
| --- | --- |
| 200 | { "ok": true, "kept": 1, "dropped": 0 } |
| 400 | The body is not JSON, has no valid key, or names no host. |
| 403 | The host is not the one the key is bound to, or a subdomain of it. |
| 404 | No such key. After twenty unknown keys in a minute from one address, the answer is 429, so keys cannot be guessed. |
| 405 | Anything but POST. |
| 429 | More than 120 bodies a minute for one key. Batch them. |

Bodies are capped at sixty kilobytes. There is no authentication beyond the key, which is why the key is bound to a host and rate-limited: the worst a leaked key can do is add lines about the site it was made for, and making a new key ends that.

## The crawlers recognised

Matched by these words in the user agent, ignoring case. ChatGPT comes as three, and which one you see says what happened: **OAI-SearchBot** builds the index ChatGPT’s cited answers are drawn from, **ChatGPT-User** opens a page live during a conversation, and **GPTBot** collects training material and never puts a page in an answer.

OAI-SearchBot visits on OpenAI’s schedule, not when somebody asks, so a day with no read from it is not a day without ChatGPT citations. ChatGPT-User appears only when the model chooses to open your page during a conversation, which is a minority of citations. In the table, **live** reads a page at the moment somebody asks, **index** builds the index an answer is drawn from, and **training** feeds a model’s memory and is not a citation.

| Name in the user agent | Feeds | Kind |
| --- | --- | --- |
| OAI-SearchBot | ChatGPT, the search index its cited answers are drawn from; visits on OpenAI’s schedule, not per question | index |
| ChatGPT-User | ChatGPT, opens the page live during a conversation, only when the model chooses to | live |
| GPTBot | ChatGPT, training only; never puts a page in an answer | training |
| Perplexity-User | Perplexity, live fetch | live |
| PerplexityBot | Perplexity, indexing | index |
| Claude-SearchBot | Claude web search, indexing | index |
| Claude-User | Claude, live fetch when a user asks | live |
| ClaudeBot | Claude, training | training |
| anthropic-ai | Anthropic, older agent | training |
| Google-CloudVertexBot | Vertex AI, grounding for customer agents | index |
| Googlebot | Google Search, AI Overviews and Gemini, indexing | index |
| Bingbot | Bing and Copilot, indexing | index |
| Applebot | Siri and Apple Intelligence | index |
| meta-externalfetcher | Meta AI, live fetch when a user asks | live |
| meta-externalagent | Meta AI, training | training |
| MistralAI-User | Le Chat, live fetch | live |
| DuckAssistBot | DuckDuckGo AI answers, live fetch | live |
| Amazonbot | Alexa and Rufus | index |
| YouBot | You.com | index |
| cohere-ai | Cohere, training corpus | training |
| Bytespider | ByteDance | training |
| CCBot | Common Crawl, feeds many models | training |
| AI2Bot | Allen Institute, open training corpora | training |

Google-Extended and Applebot-Extended are not here because they are robots.txt names, not user agents; no request carries them. The page reports what claimed to be a crawler: a scraper sending GPTBot’s name is counted as GPTBot.

## Troubleshooting

| Symptom | Cause and fix |
| --- | --- |
| The test says the site did not answer | The website on the business is wrong, or the site turns crawlers away at the edge. Run the site check; its crawler access rows say which. |
| The test reaches the site but nothing comes back | The sensor is not live where the key is bound. On WordPress, confirm the plugin is activated on that site. On Cloudflare, confirm the route covers the bound domain, including www if you use it, and that only one worker holds that route. |
| The beacon answers 403 | The host in the body is not the one the key is bound to. Fix the host the sensor sends, or bind the key to that subdomain. |
| The beacon answers 404 | The key is unknown: a new key was made, or the sensor was turned off. Install the current one. |
| Nothing after a week | The sensor works and no crawler has come. Check the site is in a sitemap the search engines know, and that robots.txt does not disallow the AI crawlers. |
| Only Googlebot and Bingbot appear | Normal for a new site. The AI companies’ crawlers arrive once the site is cited somewhere they read; see how to rank on ChatGPT. |
| My site is on Shopify, Wix or a similar host | Those hosts allow neither plugins nor a worker in the request path. Put the domain behind Cloudflare and use the worker. |

## Questions

### Can I paste a script tag on my pages instead?

No. The AI crawlers fetch the page’s HTML and do not run its JavaScript, so a tag in the page would never see them. The sensor has to sit where the request arrives.

### Does the sensor record my visitors?

No. A line is kept only when the user agent is a known AI crawler, or when a visitor arrived from an AI answer, and then only the assistant and the page. The server drops anything else. No address of anyone is kept, and records are deleted after ninety days.

### Does it slow my site down?

No. The plugin posts without waiting for a reply, and the worker posts after the response has gone out, so neither a crawler nor a visitor waits for it.

### How do I know it is working?

Press **Test the sensor**. Oftheard fetches your home page as GPTBot would and tells you whether the report came back, and the sensor shows the time it was last tested.

### What if my site key leaks?

Press **Make a new key** and install the new download or the new worker. The old key stops being accepted at once, and the reads already stored are kept.

### Which plans include it?

Starter and Growth: the sensor is part of every campaign, step 2 of its setup. The free site check says whether the crawlers are allowed in; the sensor says who came. See [pricing](https://oftheard.in/pricing).

All documentation: https://oftheard.in/docs