Related guides: How to rank on ChatGPT, How to rank on Gemini, How to rank on Perplexity, How to get ChatGPT to recommend your business.
19 of the hundred checks are in this group, 8 of them critical. Each is listed with its weight in the score and whether it affects AI search directly, and for each: why it matters, and what to do. The hub page has the other groups and how the score is made.
- AI crawler access
- Search crawler access
- llms.txt
- llms-full.txt
- ai.txt
- TDM reservation
- Content signals
- AI crawler edge access
- Interior pages
- New URLs are real pages
- Citable surface
- Markdown twin
- Answer-shaped content
- Dates & authorship
- JS-free content
- Reached by a public crawler
- Knowledge graph entity
- Bing index
- IndexNow
AI crawler access
criticalAffects AI search
Why it matters. If GPTBot or ClaudeBot cannot fetch your pages, those answer engines literally cannot read your site — no amount of content, schema or link building will make you citable. This is the single most common cause of total AI invisibility, and it is usually an accident: a leftover from the 2023–24 wave of AI blocking that nobody reverted. Google-Extended belongs in this group even though it crawls nothing: Google documents it as governing grounding at prompt time in Gemini as well as training, so a Disallow on it is a site Gemini answers without.
What to do. Remove the Disallow rules for those user-agents in robots.txt. If you want search visibility but not training, allow OAI-SearchBot and Claude-User while disallowing GPTBot — but understand that costs you some visibility too.
Search crawler access
criticalAffects AI search
Why it matters. The answer engines with the largest reach do not use an AI crawler to decide what they know. AI Overviews and AI Mode are generated over Google's ordinary search index, which Googlebot builds; Copilot answers out of Bing's index, which Bingbot builds, and ChatGPT still retrieves from Bing alongside its own OAI-SearchBot. Google-Extended is a training opt-out token and controls no crawling at all, so a site can allow every AI user-agent on the page above, block Googlebot, and be absent from every AI answer Google produces. This is rare, because blocking Googlebot also removes you from search — but when it happens it is usually a staging rule that shipped, and no roster of AI crawlers can see it.
What to do. Remove the Disallow rule for Googlebot or Bingbot in robots.txt. If you meant to block a specific section, scope the Disallow to that path rather than to "/". If you meant to opt out of AI training while keeping search, that is Applebot-Extended, GPTBot or CCBot — never Googlebot, which costs you both at once, and not Google-Extended either, which also switches off Gemini reading you at answer time.
llms.txt
lowAffects AI search
Why it matters. The evidence splits, so judge it yourself. Against: three independent large-scale studies found no measurable effect — Ahrefs across 137,000 sites (97% of these files never fetched), SE Ranking across ~300,000 domains (no statistically significant citation correlation), and Otterly's 90-day crawler logs (0.1% of AI crawler requests touched it). Google's Gary Illyes stated Google does not support it and has no plans to. For: Perplexity is reported to retrieve it, and Anthropic references it in agent-writing guidance. So two engines may read it, but no study has yet shown it changes whether you are cited. It is cheap, and it is not a substitute for anything on this list.
What to do. Twenty minutes of work if you want it. Genuinely useful for docs-heavy or developer products, where it doubles as a human-readable index. Add it if you like — just do not let it displace the scored items, and do not pay an agency for it.
When it is not scored. Not present. Contested rather than settled — see the detail. Optional, and not scored.
llms-full.txt
lowAffects AI search
Why it matters. The full-text companion to llms.txt. Inherits the same unproven-benefit question and adds real maintenance cost, since it must be regenerated whenever your content changes or it silently goes stale and starts misinforming anything that does read it.
What to do. Skip unless you are a documentation platform where the whole corpus genuinely fits in one file and you can regenerate it on every build.
When it is not scored. Not present. Optional, and not scored.
ai.txt
mediumAffects AI search
Why it matters. Worth understanding before you act on it. ai.txt began as Spawning's opt-OUT protocol — a way to refuse AI training — and later grew allow-directives so you can permit search while refusing training. But no major AI company has committed to reading it: GPTBot, ClaudeBot and PerplexityBot all document robots.txt as the file they honour. So an ai.txt full of permissions is largely decorative, while one full of refusals may still be honoured by a future crawler. Absence is neutral. Presence only matters if it says no.
What to do. robots.txt is where AI crawler permission actually takes effect — set it there. Add ai.txt only if you want to state intent explicitly, and if you do, make sure it allows rather than disallows: a copied opt-out template is the one way this file can cost you.
When it is not scored. Not present — which is correct. This file signals opting OUT of AI training, so its absence is the state you want. Not scored.
TDM reservation
mediumAffects AI search
Why it matters. The W3C TDM Reservation Protocol is the machine-readable form of the European copyright directive's opt-out from text-and-data mining: a site says "reserved" or "free" in a well-known JSON file, a response header, or a meta tag, and a crawler with European exposure is expected to read it before it mines a page. No answer engine documents honouring it today, so a reservation costs no citation yet — but it is the one AI opt-out with a law behind it, and the number of crawlers reading it grows rather than shrinks. It is also the one most often set by accident: publishing plugins and some CDNs write "reserved" as a default, and nothing on the page shows it. This row exists so that a refusal you did not make is not the first thing a compliant crawler learns about you.
What to do. If you want to be read, declare it or declare nothing: remove the TDM-Reservation header and any tdm-reservation meta tag, or publish /.well-known/tdmrep.json with [{"location": "/*", "tdm-reservation": 0}]. If the header is set and you did not set it, look at your CDN and your CMS plugins. Confirm with: curl -sI https://yourdomain.com/ | grep -i tdm-reservation curl -s https://yourdomain.com/.well-known/tdmrep.json
When it is not scored. No TDM reservation declared — no /.well-known/tdmrep.json, no TDM-Reservation header, no meta tag. Absence means nothing is refused, which is the permissive state. Not scored.
Content signals
highAffects AI search
Why it matters. The third place permission is expressed, and the newest. Allow and Disallow say whether a crawler may FETCH a page; these say what it may DO with one it was allowed to fetch, so a robots.txt can be perfectly permissive on every access rule and refuse AI use on the line underneath — and a scanner reading only Disallow reports a clean pass. Cloudflare shipped Content-Signal in September 2025 and writes it into the managed robots.txt it serves by default; Content-Usage is the IETF aipref working group's spelling of the same idea. Read the keys carefully, because they are not equivalent: ai-input governs use of your page to ground a generated answer, which is citation, and search governs the ordinary index every AI answer surface is built on. ai-train is corpus inclusion — a licensing decision that costs no citation, and the one most sites actually mean when they set any of this. Nothing enforces these fields today; they are declarations of intent, which is why a refusal here is reported rather than scored as a fault. The reason it is worth reading at all is that it is very often a default nobody chose, arriving with a proxy that was turned on for unrelated reasons.
What to do. Decide the two halves separately. If you want to be citable and merely do not want to be trained on, that is "Content-Signal: search=yes, ai-input=yes, ai-train=no" — the line most sites mean. If ai-input=no is there and you did not put it there, check your CDN: Cloudflare sets it under Security → Bots → AI Scrapers and Crawlers, alongside the managed robots.txt setting. Confirm with: curl -s https://yourdomain.com/robots.txt | grep -i "content-signal\|content-usage"
When it is not scored. No Content-Signal or Content-Usage line in robots.txt. Absence is neutral — it means you have stated no preference, not that you have refused anything. Not scored.
AI crawler edge access
criticalAffects AI search
Why it matters. robots.txt is a request; your CDN is a wall. If Cloudflare, a WAF or a rate limiter refuses GPTBot or ClaudeBot, the crawler never reaches the file that grants it permission — so a perfectly written robots.txt reports a pass while the site is invisible in practice. This became widespread after Cloudflare began blocking AI crawlers by default for new domains in mid-2025, and it is invisible to every scanner that only reads robots.txt. This check fetches your homepage as each crawler and compares what comes back to what a browser gets.
What to do. Cloudflare: dashboard → Security → Bots → review "AI Scrapers and Crawlers", and check Security → WAF for custom rules matching bot user-agents. Other CDNs and WAFs have an equivalent bot-management setting. If you rate-limit by user-agent, exempt the crawlers you want to be read by. Confirm afterwards with: curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" https://yourdomain.com/
Interior pages
criticalAffects AI search
Why it matters. This audit reads your homepage, but answer engines cite interior pages far more often — and templates are where entity data usually breaks. A perfect homepage over a hundred bare product pages is the common failure, and it is invisible to a homepage-only scan.
What to do. Apply the head and schema fixes at the template level, not just the homepage, then re-check a page from each template.
New URLs are real pages
criticalAffects AI search
Why it matters. Two ways a new URL can resolve without being a page, and both pass every other check on this report. An empty shell — a <head> block pasted into a file with no body — has nothing to quote. An alias — the same bytes served under a second path, canonical still pointing at the original — has nothing NEW to quote, and its own canonical tag tells engines to index the other URL instead. Both usually come from doing the routing step of a page split and stopping before the content step, so the site appears to gain pages when it gained addresses.
What to do. Give each URL its own content and its own self-referencing canonical, or delete it and remove it from the sitemap. Splitting a homepage section into a page means moving that section's content to the new URL and writing a canonical that points at itself — serving the whole homepage again under a new path achieves nothing, and an empty file achieves less than nothing.
Citable surface
criticalAffects AI search
Why it matters. Answer engines cite pages that answer one specific question, not sites. A single landing page with anchor links offers exactly one citable unit no matter how good it is, because #pricing is not a URL a model can return. Sites that get cited have a page per question — pricing, comparisons, use cases, docs, a blog. This is the ceiling every technical fix on this page runs into, and no amount of schema or head-tag work raises it.
What to do. Give each question its own indexable URL rather than an anchor on the homepage. Start with the ones people actually ask: pricing, what it does, who it is for, how it compares. One page each, server-rendered, in the sitemap.
Markdown twin
highAffects AI search
Why it matters. AI crawlers execute little or no JavaScript and struggle with heavy page furniture. Serving a clean Markdown version of each page gives them exactly the text, with none of the navigation, modals or tracking. Almost nobody does this, which makes it one of the few remaining easy advantages.
What to do. Serve a .md variant of each page, or honour an "Accept: text/markdown" request header. Link it with rel="alternate" type="text/markdown".
Answer-shaped content
highAffects AI search
Why it matters. Answer engines do not cite pages, they cite passages. The retriever splits your document at its headings, embeds each piece, and returns the one that answers the question — so structure, not just prose quality, decides whether you are quotable. A page that is one unbroken wall of text under a single heading chunks badly: its best sentence ends up in the middle of a chunk about four other things, and never scores as a match. Headings phrased as the question, a direct answer in the first sentence beneath them, and facts in lists or tables are the difference. Every other check on this report decides whether you are read; this one decides whether you are quoted.
What to do. Break long pages into sections under h2s phrased the way people ask — "How much does X cost", not "Pricing". Answer in the first sentence under the heading, in twenty to sixty words, before the context and the caveats. Put comparisons, specs and steps into tables and lists rather than paragraphs. Keep sections under about 400 words so each one survives chunking whole.
Dates & authorship
mediumAffects AI search
Why it matters. Two questions every retrieval system asks of a candidate passage: when was this true, and who says so. An undated page cannot be preferred over a fresher competitor, and for a great many questions freshness is the tiebreak. An unbylined page carries no expertise to attach to its claims. Both are usually missing for the same reason — the template never had a slot for them — which also makes both cheap to fix everywhere at once.
What to do. Put a visible published or updated date on content pages, marked up as <time datetime="YYYY-MM-DD">, and mirror it in datePublished/dateModified. Name a real person as author, link that name to a page about them, and give the person a Person entry whose sameAs points at their professional profiles. Do not fake freshness by bumping dates on unchanged pages — it is detectable and it is the behaviour the signal exists to catch.
JS-free content
criticalAffects AI search
Why it matters. Google renders JavaScript. Most AI crawlers do not — they fetch the raw HTML, read what is there, and leave. If your content only appears after JavaScript runs, answer engines see an empty shell and you cannot be cited, no matter how good the writing is. This is the number one cause of "we publish constantly but AI never mentions us".
What to do. Server-render or pre-render your content. Verify with "curl -s https://yoursite.com | head -100" and check your actual copy is in there.
Reached by a public crawler
highAffects AI search
Why it matters. Every other check on this report measures whether a crawler could read this site. None of them measures whether one ever has. A site can pass all of them — robots.txt open, markup server-rendered, sitemap valid, pages three clicks deep — and have been fetched by nothing, and no on-site evidence can tell you that, because on-site everything is fine. Common Crawl is a public archive built by a robots-respecting crawler that runs monthly, and it is the only free way to ask the question from outside.
What to do. Absence usually has one of three causes and they need different fixes. If the site is newer than the most recent crawl, wait — there is nothing to do. If pages are reachable only through a script-driven menu or a search box, they are undiscoverable to any crawler and need real links. And if the edge is refusing unfamiliar user-agents, that is the blocker check on this report rather than this one, and it is the thing to fix first.
When it is not scored. Common Crawl was not queried. Set CC_LOOKUP=1 to enable it — it is free, unkeyed, and the only check here that reports evidence of crawling rather than permission for it.
Knowledge graph entity
highAffects AI search
Why it matters. Before an assistant can say a sentence recommending you, it has to turn your name into a referent — one organisation, distinct from every other business with a similar name. Your own markup is a claim about yourself; what actually produces a referent is agreement between independent sources, and Wikidata is the largest public one. The point is not that every business belongs on it. The point is that when an entity for you does exist, almost nobody links their site to it, and when one exists with somebody else's website recorded on it, nobody notices.
What to do. If an entity exists, add its URL to the sameAs array on your Organization node — one line, and it is the cheapest edge you can create between your site and the graph. If the entity records a different website, correct it there. If no entity exists, do not create one: notability is judged on coverage elsewhere and a self-created entry gets deleted. Build the coverage — trade bodies, registries, press — and the entity follows.
When it is not scored. Wikidata was not queried. Set ENTITY_LOOKUP=1 to enable it — it is free and unkeyed.
Bing index
criticalAffects AI search
Why it matters. Copilot and ChatGPT's search grounding are built over Bing's index, not Google's. That makes Bing coverage the most direct measurement of answer-engine visibility available at any price, and it is free. The robots.txt check on this report proves Bingbot is permitted; permission is not presence, and the gap between them is where crawl errors, edge blocks and undiscoverable pages live.
What to do. Verify the site at bing.com/webmasters if you have not — it takes a DNS record. Then read the crawl errors first: Bingbot being allowed and failing is a different problem from Bingbot not visiting, and only the errors distinguish them. Submit the sitemap there directly rather than relying on discovery, and turn on IndexNow so changes are pushed instead of waited for.
When it is not scored. Not looked up. Set BING_KEY to enable it — it is free, and unlike everything else on this report it only works for a site you have verified ownership of.
IndexNow
mediumAI search needs this
Why it matters. IndexNow pushes changed URLs to Bing and Yandex the moment they change, instead of waiting for a crawler to come back. For a site that publishes often it is the difference between hours and weeks, and it feeds the same index the answer engines read. Its failure mode is why this is checked rather than assumed: a submission naming a key file that is not served is accepted with a success status and then silently discarded, so a site can have IndexNow configured for years and have pushed nothing, with no error anywhere to notice.
What to do. Generate a key, save it as a plain text file at the root of your site named after the key itself, and confirm you can open that URL in a browser. Then have your CMS POST changed URLs to api.indexnow.org on publish — most platforms have a plugin, and the API is one request with no account.
When it is not scored. This site names no IndexNow key file in its robots.txt, and a key is a random string only its owner knows — so whether IndexNow is set up here cannot be told from outside, and nothing is being claimed either way. Declare the key file with an Allow: line and this check will verify it is actually being served.