AI Readability
Check whether AI crawlers can get into your site, whether they can read your pages without JavaScript, and the state of your llms.txt and other AI files.
AI search tools and AI assistants can only use pages that their crawlers can fetch and read. The AI readability page answers three questions:
- Can AI crawlers get in? Your robots.txt and your server must let them in.
- Can AI bots read your pages? Most AI crawlers do not run JavaScript. They read only the HTML that your server sends.
- What is the state of your AI files? For example,
llms.txt.
To open it, select Site Health Monitoring > AI readability in the sidebar.
This page tells you what AI crawlers can read. It does not tell you what a specific AI product shows in its answers.
Can AI crawlers get in?
This card compares your robots.txt with 18 AI crawlers. It needs Robots.txt Monitoring data. If Robots.txt Monitoring has no data for the domain, the card shows a link to set it up.
The sentence at the top gives the verdict, for example "AI search and AI assistant bots can read your site." Only AI search and AI assistant bots count for the verdict. AI training bots do not count, because blocking AI training is a business decision, not a problem.
The bots
| Used for | Bots |
|---|---|
| AI search | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Bingbot (Microsoft) |
| AI assistants | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity), Meta-ExternalFetcher (Meta) |
| AI training | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta), Amazonbot (Amazon), cohere-ai (Cohere), anthropic-ai (Anthropic) |
Google-Extended and Applebot-Extended never visit your site. They are robots.txt settings that other crawlers of the same company read.
The table shows the bots with a problem first.
robots.txt column
| Chip | Meaning |
|---|---|
| Allowed | robots.txt lets the bot in. |
| Blocked | robots.txt keeps an AI search or AI assistant bot out of the whole site. This is a problem. |
| Blocked on purpose | robots.txt keeps an AI training bot out. This is your decision, not a problem. |
| Partly blocked | robots.txt keeps the bot out of some paths. Hover to see the paths and the share of your pages. |
| Not checked yet | VitalSentinel did not compare the bot with your robots.txt yet. |
Your server column
If Server Log Monitoring is connected and has data, the table also shows Your server and Visits. These columns show what your server really did when the bot visited:
| Chip | Meaning |
|---|---|
| Lets in | Your server answered the bot's requests. |
| Turns away N% | Your server refused part of the visits from an AI search or AI assistant bot, although robots.txt lets it in. Usually, a firewall, CDN or rate limit causes this. |
| Worth checking | Your server refused part of the visits from an AI training bot. |
If Lets in shows for a bot that robots.txt blocks, the bot ignores your robots.txt. To stop it, block it in your firewall.
How to fix
A bot with a problem has a How to fix link:
- Blocked by robots.txt: the dialog shows the robots.txt lines to add or replace, with a Copy button. It tells you which lines to remove and which group to change.
- Refused by your server: the dialog shows how many visits your server refused. Add an exception for the bot in your firewall, CDN or security plugin.
More cards about access
- How fast your server answers AI bots shows how long each AI bot waited for a page, compared with Googlebot. It needs Server Log Monitoring. If a bot waits at least 3 times as long as Googlebot, check whether your server or CDN slows it down.
- What your robots.txt asks AI tools shows the content signals and
Content-Usagerules in your robots.txt. These rules are requests, not blocks. Crawlers can ignore them.
Can AI bots read your pages?
Most AI crawlers do not run JavaScript. If the main text of a page appears only after JavaScript runs, these crawlers see an empty page.
This card compares the HTML that your server sends with the page after JavaScript runs. It gives a verdict for each section of your site:
| Verdict | Meaning |
|---|---|
| Readable | At least 80% of the main text is in the HTML. |
| Partially readable | Between 30% and 80% of the main text is in the HTML. |
| JavaScript-dependent | Less than 30% of the main text is in the HTML. |
| Not measured yet | VitalSentinel did not check enough pages yet. |
The table of sections shows the Verdict, Text in the HTML and Need JavaScript for each section. Text in the HTML is the average share of the main text in the HTML. Need JavaScript is the number of pages that show part or most of their main text only after JavaScript runs.
To fix a section that is not readable, send the main content in the HTML. Use server-side rendering or pre-rendering.
VitalSentinel cannot see if your site sends different HTML to AI crawlers than to other visitors.
More cards about readability
- Pages that need JavaScript for their text lists these pages with their section. Click a page to open its panel. Use the Page address filter at the bottom of the screen to narrow the list.
- What only appears after JavaScript shows which elements appear only after JavaScript runs: Links, Headings, Structured data, Canonical and Robots meta. The first row is the Whole site.
- Text repeated across pages shows sections in which most pages repeat the same text. Give each page its own main text. A section needs at least 5 pages with enough text for a result.
AI files
The AI files card checks optional files for AI tools. None of these files have an effect on Google rankings.
/llms.txt, /llms-full.txt and Markdown copies come from a community proposal, not an official standard, so AI tools may ignore them.
| Item | What VitalSentinel checks |
|---|---|
| /llms.txt and /llms-full.txt | Whether the file is there, whether its format is valid, and whether its links work. A missing file is not a problem. Add one only if you want to point AI tools to your best pages. |
| Markdown copies | How many of the checked pages link to a Markdown copy of the page. |
| Article markup | How many of your article pages name an author and have a date. A date that cannot be correct shows as a problem. |
The Status column shows Present, Not present, Has errors or Worth a look. VitalSentinel looks for /llms.txt and /llms-full.txt once a day.
If Server Log Monitoring is connected, the card also shows how many times bots requested each file in the last 30 days.
Page panel
Each page panel in Site Health Monitoring has an AI & crawlers tab. The same tab is in the Page Monitoring page panel. It shows:
| Section | What it shows |
|---|---|
| Readability without JavaScript | The verdict for this page and the share of its main text that is in the HTML |
| robots.txt | Whether robots.txt lets Our crawler and Googlebot crawl this page |
| AI crawlers | Whether robots.txt lets each AI crawler read this page, and the rule that applies |
| AI and snippet directives | The tags that limit how AI tools and search results can use the page, for example nosnippet, noarchive, noai and noimageai, and whether a meta robots tag or an X-Robots-Tag header sets them |
Stop alerts for AI bot changes
If you block or allow some AI bots on purpose, you can stop issues about their robots.txt changes. See Skip robots.txt alerts for AI bots.
Related
Issue Types
Every problem that Site Health Monitoring can find, grouped as in the Problem filter, with its typical severity and what it means for your site.
Settings
Configure Site Health Monitoring - how many pages to check and which come first, page rules, pinned pages, robots.txt, exceptions, and crawler access.
