# AI Readability



AI search tools and AI assistants can only use pages that their crawlers can fetch and read. The **AI readability** page answers three questions:

1. **Can AI crawlers get in?** Your robots.txt and your server must let them in.
2. **Can AI bots read your pages?** Most AI crawlers do not run JavaScript. They read only the HTML that your server sends.
3. **What is the state of your AI files?** For example, `llms.txt`.

To open it, select **Site Health Monitoring** > **AI readability** in the sidebar.

This page tells you what AI crawlers can read. It does not tell you what a specific AI product shows in its answers.

## Can AI crawlers get in? [#can-ai-crawlers-get-in]

This card compares your robots.txt with 18 AI crawlers. It needs [Robots.txt Monitoring](/features/robots-txt-monitoring) data. If Robots.txt Monitoring has no data for the domain, the card shows a link to set it up.

The sentence at the top gives the verdict, for example "AI search and AI assistant bots can read your site." Only AI search and AI assistant bots count for the verdict. AI training bots do not count, because blocking AI training is a business decision, not a problem.

### The bots [#the-bots]

| Used for          | Bots                                                                                                                                                                                                                                   |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **AI search**     | OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Bingbot (Microsoft)                                                                                                                                  |
| **AI assistants** | ChatGPT-User (OpenAI), Claude-User (Anthropic), Perplexity-User (Perplexity), Meta-ExternalFetcher (Meta)                                                                                                                              |
| **AI training**   | GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent (Meta), Amazonbot (Amazon), cohere-ai (Cohere), anthropic-ai (Anthropic) |

Google-Extended and Applebot-Extended never visit your site. They are robots.txt settings that other crawlers of the same company read.

The table shows the bots with a problem first.

### robots.txt column [#robotstxt-column]

| Chip                   | Meaning                                                                                         |
| ---------------------- | ----------------------------------------------------------------------------------------------- |
| **Allowed**            | robots.txt lets the bot in.                                                                     |
| **Blocked**            | robots.txt keeps an AI search or AI assistant bot out of the whole site. This is a problem.     |
| **Blocked on purpose** | robots.txt keeps an AI training bot out. This is your decision, not a problem.                  |
| **Partly blocked**     | robots.txt keeps the bot out of some paths. Hover to see the paths and the share of your pages. |
| **Not checked yet**    | VitalSentinel did not compare the bot with your robots.txt yet.                                 |

### Your server column [#your-server-column]

If [Server Log Monitoring](/server-logs/introduction) is connected and has data, the table also shows **Your server** and **Visits**. These columns show what your server really did when the bot visited:

| Chip               | Meaning                                                                                                                                                           |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Lets in**        | Your server answered the bot's requests.                                                                                                                          |
| **Turns away N%**  | Your server refused part of the visits from an AI search or AI assistant bot, although robots.txt lets it in. Usually, a firewall, CDN or rate limit causes this. |
| **Worth checking** | Your server refused part of the visits from an AI training bot.                                                                                                   |

If **Lets in** shows for a bot that robots.txt blocks, the bot ignores your robots.txt. To stop it, block it in your firewall.

### How to fix [#how-to-fix]

A bot with a problem has a **How to fix** link:

* **Blocked by robots.txt:** the dialog shows the robots.txt lines to add or replace, with a **Copy** button. It tells you which lines to remove and which group to change.
* **Refused by your server:** the dialog shows how many visits your server refused. Add an exception for the bot in your firewall, CDN or security plugin.

### More cards about access [#more-cards-about-access]

* **How fast your server answers AI bots** shows how long each AI bot waited for a page, compared with Googlebot. It needs Server Log Monitoring. If a bot waits at least 3 times as long as Googlebot, check whether your server or CDN slows it down.
* **What your robots.txt asks AI tools** shows the content signals and `Content-Usage` rules in your robots.txt. These rules are requests, not blocks. Crawlers can ignore them.

## Can AI bots read your pages? [#can-ai-bots-read-your-pages]

Most AI crawlers do not run JavaScript. If the main text of a page appears only after JavaScript runs, these crawlers see an empty page.

This card compares the HTML that your server sends with the page after JavaScript runs. It gives a verdict for each section of your site:

| Verdict                  | Meaning                                              |
| ------------------------ | ---------------------------------------------------- |
| **Readable**             | At least 80% of the main text is in the HTML.        |
| **Partially readable**   | Between 30% and 80% of the main text is in the HTML. |
| **JavaScript-dependent** | Less than 30% of the main text is in the HTML.       |
| **Not measured yet**     | VitalSentinel did not check enough pages yet.        |

The table of sections shows the **Verdict**, **Text in the HTML** and **Need JavaScript** for each section. **Text in the HTML** is the average share of the main text in the HTML. **Need JavaScript** is the number of pages that show part or most of their main text only after JavaScript runs.

To fix a section that is not readable, send the main content in the HTML. Use server-side rendering or pre-rendering.

<Callout type="info">
  VitalSentinel cannot see if your site sends different HTML to AI crawlers than to other visitors.
</Callout>

### More cards about readability [#more-cards-about-readability]

* **Pages that need JavaScript for their text** lists these pages with their section. Click a page to open its panel. Use the **Page address** filter at the bottom of the screen to narrow the list.
* **What only appears after JavaScript** shows which elements appear only after JavaScript runs: **Links**, **Headings**, **Structured data**, **Canonical** and **Robots meta**. The first row is the **Whole site**.
* **Text repeated across pages** shows sections in which most pages repeat the same text. Give each page its own main text. A section needs at least 5 pages with enough text for a result.

## AI files [#ai-files]

The **AI files** card checks optional files for AI tools. None of these files have an effect on Google rankings.

<Callout type="info">
  `/llms.txt`, `/llms-full.txt` and Markdown copies come from a community proposal, not an official standard, so AI tools may ignore them.
</Callout>

| Item                                 | What VitalSentinel checks                                                                                                                                                           |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **/llms.txt** and **/llms-full.txt** | Whether the file is there, whether its format is valid, and whether its links work. A missing file is not a problem. Add one only if you want to point AI tools to your best pages. |
| **Markdown copies**                  | How many of the checked pages link to a Markdown copy of the page.                                                                                                                  |
| **Article markup**                   | How many of your article pages name an author and have a date. A date that cannot be correct shows as a problem.                                                                    |

The **Status** column shows **Present**, **Not present**, **Has errors** or **Worth a look**. VitalSentinel looks for `/llms.txt` and `/llms-full.txt` once a day.

If Server Log Monitoring is connected, the card also shows how many times bots requested each file in the last 30 days.

## Page panel [#page-panel]

Each page panel in Site Health Monitoring has an **AI & crawlers** tab. The same tab is in the [Page Monitoring](/page-monitoring/watching-pages#page-details-tab) page panel. It shows:

| Section                            | What it shows                                                                                                                                                                                           |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Readability without JavaScript** | The verdict for this page and the share of its main text that is in the HTML                                                                                                                            |
| **robots.txt**                     | Whether robots.txt lets **Our crawler** and **Googlebot** crawl this page                                                                                                                               |
| **AI crawlers**                    | Whether robots.txt lets each AI crawler read this page, and the rule that applies                                                                                                                       |
| **AI and snippet directives**      | The tags that limit how AI tools and search results can use the page, for example `nosnippet`, `noarchive`, `noai` and `noimageai`, and whether a meta robots tag or an `X-Robots-Tag` header sets them |

## Stop alerts for AI bot changes [#stop-alerts-for-ai-bot-changes]

If you block or allow some AI bots on purpose, you can stop issues about their robots.txt changes. See [Skip robots.txt alerts for AI bots](/site-health/settings#skip-robotstxt-alerts-for-ai-bots).

## Related [#related]

<Cards>
  <Card title="Robots.txt Monitoring" href="/features/robots-txt-monitoring">
    robots.txt changes, validation and the AI Bots page
  </Card>

  <Card title="Server Log Monitoring" href="/server-logs/introduction">
    See which AI crawlers really visit your site
  </Card>
</Cards>