# AI bots

The **AI bots** page of Robots.txt Monitoring lists every AI bot that VitalSentinel tracks. It shows if your robots.txt lets each bot in, and how many visitors each AI company sent you. Use it to decide which AI tools may read your site.

## Before you start [#before-you-start]

Open **Robots.txt Monitoring** > **AI bots** in the sidebar. The page reads your live robots.txt. If VitalSentinel did not read your file yet, the page shows "No robots.txt read yet".

The **Visitors sent** column needs [Web Analytics](/web-analytics/overview). How often each bot really visits is in [Server Log Monitoring](/server-logs/bots).

## AI bot purposes [#ai-bot-purposes]

The tiles at the top of the **Every AI bot** card of Robots.txt Monitoring split the bots by purpose. Click a tile to show only those bots.

| Tile | What the bots do | What blocking costs |
|------|------------------|---------------------|
| **All bots** | Every AI bot that VitalSentinel tracks. | - |
| **Training** | Copy pages to train AI models, such as ChatGPT or Claude. | No traffic. It only keeps your content out of future models. |
| **AI search** | Read pages so AI tools can find and cite them in answers. | Citations and visits from AI answers. |
| **On request** | Read one page when a person asks an AI tool about its link, or has it act for them. | People who ask an AI about your page cannot get it. |

Each purpose tile shows the number of bots and a chip: **Allowed**, **Blocked** or **Partly blocked**. **Partly blocked** also shows how many bots may read your site, for example "3 of 5 may read your site". Hover the **?** of a tile to see the rule that decides for each bot.

Blocking training bots is a choice and shows as neutral. Blocking AI search or on-request bots shows as a warning, because it can cost visits.

## The AI bots table [#the-ai-bots-table]

Each row of the Robots.txt Monitoring **AI bots** table is one bot. Bots are sorted by purpose, then by name.

| Column | What it shows |
|--------|---------------|
| **Bot** | The bot name and its company, for example GPTBot (OpenAI). |
| **Purpose** | **Training**, **AI search** or **On request**. |
| **Your robots.txt** | **Allowed** or **Blocked**: if your file lets the bot read your homepage. |
| **Visitors sent** | Visitors who came from the company's AI product in the last 28 days, from Web Analytics. |

Hover **Your robots.txt** to see the rule that decides. It also shows if the bot has **Its own section** or uses **The catch-all (\*) rules**. If no rule covers the bot, the hover says "No rule covers it, so it is allowed".

### Visitors sent [#visitors-sent]

**Visitors sent** counts a visit only when the visit names the AI product. Many AI assistants send no referrer, so the real number is higher. The figure is for the whole company. It shows once, on the first bot of that company in the list. If Web Analytics has no data, the column is left out.

### Row menu [#row-menu]

The menu on each row opens the bot in **Server Log Monitoring**. If the bot visited in the last 28 days, it opens every page that the bot fetched. Otherwise, it opens the list of AI crawlers.

Some bots, for example Google-Extended and Applebot-Extended, are robots.txt settings only. They never visit your site. They control if the company's other crawlers may use your pages for AI.

## Get rules to block or allow AI bots [#get-rules-to-block-or-allow-ai-bots]

VitalSentinel can write the robots.txt lines that block or allow all bots of one purpose. It never changes your robots.txt. You copy the lines into your file yourself.

<Callout type="warn">
If the dialog shows **Remove an Allow line first**, a bot already has an Allow rule in its own section. Remove that rule from your file. An Allow rule wins over the added Disallow, so the bot stays in.
</Callout>

1. In the **Every AI bot** card, click a purpose tile, for example **Training**.
2. Click **Get the rules to block…**. If all bots of the purpose are already blocked, the button is **Get the rules to allow…**.
3. In the dialog, check the marked lines. They are added to a copy of your live file.
4. Click **Copy the whole file**. You can also select and copy only the marked lines.
5. Paste the lines into your robots.txt on your server.

The dialog title names the purpose, for example "Rules to block training bots". It says "We never change your robots.txt. Paste the marked lines into it yourself."

### Test the rules before you publish [#test-the-rules-before-you-publish]

In the dialog, click **Test it**. The [Test a page](/robots-txt-monitoring/test-a-page) panel opens with the new file as your draft. Check the draft there before you publish it. The draft stays in your browser tab only.

## AI policy lines [#ai-policy-lines]

Some robots.txt files have AI policy lines. These lines ask AI tools to allow or not allow uses such as **Search results**, **AI answers** or **AI training**. They are requests and block nothing. Hover the **?** of the **Every AI bot** card to see the policy lines of your file.

If a policy line and your crawler rules disagree, the card shows "Your AI policy line and your rules disagree" with the details. Change the line or the rules so that they say the same thing.

### After your last robots.txt change [#after-your-last-robotstxt-change]

The **?** of the **Every AI bot** card can also show what each AI company did after your latest robots.txt change. It compares the crawls and the visitors per day before and after the change.

## Troubleshooting [#troubleshooting]

### How do I allow or block AI crawlers? [#how-do-i-allow-or-block-ai-crawlers]

Click the purpose tile, for example **Training**, then click **Get the rules to block…**. Copy the marked lines into your robots.txt. Test the result first with **Test it**.

### Why does an AI bot still visit after I blocked it? [#why-does-an-ai-bot-still-visit-after-i-blocked-it]

robots.txt is a request. Some bots do not follow it. Check the bot in **Do bots follow your rules?** on the [Crawlers](/robots-txt-monitoring/crawlers) tab. If it still visits blocked pages, block it in your firewall or CDN.

### Why is the Visitors sent column missing? [#why-is-the-visitors-sent-column-missing]

The column needs Web Analytics data. If Web Analytics is not installed, or recorded no visits from AI products, the column is left out.

## Related [#related]

- [Test a page](/robots-txt-monitoring/test-a-page) - test the AI rules before you publish them
- [Crawlers](/robots-txt-monitoring/crawlers) - which bots follow your rules
- [Server Log Monitoring bots](/server-logs/bots) - how often each AI bot really visits
- [Overview](/robots-txt-monitoring/overview) - the live file and its change history
- [AI readability](/site-health/ai-readability) - if AI crawlers can get in and read your pages
- [Glossary](/help/glossary#bot-categories) - the AI bot categories