AI bots
The Robots.txt Monitoring AI bots page lists every AI bot by purpose, shows if your robots.txt lets it in, and gives rules to block or allow it.
The AI bots page of Robots.txt Monitoring lists every AI bot that VitalSentinel tracks. It shows if your robots.txt lets each bot in, and how many visitors each AI company sent you. Use it to decide which AI tools may read your site.
Before you start
Open Robots.txt Monitoring > AI bots in the sidebar. The page reads your live robots.txt. If VitalSentinel did not read your file yet, the page shows "No robots.txt read yet".
The Visitors sent column needs Web Analytics. How often each bot really visits is in Server Log Monitoring.
AI bot purposes
The tiles at the top of the Every AI bot card of Robots.txt Monitoring split the bots by purpose. Click a tile to show only those bots.
| Tile | What the bots do | What blocking costs |
|---|---|---|
| All bots | Every AI bot that VitalSentinel tracks. | - |
| Training | Copy pages to train AI models, such as ChatGPT or Claude. | No traffic. It only keeps your content out of future models. |
| AI search | Read pages so AI tools can find and cite them in answers. | Citations and visits from AI answers. |
| On request | Read one page when a person asks an AI tool about its link, or has it act for them. | People who ask an AI about your page cannot get it. |
Each purpose tile shows the number of bots and a chip: Allowed, Blocked or Partly blocked. Partly blocked also shows how many bots may read your site, for example "3 of 5 may read your site". Hover the ? of a tile to see the rule that decides for each bot.
Blocking training bots is a choice and shows as neutral. Blocking AI search or on-request bots shows as a warning, because it can cost visits.
The AI bots table
Each row of the Robots.txt Monitoring AI bots table is one bot. Bots are sorted by purpose, then by name.
| Column | What it shows |
|---|---|
| Bot | The bot name and its company, for example GPTBot (OpenAI). |
| Purpose | Training, AI search or On request. |
| Your robots.txt | Allowed or Blocked: if your file lets the bot read your homepage. |
| Visitors sent | Visitors who came from the company's AI product in the last 28 days, from Web Analytics. |
Hover Your robots.txt to see the rule that decides. It also shows if the bot has Its own section or uses The catch-all (*) rules. If no rule covers the bot, the hover says "No rule covers it, so it is allowed".
Visitors sent
Visitors sent counts a visit only when the visit names the AI product. Many AI assistants send no referrer, so the real number is higher. The figure is for the whole company. It shows once, on the first bot of that company in the list. If Web Analytics has no data, the column is left out.
Row menu
The menu on each row opens the bot in Server Log Monitoring. If the bot visited in the last 28 days, it opens every page that the bot fetched. Otherwise, it opens the list of AI crawlers.
Some bots, for example Google-Extended and Applebot-Extended, are robots.txt settings only. They never visit your site. They control if the company's other crawlers may use your pages for AI.
Get rules to block or allow AI bots
VitalSentinel can write the robots.txt lines that block or allow all bots of one purpose. It never changes your robots.txt. You copy the lines into your file yourself.
If the dialog shows Remove an Allow line first, a bot already has an Allow rule in its own section. Remove that rule from your file. An Allow rule wins over the added Disallow, so the bot stays in.
- In the Every AI bot card, click a purpose tile, for example Training.
- Click Get the rules to block…. If all bots of the purpose are already blocked, the button is Get the rules to allow….
- In the dialog, check the marked lines. They are added to a copy of your live file.
- Click Copy the whole file. You can also select and copy only the marked lines.
- Paste the lines into your robots.txt on your server.
The dialog title names the purpose, for example "Rules to block training bots". It says "We never change your robots.txt. Paste the marked lines into it yourself."
Test the rules before you publish
In the dialog, click Test it. The Test a page panel opens with the new file as your draft. Check the draft there before you publish it. The draft stays in your browser tab only.
AI policy lines
Some robots.txt files have AI policy lines. These lines ask AI tools to allow or not allow uses such as Search results, AI answers or AI training. They are requests and block nothing. Hover the ? of the Every AI bot card to see the policy lines of your file.
If a policy line and your crawler rules disagree, the card shows "Your AI policy line and your rules disagree" with the details. Change the line or the rules so that they say the same thing.
After your last robots.txt change
The ? of the Every AI bot card can also show what each AI company did after your latest robots.txt change. It compares the crawls and the visitors per day before and after the change.
Troubleshooting
How do I allow or block AI crawlers?
Click the purpose tile, for example Training, then click Get the rules to block…. Copy the marked lines into your robots.txt. Test the result first with Test it.
Why does an AI bot still visit after I blocked it?
robots.txt is a request. Some bots do not follow it. Check the bot in Do bots follow your rules? on the Crawlers tab. If it still visits blocked pages, block it in your firewall or CDN.
Why is the Visitors sent column missing?
The column needs Web Analytics data. If Web Analytics is not installed, or recorded no visits from AI products, the column is left out.
Related
- Test a page - test the AI rules before you publish them
- Crawlers - which bots follow your rules
- Server Log Monitoring bots - how often each AI bot really visits
- Overview - the live file and its change history
- AI readability - if AI crawlers can get in and read your pages
- Glossary - the AI bot categories
Test a page
Use Test a page in Robots.txt Monitoring to test pages against your live robots.txt or a draft, and see what a draft changes before you publish it.
Overview
Uptime Monitoring checks all day that your site answers, records every outage and its cost, tracks response times, and watches your SSL certificate and domain.
