Crawlers
The Robots.txt Monitoring Crawlers tab shows which search engines and AI crawlers your rules let in, which bots follow your rules, and the robots.txt of each hostname.
The Crawlers tab of Robots.txt Monitoring shows which crawlers your rules allow or block. With Server Log Monitoring, it also shows which bots follow your rules. Check that no search engine that you need is blocked.
Before you start
Open Robots.txt Monitoring in the sidebar and select the Crawlers tab on the Overview. Who can get in works without other modules. Do bots follow your rules? and the Googlebot visits of Hostnames need Server Log Monitoring.
Who can get in
The Who can get in card of Robots.txt Monitoring shows which crawlers your rules let read your homepage. A crawler is a program that reads your pages for a search engine or an AI tool.
The first line gives the answer, for example "3 of 30 crawlers we check are kept out". Then one tile for each group shows how many crawlers of the group may read the page:
- Search engines: for example Googlebot, Bingbot and DuckDuckBot.
- AI crawlers: for example GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
- Regional engines: for example YandexBot, Baiduspider and SeznamBot.
- Everyone else: Other crawlers (*), the default rules for any crawler without its own section.
Under the tiles, Kept out lists every crawler that your rules block. More open than the catch-all lists crawlers whose own section lets them into pages that the * rules keep others out of. Hover a crawler to see what it does and the rule that decides.
If your site has no robots.txt, the card shows "No robots.txt to read yet". Without a file, every crawler may read everything.
Do bots follow your rules?
The Do bots follow your rules? card of Robots.txt Monitoring shows how closely each bot followed your robots.txt rules. It needs Server Log Monitoring. Without it, the card shows "See which bots really follow your rules" with Connect Server Log Monitoring.
| Column | What it shows |
|---|---|
| Bot | The bot name. |
| Category | The kind of bot, for example a search engine or an AI crawler. |
| Verified visits | Visits from bots that VitalSentinel confirmed by their address, not only by the name they give. |
| On blocked pages | Verified visits to pages that your User-agent: * rules block. |
| Followed your rules | The share of visits that stayed off the blocked pages. |
Only verified bots count, and only the * rules are checked. A bot that ignores a rule needs a firewall rule instead. If no bot visited a blocked page, the card shows "No bot visited a blocked page".
If the logs of the period are incomplete, the rates are not shown.
Hostnames
The Hostnames card of Robots.txt Monitoring lists the hostnames of your site, for example www.example.com and shop.example.com. Each hostname has its own robots.txt. The list shows the hostnames that Google or VitalSentinel reached.
| Column | What it shows |
|---|---|
| Host | The hostname. |
| Answers | How its /robots.txt answered at the last look: 200 OK, 404 Not found, 403 Refused, a server error, a redirect or No answer. |
| Rules for Googlebot | If the hostname serves the same robots.txt as your main one. Differs from www.example.com means that the rules are different. |
| Googlebot visits | Requests from the real Googlebot to this hostname in the last 28 days. Needs Server Log Monitoring. |
A hostname marked Rules differ gets crawls from Google, but its robots.txt differs from your main one or is missing. Not checked yet means that Page Monitoring did not read the hostname's robots.txt yet. It does when it crawls a page on that hostname.
Troubleshooting
Is my robots.txt blocking a search engine?
Look at the Search engines tile in Who can get in. If it is less than all, find the crawler under Kept out and hover it for the rule. Remove or change that rule in your file.
A bot ignores my robots.txt. What do I do?
robots.txt is a request, not a lock. If a verified bot still visits blocked pages, block it in your firewall or CDN. To see every page that a bot fetched, open Bots in Server Log Monitoring.
Why does a subdomain show Rules differ?
The subdomain serves its own robots.txt, and it is different from your main file. If Google crawls the subdomain, make sure that its rules are what you want.
Related
- Overview - the live file and its change history
- AI bots - allow or block AI crawlers by purpose
- Server Log Monitoring bots - what each bot really fetched
- AI readability - if AI crawlers can get in and read your pages
- Glossary - verified, unverified and spoofed bots
Blocked pages
The Robots.txt Monitoring Blocked pages tab lists the visited pages and sitemap pages that your robots.txt rules block, and pages where Google disagrees with your file.
Test a page
Use Test a page in Robots.txt Monitoring to test pages against your live robots.txt or a draft, and see what a draft changes before you publish it.
