Overview
Robots.txt Monitoring reads your robots.txt every hour, explains each rule, marks errors, and records every change with its effect on your pages.
Robots.txt Monitoring watches your robots.txt file. This file tells search engines and AI tools which parts of your site they may visit. One wrong line can hide your whole site from Google, so check here after anyone edits the file.
What Robots.txt Monitoring does
Robots.txt Monitoring needs no setup. It starts after you verify that you own the domain, and it reads your robots.txt every hour. It does these things:
- It keeps a copy of the live file and explains each line in plain words.
- It checks the file against RFC 9309, the official robots.txt standard, and against Google's reading of it. It marks each error or warning on its line.
- It records every change in the last year, and shows which pages each change blocked or allowed.
- It shows which pages and which crawlers your rules keep out.
A crawler is a program that reads your pages, for example Googlebot or an AI bot.
Open the module from Crawling & availability > Robots.txt Monitoring in the sidebar. It has two pages:
| Page | What it shows |
|---|---|
| Overview | The live file, its change history, the pages your rules block and the crawlers they let in. This page. |
| AI bots | Every AI bot, if your file lets it in, and ready-made rules to block or allow it. See AI bots. |
The Overview tabs
The Robots.txt Monitoring Overview has three tabs under the tiles:
| Tab | What it shows |
|---|---|
| Your file | The live file line by line, how your server sends it, and the change history. This page. |
| Blocked pages | The pages your rules keep crawlers away from. See Blocked pages. |
| Crawlers | Which crawlers your rules let in, if bots follow your rules, and the robots.txt of each hostname. See Crawlers. |
The Test a page button at the top right opens a panel. There, you can test pages against your file or against a draft. See Test a page.
If your server refuses our check
If your server refuses the VitalSentinel check of robots.txt, a red banner shows a message such as:
Not measured because your server refuses our check of robots.txt (403 on 100% of them)
VitalSentinel then cannot see your file or tell when it changes.
To fix it, allow VitalSentinel in your firewall or CDN. Use our user agent or our IP list. Click How to allow our checks for the steps, or see Monitoring IPs. Until then, the tiles describe the last copy that VitalSentinel read.
File and Last changed tiles
Two tiles at the top of the Robots.txt Monitoring Overview give the state of the file.
The File tile shows how the file answers and, when it loads, if it is valid:
| Value | Meaning |
|---|---|
| Valid | The file loads and has no syntax error. |
| Invalid | The file has syntax errors. Fix the lines marked Error in Your file. |
| Blocks every crawler | Disallow: / keeps every crawler out of the whole site. |
| Blocks Google | The file keeps Googlebot out of the whole site. |
| Redirects | /robots.txt redirects to another address. |
| Not found | There is no robots.txt (404 or 410). Every crawler may read everything. |
| Error 503 (or another code) | The server returned an error. |
The note under the value gives the size of the file and the number of errors and warnings. Google reads at most 500 KiB of a robots.txt and ignores the rest.
The Last changed tile shows the date of the last change, and how many changes the file had in the last year. Click it to go to Change history.
Your robots.txt
The Your robots.txt card of Robots.txt Monitoring shows the live file, one row for each line. Blank lines are left out. The line numbers are the numbers in your file.
| Column | What it shows |
|---|---|
| Line | The line number. |
| In your file | The line as it is written. |
| What it does | The meaning of the line in a sentence, and chips with facts about the rule. |
If your file does not exist yet, the card shows "No robots.txt yet". VitalSentinel checks the file every hour.
Chips on a line
| Chip | Meaning |
|---|---|
| Error, Warning or Note | A finding of the validation on this line. Hover it for the message and a link to the rule. Fix the lines marked Error first. |
| 12 sitemap URLs | The number of sitemap URLs that this Disallow rule blocks. Click it to open Pages this rule blocks. |
| Crossed by bots | Verified bots still crawled pages under this rule in the last 28 days. Click it to see them in Server Log Monitoring. |
| Same as line 4 | Another line already says the same for the same crawlers. |
Googlebot's last read
A line at the top of the card shows when Googlebot last read your robots.txt, and the status it got. Unverified means that only requests that claim to be Googlebot read it. +3 bots lists other bots that read the file.
This line needs Server Log Monitoring. Without it, the card shows "See when Google reads your file" with Connect Server Log Monitoring.
How the file answers
The How the file answers card of Robots.txt Monitoring shows how your server answered the VitalSentinel checks of robots.txt:
| Tile | What it shows |
|---|---|
| Answered OK | The share of checks that got 200 (the server sent the file) or 404/410 (no file, so nothing is blocked). |
| Longest failing stretch | The longest time the file returned an error or no answer. |
| Typical answer time | How long your server takes to send the file. A slow answer can make crawlers give up. |
| Cache-Control | How long crawlers may keep a copy. Google keeps a copy for up to 24 hours and follows this header. |
| ETag | A stamp of the file's version, so bots can check the file again without a full download. |
If your server sends an error or no answer for robots.txt, Google stops crawling the site for about 12 hours. Then Google uses its last good copy for up to 30 days.
Change history
The Change history card of Robots.txt Monitoring lists every change to your robots.txt in the last year, newest first. Each entry shows:
- The date and time, and the title "robots.txt changed".
- The lines changed, for example "+2 lines", "-1 line" or "~1 line" (changed). "Only whitespace or comments changed" means that no rule moved.
- The effect on your pages, for example "Blocked 12 pages that had 340 Google clicks in the 28 days before". Hover it to see the sitemap pages and the visited pages that the change blocked or allowed.
- First read by 2 bots: the bots that read the new file after the change. This needs server logs.
Turn on Only changes that moved a rule to hide changes to comments and whitespace. Without Google Search Console, the card shows "See what each change cost in Google clicks".
Compare two versions
Click Compare on a change to open "Changes to your robots.txt". It shows the file before and after the change. Green + lines were added. Red - lines were removed. Select Combined or Split for the view.
If one copy of the file is not available, Compare is not active.
Crawl since this change
Click Crawl since this change to see what crawlers did before and after the change. The panel shows Visits before, Visits after and Change (per day with log data). It also has a Status mix table and a Section mix table. For AI companies, it compares crawls and visitors before and after.
A change and a crawl shift close together do not prove that one caused the other. This panel needs Server Log Monitoring.
Add a note to a change
Owners and admins can add a note to a change, for example to record why it happened.
- On the change, click Add note.
- Enter the note. A note can have up to 2,000 characters.
- Click Save.
To change a note, click the edit button next to it, change the text, and click Save. To delete a note, click the delete button, then click Delete in "Delete note?". Members can read the notes but cannot change them.
Issues in Site Health Monitoring
Robots.txt Monitoring does not keep its own list of problems. Problems such as syntax errors and rules that block sitemap pages open as issues in Site Health Monitoring. A banner at the top of the page shows the open robots.txt issues.
Troubleshooting
Is my robots.txt blocking Google?
Look at the File tile. If it shows Blocks Google or Blocks every crawler, your file keeps Googlebot out of the whole site. To check single pages, click Test a page and select Googlebot. See Test a page.
How often does Robots.txt Monitoring check my file?
Every hour. A change shows in Change history after the next check.
Why does the File tile show Not found?
Your server returns 404 or 410 for /robots.txt. Without a file, every crawler may read everything. If you want rules, put a robots.txt file at the root of your site.
Why does Robots.txt Monitoring not see my latest change?
Wait for the next hourly check. If a red banner says that your server refuses our check, allow VitalSentinel in your firewall or CDN. A CDN cache can also serve the old file for a time.
Related
- Blocked pages - the pages your rules keep crawlers away from
- Crawlers - which crawlers your rules let in
- Test a page - test pages and drafts before you publish
- AI bots - allow or block AI crawlers
- Sitemap Monitoring - the sitemap pages that your rules can block
- Server Log Monitoring bots - what each bot really fetched
Files
The Sitemap Monitoring Files tab lists every sitemap file with its status, URL count and Googlebot reads, and checks each file against the official sitemap limits.
Blocked pages
The Robots.txt Monitoring Blocked pages tab lists the visited pages and sitemap pages that your robots.txt rules block, and pages where Google disagrees with your file.
