Blocked pages
The Robots.txt Monitoring Blocked pages tab lists the visited pages and sitemap pages that your robots.txt rules block, and pages where Google disagrees with your file.
The Blocked pages tab of Robots.txt Monitoring lists the pages that your robots.txt rules keep crawlers away from. A page that you want in Google Search should not be on this list.
Before you start
Open Robots.txt Monitoring in the sidebar and select the Blocked pages tab on the Overview. Each group of pages uses data from another module:
| Group | Needs |
|---|---|
| Visited pages | Google Search Console clicks or Web Analytics page views |
| In your sitemap | Sitemap Monitoring, which starts by itself |
| Waiting for Google, Newly blocked, still crawled, Indexed though blocked | Indexing Monitoring |
Groups of blocked pages
The tiles at the top of the Robots.txt Monitoring Blocked pages tab select a group. Each tile shows the number of pages in the group.
| Tile | What it holds |
|---|---|
| Visited pages | The most-visited pages of the last 28 days that your rules block. |
| In your sitemap | Pages your sitemaps ask Google to index, but your rules keep Google away from. The note says "Blocked, of 5,000 listed". |
| Waiting for Google | Pages your file allows now. Google read them before the change and saw them blocked. Nothing is wrong. Google has not read your fix yet. |
| Newly blocked, still crawled | Google last saw these pages open, and your file blocks them now. Google possibly still crawls and lists them until it reads the new file. |
| Indexed though blocked | Google lists these pages although robots.txt blocks them. |
The last three tiles come from Google's URL Inspection of a rotating sample of your pages. The ? of each tile shows how many pages Google inspected.
Indexed though blocked
A robots.txt block does not remove a page from Google. Google also cannot read a noindex tag on a blocked page. To remove such a page from Google, unblock it so that Google sees its noindex. If you accept the listing, keep the block.
Visited pages
The Visited pages list of Robots.txt Monitoring shows the pages that got visits in the last 28 days and that your live file blocks. Visits come from Google Search Console clicks and Web Analytics page views.
| Column | What it shows |
|---|---|
| Page | The address. |
| Google clicks | Clicks from Google Search in the last 28 days. |
| Page views | Page views in the last 28 days, from Web Analytics. |
| Blocked by | The rule that blocks the page, with a link to its line. |
| Google saw | Google's robots.txt verdict at its last crawl of the page, from Indexing Monitoring. |
| Googlebot last visit | The last day a verified Googlebot requested the page, from Server Log Monitoring. |
Turn on Only pages with Google clicks to see only the blocked pages that cost you search traffic. If no visited page is blocked, the list shows "No visited page is blocked".
A visit from Googlebot after your rule appeared means that Google did not read the new file yet. If Indexing Monitoring or Server Log Monitoring is not set up, a line above the table offers to connect it.
Sitemap pages
The In your sitemap list of Robots.txt Monitoring shows the sitemap pages that your rules block. The columns are Page, Blocked by and Crawler. Crawler is Googlebot, or all other crawlers when the rule is in the section for every crawler without its own section.
If no sitemap page is blocked, the list shows "No sitemap page is blocked". If Sitemap Monitoring did not read a sitemap yet, the list shows "No sitemap read yet".
When the sitemap numbers are out of date
VitalSentinel compares your sitemap with your robots.txt. If the comparison is old, a note says "These blocked-URL numbers are out of date" and tells you why. For example, your sitemap now lists more pages than the comparison used. Owners and admins can click Compare again. The toast says "Comparison started. The numbers update in a minute or two."
What Google last saw
The Waiting for Google, Newly blocked, still crawled and Indexed though blocked lists of Robots.txt Monitoring compare Google's view with your live file:
| Column | What it shows |
|---|---|
| Page | The address. |
| Google saw | Google's robots.txt verdict at its last crawl, from URL Inspection. |
| Google last read it | When Google last crawled the page. Compare it with the date your file changed. |
| Your file now | What the live file says to Googlebot. A blocked page names the rule and links to its line. |
If Google did not inspect a page yet, the list shows "Google has not inspected any page yet". Set up Indexing Monitoring to fill these lists.
Pages this rule blocks
Click a rule, or a chip such as 12 sitemap URLs in Your file, to open Pages this rule blocks. The header shows the line and the rule, for example "Line 4: Disallow: /search/". Select Visited pages or In your sitemap to see the pages that the rule blocks.
Troubleshooting
What do I do if a page I want in Google is blocked?
Find the rule in the Blocked by column and click it to go to its line. Change or remove the rule in your robots.txt. Before you publish the change, test it in Test a page.
Why is a page still blocked in Google after I fixed my file?
Google keeps a copy of robots.txt for up to 24 hours, and crawls each page on its own schedule. The page shows in Waiting for Google until Google reads it again.
Why are the Visited pages empty?
The list needs Google Search Console clicks or Web Analytics page views. If neither is connected, connect one of them. If no page got visits in the last 28 days, the list shows "No visits to check yet".
Related
- Overview - the live file and its change history
- Test a page - test a fix before you publish it
- Indexing Monitoring - what Google saw at its last crawl
- Sitemap Monitoring - the pages your sitemaps list
- Crawl budget - crawls that go to blocked URLs
Overview
Robots.txt Monitoring reads your robots.txt every hour, explains each rule, marks errors, and records every change with its effect on your pages.
Crawlers
The Robots.txt Monitoring Crawlers tab shows which search engines and AI crawlers your rules let in, which bots follow your rules, and the robots.txt of each hostname.
