Test a page
Use Test a page in Robots.txt Monitoring to test pages against your live robots.txt or a draft, and see what a draft changes before you publish it.
The Test a page panel of Robots.txt Monitoring tests pages against your robots.txt rules. You can test against the live file, or write a draft and see what it changes before you publish it. VitalSentinel never saves your draft and never changes your robots.txt.
Before you start
Open Robots.txt Monitoring in the sidebar and click Test a page at the top right of the Overview. Every user can use the panel.
If your site has no robots.txt yet, the panel says "No robots.txt is stored for this site yet". Without a file, every crawler may read everything. You can write a draft to check it before you publish it.
Test pages against your live file
- In Pages (one address per line), enter one or more addresses. You can enter a full address or a path, for example
/path/to/page. - In Crawlers to test as, select one or more crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot or Other.
- If you selected Other, enter the crawler name.
- Click Test page (or Test 5 pages for more addresses).
You can test up to 500 addresses at a time. The results show "Tested against your live file."
Test results
| Column | What it shows |
|---|---|
| Page | The address that you tested. |
| Crawler | The crawler that the row is for. |
| Result | Allowed or Blocked. |
| Why | The rule that decides, with its line, for example "Line 4: Disallow: /search/". |
An address that is not valid shows "Not a valid address, so it was not tested."
Check a draft robots.txt
Use a draft to test a change before you publish it.
- In the Test pages card, select Test against my draft.
- In Draft robots.txt, change the text. The draft starts from your live file.
- Click Check my draft.
The Your draft card then shows a verdict, for example "Your draft is valid and blocks no crawler that is let in today". It lists the mistakes in your draft with Line, Kind and Problem. It also lists the crawlers whose access changes, with Now and With this draft.
Quick rules
The Your draft card has buttons that add common rules to your draft:
- Block AI training bots: adds rules for the bots that copy pages to train AI models.
- Block all AI bots: adds rules for every AI bot that VitalSentinel tracks.
- Block this folder: enter a folder, for example
/private/, then click the button.
If the verdict says "Your draft blocks Googlebot from the whole site", do not publish the draft. Google then cannot crawl any page of your site.
What this changes
After you check a draft, the What this changes card of Robots.txt Monitoring compares your draft with your live file. For Googlebot, it uses the pages in your sitemaps and the pages that got Google clicks or visits in the last 28 days. For AI crawlers, it uses the visited pages.
| Tile | What it shows |
|---|---|
| Newly blocked | Pages your draft blocks, with the Google clicks and page views at risk. |
| Newly allowed | Pages your draft opens up. |
| AI bots affected | Crawlers whose access changes. |
The tables show each rule with its Pages, Google clicks and Page views. Click a rule to open Pages this rule changes and see the pages. The AI bots table shows Crawler, Now, With this draft and Visited pages that change.
If no sitemap was read and neither Search Console nor Web Analytics reported a page, the card shows "Not measured".
Publish your change
VitalSentinel does not publish your draft. When the draft is correct, copy it and put it on your server as /robots.txt. Robots.txt Monitoring reads the new file at the next hourly check and records the change in Change history.
Troubleshooting
How do I test a URL against my robots.txt?
Click Test a page, enter the address, select the crawler and click Test page. The Result column shows Allowed or Blocked, and Why names the rule.
Why can I not select Test against my draft?
The draft starts from your live file. Wait until the live file loads. If the panel says "We couldn't read your live robots.txt", click Try again.
Does Test a page change my robots.txt?
No. The draft is never saved and VitalSentinel never changes your file. You put the new file on your server yourself.
Related
- Overview - the live file and its change history
- Blocked pages - the pages your live rules block
- AI bots - get ready-made rules for AI bots and test them here
- Crawlers - which bots follow your rules
- Sitemap Monitoring - the sitemap pages that the test uses
Crawlers
The Robots.txt Monitoring Crawlers tab shows which search engines and AI crawlers your rules let in, which bots follow your rules, and the robots.txt of each hostname.
AI bots
The Robots.txt Monitoring AI bots page lists every AI bot by purpose, shows if your robots.txt lets it in, and gives rules to block or allow it.
