# Test a page

The **Test a page** panel of Robots.txt Monitoring tests pages against your robots.txt rules. You can test against the live file, or write a draft and see what it changes before you publish it. VitalSentinel never saves your draft and never changes your robots.txt.

## Before you start [#before-you-start]

Open **Robots.txt Monitoring** in the sidebar and click **Test a page** at the top right of the **Overview**. Every user can use the panel.

If your site has no robots.txt yet, the panel says "No robots.txt is stored for this site yet". Without a file, every crawler may read everything. You can write a draft to check it before you publish it.

## Test pages against your live file [#test-pages-against-your-live-file]

1. In **Pages (one address per line)**, enter one or more addresses. You can enter a full address or a path, for example `/path/to/page`.
2. In **Crawlers to test as**, select one or more crawlers: **Googlebot**, **Bingbot**, **GPTBot**, **ClaudeBot** or **Other**.
3. If you selected **Other**, enter the crawler name.
4. Click **Test page** (or **Test 5 pages** for more addresses).

You can test up to 500 addresses at a time. The results show "Tested against your live file."

### Test results [#test-results]

| Column | What it shows |
|--------|---------------|
| **Page** | The address that you tested. |
| **Crawler** | The crawler that the row is for. |
| **Result** | **Allowed** or **Blocked**. |
| **Why** | The rule that decides, with its line, for example "Line 4: Disallow: /search/". |

An address that is not valid shows "Not a valid address, so it was not tested."

## Check a draft robots.txt [#check-a-draft-robotstxt]

Use a draft to test a change before you publish it.

1. In the **Test pages** card, select **Test against my draft**.
2. In **Draft robots.txt**, change the text. The draft starts from your live file.
3. Click **Check my draft**.

The **Your draft** card then shows a verdict, for example "Your draft is valid and blocks no crawler that is let in today". It lists the mistakes in your draft with **Line**, **Kind** and **Problem**. It also lists the crawlers whose access changes, with **Now** and **With this draft**.

### Quick rules [#quick-rules]

The **Your draft** card has buttons that add common rules to your draft:

- **Block AI training bots**: adds rules for the bots that copy pages to train AI models.
- **Block all AI bots**: adds rules for every AI bot that VitalSentinel tracks.
- **Block this folder**: enter a folder, for example `/private/`, then click the button.

<Callout type="warn">
If the verdict says "Your draft blocks Googlebot from the whole site", do not publish the draft. Google then cannot crawl any page of your site.
</Callout>

## What this changes [#what-this-changes]

After you check a draft, the **What this changes** card of Robots.txt Monitoring compares your draft with your live file. For Googlebot, it uses the pages in your sitemaps and the pages that got Google clicks or visits in the last 28 days. For AI crawlers, it uses the visited pages.

| Tile | What it shows |
|------|---------------|
| **Newly blocked** | Pages your draft blocks, with the Google clicks and page views at risk. |
| **Newly allowed** | Pages your draft opens up. |
| **AI bots affected** | Crawlers whose access changes. |

The tables show each rule with its **Pages**, **Google clicks** and **Page views**. Click a rule to open **Pages this rule changes** and see the pages. The AI bots table shows **Crawler**, **Now**, **With this draft** and **Visited pages that change**.

If no sitemap was read and neither [Search Console](/google-search-console/overview) nor [Web Analytics](/web-analytics/overview) reported a page, the card shows "Not measured".

## Publish your change [#publish-your-change]

VitalSentinel does not publish your draft. When the draft is correct, copy it and put it on your server as `/robots.txt`. Robots.txt Monitoring reads the new file at the next hourly check and records the change in **Change history**.

## Troubleshooting [#troubleshooting]

### How do I test a URL against my robots.txt? [#how-do-i-test-a-url-against-my-robotstxt]

Click **Test a page**, enter the address, select the crawler and click **Test page**. The **Result** column shows **Allowed** or **Blocked**, and **Why** names the rule.

### Why can I not select Test against my draft? [#why-can-i-not-select-test-against-my-draft]

The draft starts from your live file. Wait until the live file loads. If the panel says "We couldn't read your live robots.txt", click **Try again**.

### Does Test a page change my robots.txt? [#does-test-a-page-change-my-robotstxt]

No. The draft is never saved and VitalSentinel never changes your file. You put the new file on your server yourself.

## Related [#related]

- [Overview](/robots-txt-monitoring/overview) - the live file and its change history
- [Blocked pages](/robots-txt-monitoring/blocked-pages) - the pages your live rules block
- [AI bots](/robots-txt-monitoring/ai-bots) - get ready-made rules for AI bots and test them here
- [Crawlers](/robots-txt-monitoring/crawlers) - which bots follow your rules
- [Sitemap Monitoring](/sitemap-monitoring/overview) - the sitemap pages that the test uses