# Robots.txt Monitoring



The robots.txt file tells search engines which pages they can and cannot crawl. VitalSentinel monitors your robots.txt for errors and changes that could impact SEO.

## What is robots.txt? [#what-is-robotstxt]

The robots.txt file is located at your domain root (e.g., `https://example.com/robots.txt`) and contains directives for search engine crawlers:

```txt
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml
```

## Why Monitor robots.txt? [#why-monitor-robotstxt]

### Prevent Accidental Blocking [#prevent-accidental-blocking]

A misconfigured robots.txt can:

* Block search engines from indexing your site
* Prevent crawling of important pages
* Cause pages to drop from search results

VitalSentinel automatically detects when your robots.txt rules conflict with your sitemap, alerting you when URLs you want indexed are being blocked.

### Detect Unauthorized Changes [#detect-unauthorized-changes]

Monitor for:

* Unintended edits
* Malicious modifications
* Deployment errors

### Ensure Proper Syntax [#ensure-proper-syntax]

Invalid syntax can cause crawlers to:

* Ignore your directives
* Misinterpret blocking rules
* Miss sitemap references

## What VitalSentinel Checks [#what-vitalsentinel-checks]

### Accessibility [#accessibility]

* File exists at the correct location
* Returns 200 status code
* Content-type is correct (text/plain)

### Syntax Validation [#syntax-validation]

* Valid User-agent directives
* Proper Allow/Disallow format
* No conflicting rules
* Valid sitemap references

### Validation Findings [#validation-findings]

Validation runs in two tiers: RFC 9309 (the official robots.txt standard) and a Google-specific overlay. Each finding is an error or a warning with a line number and a suggested fix. Common findings include:

| Finding             | Description                      |
| ------------------- | -------------------------------- |
| Missing file        | No robots.txt found              |
| Invalid syntax      | Malformed directives             |
| Blocking all        | `Disallow: /` blocks entire site |
| Invalid sitemap URL | Sitemap reference is broken      |
| Encoding issues     | Non-UTF8 characters              |

## Dashboard Overview [#dashboard-overview]

VitalSentinel checks your robots.txt every hour. The monitoring page header shows three tiles:

### Validation [#validation]

**Valid** or **Invalid**, with a count of errors and warnings. Validation runs against RFC 9309 (the official robots.txt standard) plus Google-specific rules.

### Latest check [#latest-check]

When VitalSentinel last fetched your robots.txt, with the current file size.

### Total versions [#total-versions]

The number of distinct file versions recorded in the last year.

### Latest robots.txt [#latest-robotstxt]

A numbered, line-by-line view of your current robots.txt, so you can cross-reference the line numbers in validation findings.

### Sitemap Conflicts [#sitemap-conflicts]

VitalSentinel checks whether your robots.txt rules block any URLs from your sitemap. If sitemap URLs are blocked, you'll see:

* Number of blocked URLs
* List of specific URLs being blocked
* The rule causing the block

<Callout type="warning">
  URLs in your sitemap should generally be accessible to crawlers. Blocking sitemap URLs may prevent search engines from indexing those pages.
</Callout>

## AI Bot Governance [#ai-bot-governance]

The **AI Bots** page shows what AI tools and crawlers are allowed to read from your site, based on the rules in your current robots.txt.

* A per-crawler access matrix covers AI agents such as **GPTBot** (OpenAI), **ClaudeBot** (Anthropic), **PerplexityBot**, and **Google-Extended**.
* Each row shows whether that user-agent is allowed or blocked, and which rule applies.
* Use it to confirm you are blocking (or allowing) AI training and answer engines the way you intend.

## URL Testers [#url-testers]

The **Testers** page lets you check any URL against your robots.txt rules:

* **URL Tester** - Enter a URL and see whether it is allowed or blocked, and by which rule.
* **Pre-publish Simulator** - Paste a draft robots.txt and preview how it would affect your URLs before you publish it live.

## Change Timeline [#change-timeline]

Every robots.txt change is recorded so you can see exactly what changed and when.

* **Version history** - A timeline of every recorded version, with lines added, removed, and changed for each update.
* **Compare changes** - Open a side-by-side diff between any two versions.
* **URL impact** - Each change shows how many of your URLs it started or stopped blocking.
* **Annotations** - Add a note to any change to record why it happened.

## Common Issues [#common-issues]

### Blocking Important Pages [#blocking-important-pages]

**Problem:**

```txt
User-agent: *
Disallow: /products/
```

This blocks all product pages from search engines.

**Solution:**
Remove or modify the disallow rule if products should be indexed.

### Blocking CSS/JS [#blocking-cssjs]

**Problem:**

```txt
User-agent: *
Disallow: /assets/
```

Blocking CSS/JS prevents search engines from rendering your page properly.

**Solution:**
Allow access to assets needed for rendering.

### Wildcard Mistakes [#wildcard-mistakes]

**Problem:**

```txt
Disallow: *private*
```

Invalid syntax - wildcards work differently in robots.txt.

**Solution:**

```txt
Disallow: /private/
```

### Missing Sitemap [#missing-sitemap]

**Problem:**
No sitemap directive in robots.txt.

**Solution:**
Add:

```txt
Sitemap: https://example.com/sitemap.xml
```

## Best Practices [#best-practices]

### Keep It Simple [#keep-it-simple]

* Use clear, specific rules
* Avoid complex patterns
* Test changes before deploying

### Don't Block Googlebot Specifically [#dont-block-googlebot-specifically]

Unless necessary, avoid:

```txt
User-agent: Googlebot
Disallow: /
```

### Reference Your Sitemap [#reference-your-sitemap]

Always include:

```txt
Sitemap: https://example.com/sitemap.xml
```

### Test Before Deploying [#test-before-deploying]

Use the built-in **Testers** page to check URLs against your current rules, and the Pre-publish Simulator to preview a draft robots.txt before you push it live.

## Alerts [#alerts]

Set up alerts for robots.txt issues. Available alert presets:

* **Robots.txt file changed** - Get notified when the file content changes
* **Robots.txt has syntax errors** - Alert when syntax errors are detected
* **Robots.txt blocks sitemap URLs** - Alert when robots.txt rules block URLs from your sitemap

See [Setting Up Alerts](/alerts/setting-up-alerts) for configuration details.

## Related Monitoring [#related-monitoring]

Use these complementary features for complete SEO visibility:

* **Sitemap Scanning** - VitalSentinel automatically checks your sitemap URLs against robots.txt rules to detect conflicts
* **Indexing Monitoring** - See which pages Google has actually indexed
* **Google Search Console** - View crawl errors and indexing issues reported by Google

## Troubleshooting [#troubleshooting]

### File Not Found [#file-not-found]

1. Verify the file exists at your domain root
2. Check file permissions
3. Ensure your server returns 200, not 404

### Syntax Errors [#syntax-errors]

1. Review the error message
2. Check the specific line number
3. Validate syntax using online tools

### Changes Not Detected [#changes-not-detected]

1. Wait for the next check cycle
2. Ensure the file URL is accessible from external networks
3. Check for caching issues

## Related Features [#related-features]

<Cards>
  <Card title="Indexing Monitoring" href="/features/indexing-monitoring">
    Track which pages are indexed
  </Card>

  <Card title="Google Search Console" href="/features/google-search-console">
    See Google's view of your site
  </Card>

  <Card title="Setting Up Alerts" href="/alerts/setting-up-alerts">
    Get notified of robots.txt issues
  </Card>
</Cards>