Robots.txt Monitoring

The robots.txt file tells search engines which pages they can and cannot crawl. VitalSentinel monitors your robots.txt for errors and changes that could impact SEO.

What is robots.txt?

The robots.txt file is located at your domain root (e.g., https://example.com/robots.txt) and contains directives for search engine crawlers:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

Sitemap: https://example.com/sitemap.xml

Why Monitor robots.txt?

Prevent Accidental Blocking

A misconfigured robots.txt can:

• Block search engines from indexing your site
• Prevent crawling of important pages
• Cause pages to drop from search results

VitalSentinel automatically detects when your robots.txt rules conflict with your sitemap, alerting you when URLs you want indexed are being blocked.

Detect Unauthorized Changes

Monitor for:

• Unintended edits
• Malicious modifications
• Deployment errors

Ensure Proper Syntax

Invalid syntax can cause crawlers to:

• Ignore your directives
• Misinterpret blocking rules
• Miss sitemap references

What VitalSentinel Checks

Accessibility

• File exists at the correct location
• Returns 200 status code
• Content-type is correct (text/plain)

Syntax Validation

• Valid User-agent directives
• Proper Allow/Disallow format
• No conflicting rules
• Valid sitemap references

Validation Findings

Validation runs in two tiers: RFC 9309 (the official robots.txt standard) and a Google-specific overlay. Each finding is an error or a warning with a line number and a suggested fix. Common findings include:

Dashboard Overview

VitalSentinel checks your robots.txt every hour. The monitoring page header shows three tiles:

Validation

Valid or Invalid, with a count of errors and warnings. Validation runs against RFC 9309 (the official robots.txt standard) plus Google-specific rules.

Latest check

When VitalSentinel last fetched your robots.txt, with the current file size.

Total versions

The number of distinct file versions recorded in the last year.

Latest robots.txt

A numbered, line-by-line view of your current robots.txt, so you can cross-reference the line numbers in validation findings.

Sitemap Conflicts

VitalSentinel checks whether your robots.txt rules block any URLs from your sitemap. If sitemap URLs are blocked, you'll see:

• Number of blocked URLs
• List of specific URLs being blocked
• The rule causing the block

AI Bot Governance

The AI Bots page shows what AI tools and crawlers are allowed to read from your site, based on the rules in your current robots.txt.

• A per-crawler access matrix covers AI agents such as GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended.
• Each row shows whether that user-agent is allowed or blocked, and which rule applies.
• Use it to confirm you are blocking (or allowing) AI training and answer engines the way you intend.

URL Testers

The Testers page lets you check any URL against your robots.txt rules:

• URL Tester - Enter a URL and see whether it is allowed or blocked, and by which rule.
• Pre-publish Simulator - Paste a draft robots.txt and preview how it would affect your URLs before you publish it live.

Change Timeline

Every robots.txt change is recorded so you can see exactly what changed and when.

• Version history - A timeline of every recorded version, with lines added, removed, and changed for each update.
• Compare changes - Open a side-by-side diff between any two versions.
• URL impact - Each change shows how many of your URLs it started or stopped blocking.
• Annotations - Add a note to any change to record why it happened.

Common Issues

Blocking Important Pages

Problem:

User-agent: *
Disallow: /products/

This blocks all product pages from search engines.

Solution: Remove or modify the disallow rule if products should be indexed.

Blocking CSS/JS

Problem:

User-agent: *
Disallow: /assets/

Blocking CSS/JS prevents search engines from rendering your page properly.

Solution: Allow access to assets needed for rendering.

Wildcard Mistakes

Problem:

Disallow: *private*

Invalid syntax - wildcards work differently in robots.txt.

Solution:

Disallow: /private/

Missing Sitemap

Problem: No sitemap directive in robots.txt.

Solution: Add:

Sitemap: https://example.com/sitemap.xml

Best Practices

Keep It Simple

• Use clear, specific rules
• Avoid complex patterns
• Test changes before deploying

Don't Block Googlebot Specifically

Unless necessary, avoid:

User-agent: Googlebot
Disallow: /

Reference Your Sitemap

Always include:

Sitemap: https://example.com/sitemap.xml

Test Before Deploying

Use the built-in Testers page to check URLs against your current rules, and the Pre-publish Simulator to preview a draft robots.txt before you push it live.

Alerts

Set up alerts for robots.txt issues. Available alert presets:

• Robots.txt file changed - Get notified when the file content changes
• Robots.txt has syntax errors - Alert when syntax errors are detected
• Robots.txt blocks sitemap URLs - Alert when robots.txt rules block URLs from your sitemap

See Setting Up Alerts for configuration details.

Related Monitoring

Use these complementary features for complete SEO visibility:

• Sitemap Scanning - VitalSentinel automatically checks your sitemap URLs against robots.txt rules to detect conflicts
• Indexing Monitoring - See which pages Google has actually indexed
• Google Search Console - View crawl errors and indexing issues reported by Google

Troubleshooting

File Not Found

1. Verify the file exists at your domain root
2. Check file permissions
3. Ensure your server returns 200, not 404

Syntax Errors

1. Review the error message
2. Check the specific line number
3. Validate syntax using online tools

Changes Not Detected

1. Wait for the next check cycle
2. Ensure the file URL is accessible from external networks
3. Check for caching issues

Related Features