Indexing Monitoring

Indexing monitoring helps you understand which of your pages are in Google's index and why some pages might not appear in search results.

What is Indexing?

When Google crawls your website, it adds pages to its index. Only indexed pages can appear in search results. Common indexing issues include:

• Pages not discovered by Google
• Pages discovered but not indexed
• Pages blocked by robots.txt
• Pages with noindex directives

How It Works

VitalSentinel integrates with Google Search Console's URL Inspection API to provide:

1. URL inspection data - Status of individual URLs
2. Coverage reports - Overview of indexing across your site
3. Issue tracking - Problems affecting indexing
4. Mobile usability - Mobile-friendliness issues
5. Rich results - Structured data detection and validation
6. AMP status - Accelerated Mobile Pages detection (if applicable)

Properties & Coverage

Indexing monitoring works across one or more Google Search Console properties per domain. Manage them from the Properties tab.

Choosing properties

• Add the properties you want checked from the Search Console properties that match this domain.
• Route URL paths into groups so each property covers the right part of your site.
• Each property is inspected at up to roughly 2,000 URLs per day. The Properties page shows each property's daily quota, today's progress, and the estimated time for a full re-check cycle.

Priority URLs

Pin the pages that matter most as Priority URLs. These are re-checked every day before anything else, so a regression on a key page surfaces quickly.

Plan limits

The number of Search Console properties you can monitor for indexing depends on your plan:

On Free and Starter, only whole-site properties can be monitored: Domain properties, or URL-prefix properties at the site root. Lite and above can also add subpath (URL-prefix) properties such as https://example.com/shop/.

Dashboard Overview

Key Metrics

The dashboard header shows four tiles:

Status Breakdown

VitalSentinel surfaces Google's coverage states using Google's exact wording, so the figures line up with what you see in Search Console. Common states include:

• Submitted and indexed - In Google's index, can appear in results
• Crawled - currently not indexed - Google found it but chose not to index
• Discovered - currently not indexed - Known to Google but not yet crawled
• Blocked by robots.txt - Crawling is blocked by a robots.txt rule
• Excluded by 'noindex' tag - A noindex directive keeps it out of the index
• Duplicate without user-selected canonical - Treated as a duplicate with no canonical set
• Soft 404 - Returns 200 but looks like an error page to Google

The dashboard tracks roughly 19 coverage states in total.

Dashboard Sections

Below the metric tiles, the Overview page has three cards:

• Indexing trend - How your URLs moved between coverage states over time, across a window you choose.
• What changed in the last 7 days - Status changes Google made to your URLs, such as pages newly indexed or newly dropped.
• Where issues are right now - URLs with problems grouped by reason, so you can jump straight to the ones that need fixing.

URL Inspection Details

For each URL, you can see:

URL Status Categories

Submitted and indexed

Your page is in Google's index. This means:

• Google has crawled the page
• Content was deemed indexable
• Page can appear in search results

Crawled - currently not indexed

Google crawled the page but didn't add it to the index. Common reasons:

• Low-quality content - Thin or duplicate content
• Soft 404 - Page looks like an error but returns 200
• Duplicate content - Very similar to another page
• Low value - Not useful for searchers

What to do:

• Improve content quality
• Add unique value
• Check for duplicate content issues

Discovered - currently not indexed

Google knows the URL exists but hasn't crawled it yet. Reasons:

• Crawl budget - Site has too many URLs
• Low priority - Page seems less important
• Recent submission - Not yet processed

What to do:

• Wait for Google to crawl (can take weeks)
• Improve internal linking
• Submit sitemap

Blocked by robots.txt

The page is blocked in robots.txt. If intentional, this is expected. If not:

What to do:

• Check your robots.txt file
• Remove blocking rules for important pages
• See Robots.txt Monitoring

Excluded by 'noindex' tag

The page has a noindex directive. Check:

• Meta robots tag: <meta name="robots" content="noindex">
• X-Robots-Tag HTTP header

What to do:

• Remove noindex if the page should be indexed
• Keep noindex for pages that shouldn't appear in search

Submitting URLs

Via Sitemap

The best way to inform Google about your pages:

1. Create an XML sitemap
2. Reference it in robots.txt
3. Submit in Google Search Console

URL Inspection

Review the status of specific URLs:

1. Open Indexing in VitalSentinel and go to the Inspections tab
2. Filter or search for the URL, then open its detail drawer to see its coverage state, canonical, and last crawl
3. Pin important pages as Priority URLs so they are re-checked daily
4. Use "Request Indexing" in Search Console if you need Google to recrawl immediately

Common Issues

Crawl Budget Issues

If you have many pages, Google may not crawl them all. Solutions:

• Remove low-quality pages
• Improve site speed
• Fix redirect chains
• Block non-essential pages

Duplicate Content

Multiple URLs with the same content. Solutions:

• Use canonical tags
• Implement redirects
• Remove duplicate pages

Redirect Chains

Multiple redirects slow crawling. Solutions:

• Redirect directly to final URL
• Update internal links
• Fix redirect loops

Soft 404s

Pages that look like errors but return 200. Solutions:

• Return proper 404 for missing content
• Add content to thin pages
• Redirect to relevant pages

Best Practices

Check New Content

After publishing:

1. Wait a few days for discovery
2. Check indexing status
3. Request indexing if needed

Fix Issues Promptly

Indexing issues can:

• Delay content appearing in search
• Reduce organic traffic
• Impact SEO performance

Data Source

Indexing data comes from the Google Search Console API. You must connect your GSC account to use this feature.

Alerts

Set up alerts on any coverage state. Each alert triggers when that state's share of your URLs changes by more than a set percentage (default 15%). Examples:

• Submitted and indexed - Alert when the indexed share drops
• Crawled - currently not indexed - Alert when more pages are crawled but skipped
• Discovered - currently not indexed - Alert when discovery outpaces crawling
• Blocked by robots.txt - Alert when more pages become blocked
• Soft 404 - Alert when soft-404s rise

See Setting Up Alerts for configuration details.

Related Features