Crawl and index reports
The Crawl and index group of the VitalSentinel SEO Extension shows if Google can index the page, which crawlers robots.txt allows, and the links on the page.
The Crawl and index group has three reports: Indexability, Crawlers and Links. The extension reads the page and the response that the browser got. It also fetches files from the site, for example robots.txt and the sitemaps.
Before you start
- Open the popup and click Crawl and index in the sidebar.
- If the page was open before you installed the extension, reload it. The extension needs the response of the page load to read the status, the redirects and the
X-Robots-Tagheader.
Indexability
The banner at the top gives one verdict:
- Google can index this page: the status, the robots directives, the canonical and robots.txt all allow indexing.
- Google indexes the clean URL of this page: the address only adds tracking parameters to the canonical address.
- Google will not index this page: the banner lists the reasons, for example "Meta robots says noindex".
If you are signed in and the site is monitored, the banner also shows if the page is in the Google index. This status comes from Google Search Console through VitalSentinel. See Indexing Monitoring.
Signals
| Signal | What the extension checks |
|---|---|
| HTTP status | The status code and the number of redirects before it |
| Meta robots | The robots and googlebot meta tags |
| X-Robots-Tag | The X-Robots-Tag response header |
| Directives | All the robots directives together, or "None (index, follow)" |
| Canonical | Self-referencing, "Points to another URL", or a canonical in the Link header |
| robots.txt | If robots.txt allows the page for Googlebot, with the line number of the deciding rule |
| Sitemap | If the page is in a sitemap that the extension read |
If robots.txt answers with a server error, the signal is red. Google crawls nothing on the site until it can read the file again.
Google index
If you are signed in and the site is monitored, the Google index section shows the index status and the last Google crawl. It also shows the canonical that Google selected. If the Google canonical is not the same as the declared canonical, the row is amber.
Sitemaps
The extension reads the sitemaps that robots.txt declares. If robots.txt declares no sitemap, the extension looks at /sitemap.xml and /sitemap_index.xml. The table lists each sitemap file that it read, grouped by host. Each row shows the HTTP status, the kind (Index or Sitemap) and the number of URLs.
hreflang
The hreflang section lists the hreflang annotations of the page, with the language code and the address. The extension warns about invalid language codes, duplicate codes, a missing entry for the page itself and a missing x-default.
Raw HTML vs rendered page
The section Raw HTML vs rendered page compares the HTML that the server sends with the page after JavaScript runs. A crawler that does not run JavaScript sees only the raw HTML.
The table compares the title, description, meta robots, canonical, H1, word count, links, JSON-LD blocks and hreflang entries. Amber marks a field that JavaScript changes. For example, "The H1 only appears after JavaScript runs".
Crawlers
The Crawlers report shows which crawlers robots.txt allows to fetch the page.
| Section | Crawlers |
|---|---|
| Search engines | Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, YandexBot, Applebot |
| AI crawlers | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot |
Each row shows Allowed or Blocked, and the deciding rule with its line number, for example L4 Disallow: /admin/. If no rule matches, the row says "No matching rule".
The robots.txt section shows the full file. Green lines allow and red lines block one or more of the crawlers in the tables. The Declared sitemaps section lists the Sitemap lines of the file. Click Open robots.txt to open the file in a new tab.
If the site has no robots.txt, every crawler can fetch every address. If robots.txt answers with a server error, Google treats the file as "block everything". To watch robots.txt for changes, see Robots.txt Monitoring.
Links
The Links report lists every link on the page. The tiles at the top count the total, internal, external, nofollow and unique links. A link is internal when it goes to the same site.
The report warns about links without text, internal links with nofollow, and links to http:// addresses. Use the filters All, Internal, External and Nofollow to show some of the links. Each row shows the anchor text, the target, the type and the rel values nofollow, sponsored and ugc.
Check the status of links
The extension does not check the links until you ask.
- Select a filter, if you want to check only some links.
- Click Check status.
The extension checks the first 300 links of the current list, six at a time, from your browser. The Status column shows the HTTP status of each link. Red marks a broken link. After the check, the Broken filter shows the broken links.
In offline mode, the extension checks only the links on the same host.
Related
- Overview report - the summary of the page
- AI readiness - what AI agents can read on the page
- Indexing Monitoring - track the Google index status of your pages
On-page reports
The On-page group of the VitalSentinel SEO Extension checks the content, structured data, social tags and images of the page that you have open.
Performance
The Performance report of the VitalSentinel SEO Extension shows Core Web Vitals measured on your device, CrUX data, an on-page overlay and a resource audit.
