# Crawl and index reports

The **Crawl and index** group has three reports: **Indexability**, **Crawlers** and **Links**. The extension reads the page and the response that the browser got. It also fetches files from the site, for example robots.txt and the sitemaps.

## Before you start [#before-you-start]

- Open the popup and click **Crawl and index** in the sidebar.
- If the page was open before you installed the extension, reload it. The extension needs the response of the page load to read the status, the redirects and the `X-Robots-Tag` header.

## Indexability [#indexability]

The banner at the top gives one verdict:

- **Google can index this page**: the status, the robots directives, the canonical and robots.txt all allow indexing.
- **Google indexes the clean URL of this page**: the address only adds tracking parameters to the canonical address.
- **Google will not index this page**: the banner lists the reasons, for example "Meta robots says noindex".

If you are signed in and the site is monitored, the banner also shows if the page is in the Google index. This status comes from Google Search Console through VitalSentinel. See [Indexing Monitoring](/indexing-monitoring/overview).

### Signals [#signals]

| Signal | What the extension checks |
|--------|---------------------------|
| **HTTP status** | The status code and the number of redirects before it |
| **Meta robots** | The `robots` and `googlebot` meta tags |
| **X-Robots-Tag** | The `X-Robots-Tag` response header |
| **Directives** | All the robots directives together, or "None (index, follow)" |
| **Canonical** | **Self-referencing**, "Points to another URL", or a canonical in the `Link` header |
| **robots.txt** | If robots.txt allows the page for Googlebot, with the line number of the deciding rule |
| **Sitemap** | If the page is in a sitemap that the extension read |

If robots.txt answers with a server error, the signal is red. Google crawls nothing on the site until it can read the file again.

### Google index [#google-index]

If you are signed in and the site is monitored, the **Google index** section shows the index status and the last Google crawl. It also shows the canonical that Google selected. If the Google canonical is not the same as the declared canonical, the row is amber.

### Sitemaps [#sitemaps]

The extension reads the sitemaps that robots.txt declares. If robots.txt declares no sitemap, the extension looks at `/sitemap.xml` and `/sitemap_index.xml`. The table lists each sitemap file that it read, grouped by host. Each row shows the HTTP status, the kind (**Index** or **Sitemap**) and the number of URLs.

### hreflang [#hreflang]

The **hreflang** section lists the hreflang annotations of the page, with the language code and the address. The extension warns about invalid language codes, duplicate codes, a missing entry for the page itself and a missing `x-default`.

### Raw HTML vs rendered page [#raw-html-vs-rendered-page]

The section **Raw HTML vs rendered page** compares the HTML that the server sends with the page after JavaScript runs. A crawler that does not run JavaScript sees only the raw HTML.

The table compares the title, description, meta robots, canonical, H1, word count, links, JSON-LD blocks and hreflang entries. Amber marks a field that JavaScript changes. For example, "The H1 only appears after JavaScript runs".

## Crawlers [#crawlers]

The **Crawlers** report shows which crawlers robots.txt allows to fetch the page.

| Section | Crawlers |
|---------|----------|
| **Search engines** | Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, YandexBot, Applebot |
| **AI crawlers** | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Amazonbot |

Each row shows **Allowed** or **Blocked**, and the deciding rule with its line number, for example `L4 Disallow: /admin/`. If no rule matches, the row says "No matching rule".

The **robots.txt** section shows the full file. Green lines allow and red lines block one or more of the crawlers in the tables. The **Declared sitemaps** section lists the `Sitemap` lines of the file. Click **Open robots.txt** to open the file in a new tab.

If the site has no robots.txt, every crawler can fetch every address. If robots.txt answers with a server error, Google treats the file as "block everything". To watch robots.txt for changes, see [Robots.txt Monitoring](/robots-txt-monitoring/overview).

## Links [#links]

The **Links** report lists every link on the page. The tiles at the top count the total, internal, external, nofollow and unique links. A link is internal when it goes to the same site.

The report warns about links without text, internal links with `nofollow`, and links to `http://` addresses. Use the filters **All**, **Internal**, **External** and **Nofollow** to show some of the links. Each row shows the anchor text, the target, the type and the `rel` values `nofollow`, `sponsored` and `ugc`.

### Check the status of links [#check-the-status-of-links]

The extension does not check the links until you ask.

1. Select a filter, if you want to check only some links.
2. Click **Check status**.

The extension checks the first 300 links of the current list, six at a time, from your browser. The **Status** column shows the HTTP status of each link. Red marks a broken link. After the check, the **Broken** filter shows the broken links.

In offline mode, the extension checks only the links on the same host.

## Related [#related]

- [Overview report](/browser-extension/overview-report) - the summary of the page
- [AI readiness](/browser-extension/technical-reports#ai-readiness) - what AI agents can read on the page
- [Indexing Monitoring](/indexing-monitoring/overview) - track the Google index status of your pages