# Crawl vs results

The **Crawl vs results** page of Server Log Monitoring answers one question: did the crawl pay off? It puts what crawlers did to your pages next to what other VitalSentinel modules saw for the same pages. A page that Google crawls a lot but that nobody clicks is crawl you could spend elsewhere.

## Before you start [#before-you-start]

- Connect a log source first. See [Connect server logs](/connect/server-logs).
- Open a domain. In the sidebar, select **Server Log Monitoring** > **Crawl vs results**.
- Each tab needs a second module. If that module is not connected, the tab shows a connect card instead of the reports.
- Members can read this page. Only owners and admins can connect a module.

### Tabs of Crawl vs results [#tabs-of-crawl-vs-results]

| Tab | Joins your logs with | Needs |
|-----|----------------------|-------|
| **Google Search Console** | Impressions, clicks, position and URL Inspection. | A linked Google Search Console property. See [Connect Google Search Console](/connect/search-console). |
| **Web Analytics** | Visits from real people and from AI search. | The Web Analytics script. See [Web Analytics script](/connect/analytics-script). |
| **RUM Monitoring** | Real-visitor speed and Core Web Vitals. | The RUM script. See [RUM script](/connect/rum-script). |
| **Sitemap Monitoring** | The URLs your sitemap lists. | A sitemap that Sitemap Monitoring reads. See [Sitemap Monitoring](/sitemap-monitoring/overview). |
| **Page Monitoring** | Changes to the pages you watch. | At least one watched page. See [Page Monitoring](/page-monitoring/overview). |

### How the Crawl vs results tabs work [#how-the-crawl-vs-results-tabs-work]

- **Date range.** Most tabs read at most 180 days at a time. The RUM Monitoring tab reads at most 90 days. A longer range is cut, and a notice says so.
- **No comparison.** These tabs do not compare with an earlier period.
- **Filters.** The tabs use only the **URL** and **Section** filters. The RUM Monitoring tab also uses **Bot**.
- **Missing is not zero.** A figure that cannot be found shows **Not measured**, **Not observed** or **Not measurable**.

Two dates show side by side in some tables. **Googlebot last crawl** comes from your logs, and counts every verified Googlebot fetch. **Google reports** is Google's own last crawl time from URL Inspection. URL Inspection checks only a daily sample of pages, so that date can be months old.

## Google Search Console tab [#google-search-console-tab]

The **Google Search Console** tab of Crawl vs results shows what Google did with each page that Googlebot fetched. Google Search Console knows impressions, clicks and position. The logs know every fetch, its status and its timing.

If Google Search Console is not connected, the tab shows "See what Google did with each page Googlebot fetched" and a **Connect Google Search Console** button.

### Crawl to click [#crawl-to-click]

The **Crawl to click** card shows the best result each URL reached in the date range: a click, an impression, or neither. Each URL sits under one result, so the tabs add up to all URLs once.

| Tab | Meaning |
|-----|---------|
| **All URLs** | Every URL that the logs or Google Search Console reported. The note shows how many Googlebot fetched. |
| **Not crawled** | The logs never saw a verified Googlebot fetch. |
| **Crawled, not indexed** | Crawled, but URL Inspection reported it not indexed. |
| **Crawled, no impressions** | Crawled, with zero search impressions. |
| **Shown, no clicks** | Seen in search, but no clicks. |
| **Got clicks** | Google Search Console reported at least one click. |
| **Googlebot crawling not observed in this window** | Not a drop-off. The logs missed this date range. |
| **Google Search Console not observed in this window** | Not a drop-off. Google Search Console has a gap here. |

A result with no URLs has no tab. A page can earn clicks with no Googlebot fetch in the stored log days, because Google can have fetched it earlier.

### Crawl to click table [#crawl-to-click-table]

Below the tabs, a table lists the URLs of the open result. The columns are **URL**, **Stage**, **Verified bot hits**, **Googlebot last crawl**, **Impressions** and **Clicks**. **Verified bot hits** counts Googlebot, Bingbot, AI crawlers and other verified bots.

Click a row to see what each source reported about the URL: **What the logs saw**, **What Google Search Console reports** and **What Web Analytics saw**.

### Not re-crawled [#not-re-crawled]

The **Not re-crawled** card lists pages that carry impressions but that Googlebot has not fetched lately. Google can still rank an old copy of these pages.

Select a tab to set the threshold: **Not re-crawled in 7 days**, 14, 30, 60 or 90 days. A threshold tab shows only when the date range has that many days of Googlebot logs in a row. If no threshold fits, the card says **Not enough log history yet**. Select **Any impressions**, **10+ impressions**, **100+ impressions** or **1,000+ impressions** to set the smallest number of impressions.

The columns are **URL**, **Impressions**, **Clicks**, **Googlebot last crawl**, **Google reports** and **Inspection**. Update these pages, link to them from pages that Googlebot crawls often, or request indexing in Google Search Console.

### How often Googlebot visits each section [#how-often-googlebot-visits-each-section]

This card shows verified Googlebot hits per URL per day, next to how well each section ranks. It shows only if you set up site sections. A section that ranks well but is rarely crawled is a quick win.

The columns are **Section**, **URLs**, **Crawled URLs**, **Googlebot hits**, **Googlebot hits / URL / day**, **Impressions** and **Avg. position**.

## Web Analytics tab [#web-analytics-tab]

The **Web Analytics** tab of Crawl vs results shows if the pages people visit also get crawled. A popular page that bots rarely visit can be hard for Google to find.

If Web Analytics is not installed, the tab shows "See what people did on the pages crawlers fetched" and an **Install the Web Analytics script** button. While Web Analytics waits for its first data, a note says **Web Analytics connected – first data in about 5 minutes**.

### Visited, not crawled [#visited-not-crawled]

The **Visited, not crawled** card lists pages that people visited in the date range, with no verified Googlebot fetch on the same days. Link to these pages from pages that Googlebot crawls often, and add them to your sitemap.

The columns are **URL**, **Pageviews on the same days**, **Last Googlebot crawl** (any stored day) and **In sitemap**. The card needs at least one day with both server logs and Web Analytics data. Without it, it says **Needs both the logs and Web Analytics**.

### AI crawl to AI search [#ai-crawl-to-ai-search]

The **AI crawl to AI search** card lists every page an AI bot fetched, and if it later got a visit from AI search or chat. The crawl side looks back up to 90 days from the end of your date range, because a visit can follow a crawl by weeks. If less crawl detail is stored, the lookback is shorter.

Select which AI bots start the list:

- **Trains on you**: AI crawlers such as GPTBot, ClaudeBot and CCBot.
- **Indexes for answers**: AI search bots such as OAI-SearchBot and PerplexityBot.
- **Fetched live for a user**: live fetchers and agents such as ChatGPT-User and Claude-User.

### AI crawl to AI search figures [#ai-crawl-to-ai-search-figures]

The tiles show the pages the bots fetched, **Later visited from AI search**, and **Median days to first visit**. The median needs 20 new pages with a visit. Until then, it shows how many pages it has so far.

The table shows **URL**, the bots that fetched the page, **Crawl hits**, **First fetch**, **Last fetch**, **AI-search visits since crawl** and **Days to first visit**. Pages that AI bots skip are less likely to appear in AI answers.

## RUM Monitoring tab [#rum-monitoring-tab]

The **RUM Monitoring** tab of Crawl vs results shows if your server answers bots as fast as real visitors. If bots wait longer, Google can crawl your site less. **RUM** (real user monitoring) measures the speed that real visitors get.

If RUM Monitoring is not installed, the tab shows "See how fast your pages are for bots vs visitors" and an **Install the RUM script** button. This tab reads at most 90 days.

### How the RUM Monitoring tab matches pages [#how-the-rum-monitoring-tab-matches-pages]

RUM Monitoring groups similar URLs into one URL pattern, so the tab joins the two sources on these patterns. Each card shows tiles for **Server logs**, **RUM**, and the share of RUM pages that bots also crawled. A row marked with several URLs combined covers all of them together. A figure built on few samples is a hint only.

### Slow and crawled [#slow-and-crawled]

The **Slow and crawled** card lists pages below 75% good for real visitors, the most-crawled first. Select **LCP**, **INP** or **CLS**. Fix these pages first, because they cost you twice.

The columns are **URL**, **Crawl hits**, **Visitor samples**, **Good share**, the metric value (**LCP**, **INP** or **CLS**), **Needs improvement** and **Poor**. The metric value is the value of a typical visit: 3 in 4 visits are this good or better. If no page is below the bar, it says **No crawled page is below the good-share bar**.

### Bots and visitors [#bots-and-visitors]

The **Bots and visitors** card compares how fast your server answers bots and real visitors on the same pages.

| Column | Meaning |
|--------|---------|
| **Bot hits** / **Visitor samples** | How many bot hits and visitor measurements the row uses. |
| **Bot response time** / **Human response time** | The typical server response time for each: 3 in 4 hits (or visits) were this fast or faster. |
| **Delta** | Bot response time minus human. Plus (amber) means bots got the slower server. |
| **Bot / human** | Bot time divided by human time. Above 1x, bots see the slower experience. |

Select **Hide pages with few samples** to hide rows with too little data.

## Sitemap Monitoring tab [#sitemap-monitoring-tab]

The **Sitemap Monitoring** tab of Crawl vs results shows if Googlebot crawls the pages your sitemap lists. It uses the sitemaps that Sitemap Monitoring reads.

If there is no sitemap, the tab shows "See which pages Google crawls that your sitemap misses" and an **Add a sitemap** button. If Sitemap Monitoring found a sitemap recently, it can take some time to show here. This resolves on its own.

### Crawled, not listed [#crawled-not-listed]

The **Crawled, not listed** card lists pages that Googlebot fetches but that your sitemap does not list. The columns are **URL**, **Googlebot hits**, **Last status** and **Last crawled**. With Google Search Console linked, the list keeps only pages with no impressions. Files and feeds are not counted. Add the pages that matter to your sitemap, or stop the others from being crawled.

### Listed, not crawled [#listed-not-crawled]

The **Listed, not crawled** card lists sitemap URLs that Googlebot has not fetched lately. Select **Not crawled in 30 days**, 60 or 90 days. The columns are **Sitemap URL**, **Sitemap last saw it** and **Googlebot last crawled**. Check these pages for errors and internal links. A threshold tab shows only when the date range has that many days of Googlebot logs in a row. If no threshold fits, the card says **Not enough log history yet**.

### New URLs [#new-urls]

The **New URLs** card lists pages whose first verified Googlebot fetch fell in the date range. It has three tabs: **New URLs**, **In your sitemap** and **Not in your sitemap**. A new URL that is not in your sitemap can be a parameter or variant address, or one to add. This card shows even without a sitemap.

The card needs 30 days of log collection, because the first 30 days are mostly old pages. Until then, it says **Available after 30 days of collection**.

### Crawl rate [#crawl-rate]

The **Crawl rate** card compares two groups: **In the sitemap** and **Not in the sitemap (comparable URLs)**. The columns are **Counted**, **Crawled** and **Crawl rate**. Crawl rate is the share of each group that Googlebot fetched in the date range. Sitemap URLs are the pages you chose to list, so a higher crawl rate is expected either way.

## Page Monitoring tab [#page-monitoring-tab]

The **Page Monitoring** tab of Crawl vs results shows if Google came back after you changed a page. If it did not, the new version can take a long time to show in search.

If you watch no pages, the tab shows "See whether Googlebot comes back after you change a page" and a **Watch a page** button. The tab reads at most 180 days. How much change history it has depends on your plan. See [Plans and limits](/workspace/plans-and-limits).

### How fast Googlebot comes back [#how-fast-googlebot-comes-back]

The tiles show **Changes tracked**, **Re-read within 3 days** and **Typical time to re-read**. **Re-read within 3 days** counts only changes old enough to judge. Without Googlebot logs, it shows **Needs Googlebot logs**.

### Changed pages [#changed-pages]

The **Changed pages** table has three tabs:

- **Re-read**: changes that Googlebot fetched again after the change.
- **Not re-read yet**: changes that Googlebot has not fetched for longer than the page's usual gap between visits.
- **By section**: **Section**, **Changes**, **Re-read within 3 days**, **Typical time** and **Not re-read yet**.

The page tabs show **Page**, **Section**, **What changed**, **Changed on** and **Googlebot back**. **Googlebot back** is the days from the change to the next verified Googlebot fetch. Minor changes and changes from before Googlebot was logged are left out. Link to the slow pages from pages that Googlebot crawls often.

## Troubleshooting [#troubleshooting]

### Why does a Crawl vs results tab show a connect card? [#why-does-a-crawl-vs-results-tab-show-a-connect-card]

If the second module of the tab is not connected, the tab shows a connect card. Connect Google Search Console, install the Web Analytics or RUM script, add a sitemap, or watch a page.

### Why does my date range get shorter on Crawl vs results? [#why-does-my-date-range-get-shorter-on-crawl-vs-results]

Most tabs read at most 180 days, and the RUM Monitoring tab reads at most 90 days. If you pick a longer range, the tab uses the most recent days and a notice says so.

### Why does the Not re-crawled card say "Not enough log history yet"? [#why-does-the-not-re-crawled-card-say-not-enough-log-history-yet]

The card shows a threshold only when the date range has that many days of Googlebot logs in a row. If your logs cover fewer than 7 days in a row, no threshold fits. Wait until more log days are collected.

### Why does the New URLs card say "Available after 30 days of collection"? [#why-does-the-new-urls-card-say-available-after-30-days-of-collection]

The first 30 days of log collection are mostly old pages, not new ones. The card starts after 30 days.

## Related [#related]

- [Server Log Monitoring](/server-logs/overview) - how verified Googlebot is counted.
- [Google Search Console](/google-search-console/overview) - clicks, impressions and positions.
- [Web Analytics](/web-analytics/overview) - visits from people and AI search.
- [RUM Monitoring](/rum-monitoring/overview) - real-visitor speed.
- [Sitemap Monitoring](/sitemap-monitoring/overview) - the URLs your sitemap lists.
- [Page Monitoring](/page-monitoring/overview) - changes to the pages you watch.