Crawl vs results
Crawl vs results in Server Log Monitoring puts your bot crawl next to Google Search Console, Web Analytics, RUM Monitoring, Sitemap Monitoring and Page Monitoring.
The Crawl vs results page of Server Log Monitoring answers one question: did the crawl pay off? It puts what crawlers did to your pages next to what other VitalSentinel modules saw for the same pages. A page that Google crawls a lot but that nobody clicks is crawl you could spend elsewhere.
Before you start
- Connect a log source first. See Connect server logs.
- Open a domain. In the sidebar, select Server Log Monitoring > Crawl vs results.
- Each tab needs a second module. If that module is not connected, the tab shows a connect card instead of the reports.
- Members can read this page. Only owners and admins can connect a module.
Tabs of Crawl vs results
| Tab | Joins your logs with | Needs |
|---|---|---|
| Google Search Console | Impressions, clicks, position and URL Inspection. | A linked Google Search Console property. See Connect Google Search Console. |
| Web Analytics | Visits from real people and from AI search. | The Web Analytics script. See Web Analytics script. |
| RUM Monitoring | Real-visitor speed and Core Web Vitals. | The RUM script. See RUM script. |
| Sitemap Monitoring | The URLs your sitemap lists. | A sitemap that Sitemap Monitoring reads. See Sitemap Monitoring. |
| Page Monitoring | Changes to the pages you watch. | At least one watched page. See Page Monitoring. |
How the Crawl vs results tabs work
- Date range. Most tabs read at most 180 days at a time. The RUM Monitoring tab reads at most 90 days. A longer range is cut, and a notice says so.
- No comparison. These tabs do not compare with an earlier period.
- Filters. The tabs use only the URL and Section filters. The RUM Monitoring tab also uses Bot.
- Missing is not zero. A figure that cannot be found shows Not measured, Not observed or Not measurable.
Two dates show side by side in some tables. Googlebot last crawl comes from your logs, and counts every verified Googlebot fetch. Google reports is Google's own last crawl time from URL Inspection. URL Inspection checks only a daily sample of pages, so that date can be months old.
Google Search Console tab
The Google Search Console tab of Crawl vs results shows what Google did with each page that Googlebot fetched. Google Search Console knows impressions, clicks and position. The logs know every fetch, its status and its timing.
If Google Search Console is not connected, the tab shows "See what Google did with each page Googlebot fetched" and a Connect Google Search Console button.
Crawl to click
The Crawl to click card shows the best result each URL reached in the date range: a click, an impression, or neither. Each URL sits under one result, so the tabs add up to all URLs once.
| Tab | Meaning |
|---|---|
| All URLs | Every URL that the logs or Google Search Console reported. The note shows how many Googlebot fetched. |
| Not crawled | The logs never saw a verified Googlebot fetch. |
| Crawled, not indexed | Crawled, but URL Inspection reported it not indexed. |
| Crawled, no impressions | Crawled, with zero search impressions. |
| Shown, no clicks | Seen in search, but no clicks. |
| Got clicks | Google Search Console reported at least one click. |
| Googlebot crawling not observed in this window | Not a drop-off. The logs missed this date range. |
| Google Search Console not observed in this window | Not a drop-off. Google Search Console has a gap here. |
A result with no URLs has no tab. A page can earn clicks with no Googlebot fetch in the stored log days, because Google can have fetched it earlier.
Crawl to click table
Below the tabs, a table lists the URLs of the open result. The columns are URL, Stage, Verified bot hits, Googlebot last crawl, Impressions and Clicks. Verified bot hits counts Googlebot, Bingbot, AI crawlers and other verified bots.
Click a row to see what each source reported about the URL: What the logs saw, What Google Search Console reports and What Web Analytics saw.
Not re-crawled
The Not re-crawled card lists pages that carry impressions but that Googlebot has not fetched lately. Google can still rank an old copy of these pages.
Select a tab to set the threshold: Not re-crawled in 7 days, 14, 30, 60 or 90 days. A threshold tab shows only when the date range has that many days of Googlebot logs in a row. If no threshold fits, the card says Not enough log history yet. Select Any impressions, 10+ impressions, 100+ impressions or 1,000+ impressions to set the smallest number of impressions.
The columns are URL, Impressions, Clicks, Googlebot last crawl, Google reports and Inspection. Update these pages, link to them from pages that Googlebot crawls often, or request indexing in Google Search Console.
How often Googlebot visits each section
This card shows verified Googlebot hits per URL per day, next to how well each section ranks. It shows only if you set up site sections. A section that ranks well but is rarely crawled is a quick win.
The columns are Section, URLs, Crawled URLs, Googlebot hits, Googlebot hits / URL / day, Impressions and Avg. position.
Web Analytics tab
The Web Analytics tab of Crawl vs results shows if the pages people visit also get crawled. A popular page that bots rarely visit can be hard for Google to find.
If Web Analytics is not installed, the tab shows "See what people did on the pages crawlers fetched" and an Install the Web Analytics script button. While Web Analytics waits for its first data, a note says Web Analytics connected – first data in about 5 minutes.
Visited, not crawled
The Visited, not crawled card lists pages that people visited in the date range, with no verified Googlebot fetch on the same days. Link to these pages from pages that Googlebot crawls often, and add them to your sitemap.
The columns are URL, Pageviews on the same days, Last Googlebot crawl (any stored day) and In sitemap. The card needs at least one day with both server logs and Web Analytics data. Without it, it says Needs both the logs and Web Analytics.
AI crawl to AI search
The AI crawl to AI search card lists every page an AI bot fetched, and if it later got a visit from AI search or chat. The crawl side looks back up to 90 days from the end of your date range, because a visit can follow a crawl by weeks. If less crawl detail is stored, the lookback is shorter.
Select which AI bots start the list:
- Trains on you: AI crawlers such as GPTBot, ClaudeBot and CCBot.
- Indexes for answers: AI search bots such as OAI-SearchBot and PerplexityBot.
- Fetched live for a user: live fetchers and agents such as ChatGPT-User and Claude-User.
AI crawl to AI search figures
The tiles show the pages the bots fetched, Later visited from AI search, and Median days to first visit. The median needs 20 new pages with a visit. Until then, it shows how many pages it has so far.
The table shows URL, the bots that fetched the page, Crawl hits, First fetch, Last fetch, AI-search visits since crawl and Days to first visit. Pages that AI bots skip are less likely to appear in AI answers.
RUM Monitoring tab
The RUM Monitoring tab of Crawl vs results shows if your server answers bots as fast as real visitors. If bots wait longer, Google can crawl your site less. RUM (real user monitoring) measures the speed that real visitors get.
If RUM Monitoring is not installed, the tab shows "See how fast your pages are for bots vs visitors" and an Install the RUM script button. This tab reads at most 90 days.
How the RUM Monitoring tab matches pages
RUM Monitoring groups similar URLs into one URL pattern, so the tab joins the two sources on these patterns. Each card shows tiles for Server logs, RUM, and the share of RUM pages that bots also crawled. A row marked with several URLs combined covers all of them together. A figure built on few samples is a hint only.
Slow and crawled
The Slow and crawled card lists pages below 75% good for real visitors, the most-crawled first. Select LCP, INP or CLS. Fix these pages first, because they cost you twice.
The columns are URL, Crawl hits, Visitor samples, Good share, the metric value (LCP, INP or CLS), Needs improvement and Poor. The metric value is the value of a typical visit: 3 in 4 visits are this good or better. If no page is below the bar, it says No crawled page is below the good-share bar.
Bots and visitors
The Bots and visitors card compares how fast your server answers bots and real visitors on the same pages.
| Column | Meaning |
|---|---|
| Bot hits / Visitor samples | How many bot hits and visitor measurements the row uses. |
| Bot response time / Human response time | The typical server response time for each: 3 in 4 hits (or visits) were this fast or faster. |
| Delta | Bot response time minus human. Plus (amber) means bots got the slower server. |
| Bot / human | Bot time divided by human time. Above 1x, bots see the slower experience. |
Select Hide pages with few samples to hide rows with too little data.
Sitemap Monitoring tab
The Sitemap Monitoring tab of Crawl vs results shows if Googlebot crawls the pages your sitemap lists. It uses the sitemaps that Sitemap Monitoring reads.
If there is no sitemap, the tab shows "See which pages Google crawls that your sitemap misses" and an Add a sitemap button. If Sitemap Monitoring found a sitemap recently, it can take some time to show here. This resolves on its own.
Crawled, not listed
The Crawled, not listed card lists pages that Googlebot fetches but that your sitemap does not list. The columns are URL, Googlebot hits, Last status and Last crawled. With Google Search Console linked, the list keeps only pages with no impressions. Files and feeds are not counted. Add the pages that matter to your sitemap, or stop the others from being crawled.
Listed, not crawled
The Listed, not crawled card lists sitemap URLs that Googlebot has not fetched lately. Select Not crawled in 30 days, 60 or 90 days. The columns are Sitemap URL, Sitemap last saw it and Googlebot last crawled. Check these pages for errors and internal links. A threshold tab shows only when the date range has that many days of Googlebot logs in a row. If no threshold fits, the card says Not enough log history yet.
New URLs
The New URLs card lists pages whose first verified Googlebot fetch fell in the date range. It has three tabs: New URLs, In your sitemap and Not in your sitemap. A new URL that is not in your sitemap can be a parameter or variant address, or one to add. This card shows even without a sitemap.
The card needs 30 days of log collection, because the first 30 days are mostly old pages. Until then, it says Available after 30 days of collection.
Crawl rate
The Crawl rate card compares two groups: In the sitemap and Not in the sitemap (comparable URLs). The columns are Counted, Crawled and Crawl rate. Crawl rate is the share of each group that Googlebot fetched in the date range. Sitemap URLs are the pages you chose to list, so a higher crawl rate is expected either way.
Page Monitoring tab
The Page Monitoring tab of Crawl vs results shows if Google came back after you changed a page. If it did not, the new version can take a long time to show in search.
If you watch no pages, the tab shows "See whether Googlebot comes back after you change a page" and a Watch a page button. The tab reads at most 180 days. How much change history it has depends on your plan. See Plans and limits.
How fast Googlebot comes back
The tiles show Changes tracked, Re-read within 3 days and Typical time to re-read. Re-read within 3 days counts only changes old enough to judge. Without Googlebot logs, it shows Needs Googlebot logs.
Changed pages
The Changed pages table has three tabs:
- Re-read: changes that Googlebot fetched again after the change.
- Not re-read yet: changes that Googlebot has not fetched for longer than the page's usual gap between visits.
- By section: Section, Changes, Re-read within 3 days, Typical time and Not re-read yet.
The page tabs show Page, Section, What changed, Changed on and Googlebot back. Googlebot back is the days from the change to the next verified Googlebot fetch. Minor changes and changes from before Googlebot was logged are left out. Link to the slow pages from pages that Googlebot crawls often.
Troubleshooting
Why does a Crawl vs results tab show a connect card?
If the second module of the tab is not connected, the tab shows a connect card. Connect Google Search Console, install the Web Analytics or RUM script, add a sitemap, or watch a page.
Why does my date range get shorter on Crawl vs results?
Most tabs read at most 180 days, and the RUM Monitoring tab reads at most 90 days. If you pick a longer range, the tab uses the most recent days and a notice says so.
Why does the Not re-crawled card say "Not enough log history yet"?
The card shows a threshold only when the date range has that many days of Googlebot logs in a row. If your logs cover fewer than 7 days in a row, no threshold fits. Wait until more log days are collected.
Why does the New URLs card say "Available after 30 days of collection"?
The first 30 days of log collection are mostly old pages, not new ones. The card starts after 30 days.
Related
- Server Log Monitoring - how verified Googlebot is counted.
- Google Search Console - clicks, impressions and positions.
- Web Analytics - visits from people and AI search.
- RUM Monitoring - real-visitor speed.
- Sitemap Monitoring - the URLs your sitemap lists.
- Page Monitoring - changes to the pages you watch.
