Server Log Intelligence
Join server logs with Search Console, Web Analytics, RUM and your sitemap to find stale money pages, crawl-to-click drop-offs, orphan pages and slow pages Google crawls most.
Server logs record what crawlers did to your pages. Other VitalSentinel modules record what Google reported about those pages, what people did on them, how fast they were for real visitors, and what your sitemap claims. Intelligence joins them on the same canonical URL to answer questions neither source can answer alone.
Open Intelligence in the Server Log Monitoring sidebar. It has four tabs:
| Tab | Joins server logs with | Needs |
|---|---|---|
| Search Console | Google Search Console and URL Inspection | A linked Search Console property |
| Web Analytics | Web Analytics | The Web Analytics script on the domain |
| RUM | RUM Monitoring | The RUM script on the domain |
| Sitemap | Sitemap Monitoring | A sitemap that Sitemap Monitoring has read for the domain |
How these tabs work
- Window: each view covers up to 180 days. A longer range is shortened, and the first card says so.
- No filters: these reports have no bot, status or section breakdown, so the filter button is not offered. If filters carry over from another page, a notice says they don't apply here, with a button to clear them.
- No comparison: the date picker sits in the header of the first card and has comparison turned off.
- Coverage first: the first card on a tab warns when a source is missing or only covers part of the window, and figures then cover the overlapping days only. The ? next to it gives the full breakdown.
- Missing is not zero: a figure that can't be established reads Not measured, Not observed or Not measurable instead of 0.
Two dates appear side by side on several tables, on purpose:
- Logs: Googlebot last crawl comes from your server logs: every verified Googlebot fetch, nothing sampled.
- Google reports is Google's own last crawl time from URL Inspection. URL Inspection covers only 2,000 URLs per property per day, so its date can be up to three months old and most URLs have never been inspected. "Never inspected" does not mean the URL is missing from the index.
Search Console
What Google reported about each page, next to what the logs saw.
Crawl-to-click funnel
Where each URL's journey ended, from a Googlebot fetch through to a click. Every URL sits at the furthest stage it reached:
| Stage | Meaning |
|---|---|
| Not fetched by Googlebot in this window | The logs never saw a Googlebot fetch |
| Crawled, reported not indexed | Googlebot fetched it, but Google reports it as not indexed |
| Crawled, no impressions | Crawled, but no search impressions |
| Impressions, no clicks | Seen in search, but never clicked |
| Clicks | Made it all the way to a click |
| Googlebot crawling not observed in this window | Not a drop-off: the logs have no Googlebot data for this window |
| Search Console not observed in this window | Not a drop-off: Search Console has no data for this window |
Summary panels count URLs in scope, Seen crawled, With impressions, Indexation confirmed, Indexation unknown and With clicks. Click a stage to filter the table below to it.
Every URL in the window lists each URL with its section, stage, verified hits, Googlebot last crawl, impressions, clicks, position and pageviews. Click a row for everything four sources reported about that URL: what the logs saw, what Search Console reports, what URL Inspection says, and what Web Analytics saw.
Stale crawl on pages with impressions
Pages carrying search impressions that Googlebot has not fetched lately. Pick a threshold of 7, 14, 30 (default), 60 or 90 days.
- Impressions at risk: impressions on pages Googlebot hasn't fetched within the threshold
- Pages matching: how many pages that is
- Crawl-age buckets: impressed pages uncrawled past 30, 60 and 90 days
A threshold longer than your stored log history can't be checked, so its counts read as not measurable rather than zero.
These pages earn impressions, but Google is ranking an old copy. Refresh them, link to them from pages Googlebot crawls often, or request indexing in Search Console.
Underperforming pages, by crawl status
Pages earning clearly fewer clicks than their ranking position should bring, split by whether Google has re-read them since you changed them.
Enter the date you shipped a change (for example, new titles) in Content changed on. Without a date, the card falls back to a plain staleness threshold and says so.
| Verdict | Meaning |
|---|---|
| Not re-read since | Googlebot hasn't fetched the page since the change, so the old title still ranks |
| Re-read, still short | Googlebot re-fetched it and it still underperforms: the title or snippet itself is the problem |
| Crawl unknown | The log history can't place the crawl before or after the date |
The expected click-through rate comes from your property's own click curve where possible, or an industry reference curve. A page is only flagged when the whole 95% confidence range of its actual CTR sits below the expected rate. Clicks left behind estimates the clicks missing at the page's current position.
New pages: Googlebot to first impression
How long it takes from a page's first verified Googlebot fetch to its first search impression. The card shows New pages found, the median Measured latency, and pages Not yet observed in search.
It becomes available after 30 days of collection, because the first 30 days are mostly backfill rather than new pages.
Crawl frequency by ranking position
How often Googlebot re-visits each section, next to how well that section ranks: URLs, crawled URLs, Googlebot hits, Googlebot hits / URL / day, impressions and average position. Only shown when you have section rules.
Web Analytics
What crawlers did on each page, next to what people did on it. The two are collected on separate pipelines; no human request is read by the log module.
From AI crawl to AI search traffic
Every page an AI crawler fetched, and whether it later got a visit from an AI search or chat product.
- Pages AI crawlers fetched
- Later visited from AI search
- Fetched, not yet visited
- Median days to first visit
The crawl side always looks back 90 days from the end of your window, because a visit can follow a crawl by weeks. Without Web Analytics, the AI crawl rows still show and a Connect Web Analytics button appears.
Pages people visit that Googlebot hasn't crawled
Pages real people visited that verified Googlebot never fetched in the window, with pageviews, the last Googlebot crawl (across all retained history) and whether the page is in your sitemap. Help Google find them: link to them internally and add them to your sitemap.
The card needs at least one day covered by both server logs and Web Analytics.
RUM
How fast your pages served bots, next to how fast they served people. RUM redacts token-like path segments before storage, so the two are joined on URL templates. Each card shows a Template match rate; below 20%, treat the results with care.
Bot and human response time
For the same page: Bot response time and Human response time (both 75th percentile TTFB), the Delta (bot minus human) and Bot / human ratio. A positive delta means bots get a slower server than people do, which can limit how much Google crawls. A row marked Example of N URLs covers several URLs sharing one page template.
Poor Core Web Vitals on heavily-crawled pages
Pages Google crawls the most that still fail Core Web Vitals for real visitors. Pick LCP, INP or CLS. Pages below 75% good are listed worst-crawled first, with crawl hits, human events, good share, the 75th percentile value, and the needs-improvement and poor shares. Start your Core Web Vitals work at the top.
Sitemap
What your sitemap claims, next to what Googlebot actually fetched. The tab uses the sitemaps that Sitemap Monitoring reads automatically every day. Before the first check is complete, the tab shows "We're reading your sitemaps for the first time." with a See Sitemap Monitoring link.
Without a sitemap, sitemap figures are marked as not connected rather than shown as zero. A sitemap that Sitemap Monitoring found recently can take some time to show here. This resolves automatically.
Crawled by Google, missing from your sitemap
Pages Googlebot fetches that your sitemap does not list, with section, Googlebot hits, last status and last crawl. Add them to the sitemap or stop them being crawled. With Search Console linked, the list is narrowed to pages with no impressions.
Sitemap URLs Googlebot has stopped visiting
Sitemap URLs with no verified Googlebot fetch within 30 (default), 60 or 90 days. A threshold longer than the stored log history reads as not measurable.
Does the sitemap change crawl behavior
Compares the crawl rate of sitemap URLs with a comparable set of URLs that were never submitted. Lift is the sitemap crawl rate divided by the non-sitemap crawl rate. At 1x or lower, the sitemap is not measurably changing how Googlebot crawls.
New URLs Googlebot found this week
Pages whose first verified Googlebot fetch fell in the last 7 days, and whether each is in your sitemap. This card always covers the last 7 days, whatever date range you pick. A new URL that is not in your sitemap may be a parameter or variant URL, or one to add. Available after 30 days of collection.
Related
Server Log Reports
Read the Server Log Monitoring reports - Overview, Bots, AI crawlers, Security, Crawled pages, Sections, Crawl budget, Crawl map, Responses and Explore.
Server Log Settings
Manage your log source, track the monthly server log hit allowance, see what traffic is excluded and why, exclude your own agents, and group URLs into sections.
