Server Log Reports

Once a source has delivered its first request, Server Log Monitoring opens these reports. Every report counts only the traffic the module collects: search engines and AI crawlers. See What gets collected.

Date range, comparison and filtersCopy Anchor Link

The controls above each report work the same way across the module.

Date rangeCopy Anchor Link

The default window is Last 30 days, including today. Presets: Last 7 days, Last 28 days, Last 30 days, Last 90 days, Last 6 months, Last 12 months, This year (Jan - Today), Last calendar year, and Custom.

Most reports go back up to 12 months. A longer custom range is shortened at its start. When a report can only cover part of your range, an amber notice names the window it actually shows and why, for example because request-level detail is kept for a shorter time.

The date range, comparison and filters are kept in the page URL, so you can bookmark or share a view, and they carry across pages in the module.

ComparisonCopy Anchor Link

Tick Enable comparison in the date picker and choose Previous Period, Previous Month, Year over Year or Custom. Tiles then show the change against that window:

• +x% / -x%: the change, in neutral colors (more crawling is not automatically good or bad)
• Flat: less than 0.5% change
• No prior-period baseline: the comparison window had no traffic
• No comparable prior period: one of the windows had to be shortened, so the two can't be compared fairly

FiltersCopy Anchor Link

Click Filter to narrow a report. Available dimensions:

Text filters can match exactly, by "contains", or by regular expression, and each can be negated to exclude matches.

Not every report supports every filter:

• When a filter can't be applied on the current page, a notice says Not applied here and offers to remove it.
• On Overview, Bots and Security, an exact Status code filter can't be combined with Verification.
• With an exact status code, figures that need every response (response time, crawl waste, verified share, bytes) read Not available with a status-code filter.

Common figuresCopy Anchor Link

• Response time is the 75th percentile time to first byte (TTFB): how long your server took to start answering. Three in four requests were faster, so it describes a typical slow request rather than the worst one.
• Slowest responses is the 95th percentile TTFB: the slow tail.
• Status mix is a stacked bar of 2xx Success, 3xx Redirect, 4xx Client error and 5xx Server error.

OverviewCopy Anchor Link

The module's dashboard: which bots and crawlers hit your server, and how much of your crawl budget they waste.

All bots (follows your filters):

Googlebot only (verified Googlebot only; your filters do not apply here):

Below the tiles:

• Hits over time: bot hits stacked by bot category, by hour, day, week or month. Relative switches to each category's share. Markers show when robots.txt changed and when Uptime Monitoring recorded Downtime, so you can see whether a crawl change lines up with either.
• Googlebot crawl capacity: verified Googlebot hits as bars against Googlebot's response time as a line, with a one-sentence takeaway. A slow server is one of the main reasons Google crawls less.
• Top bots: the 10 busiest bots, with category, hits, share and response time. Click a bot to filter the page to it, or View all bots.
• Open findings: server log alerts currently firing for this domain. Shown only when something is firing.

BotsCopy Anchor Link

All botsCopy Anchor Link

Every bot that hit the site in the window, and its share of the traffic. Quick chips switch between All, Search engines and AI.

Click a bot to open its detail: hits, share of all bot hits, verified share, response time, first and last seen, how its requests were answered, and hits over time. Open in Explore lists the pages it hit.

AI crawlersCopy Anchor Link

Which AI crawlers, search indexers and assistants actually read your site, and whether they respect robots.txt while doing it.

• Tiles: Total bot hits, AI crawler hits, AI crawler share.
• By intent splits AI traffic into Training (AI crawler), Live fetch (AI assistant) and Other (search / agent). A training crawl is a licensing question; a live fetch means a person is asking a question about your page right now.
• All AI crawlers lists every AI bot with hits, share, verified share, Robots.txt compliance (share of its hits that stayed within what your robots.txt allows it to fetch), Blocked (403), Bytes served (where recorded) and last seen.

Two caveats:

• On very high-traffic sites, part of each day's request-level detail is trimmed before storage. A compliance rate measured on what remains would look better than reality, so it shows Not measured instead.
• Bytes served can understate what a vendor took, because some hits report no size.

AI coverage gap lists pages Googlebot crawled successfully in the window that no AI crawler fetched, ordered by Googlebot hits. Tiles show Crawled by Googlebot, Also fetched by AI and Missed by AI crawlers. Use it to find content that Google sees but AI search and assistants don't. It needs one full day of collection first, and ignores filters and comparison.

SecurityCopy Anchor Link

Bot identities faked on your site, and the networks they came from.

• Tiles: Total bot hits, Spoofed hits, Unverified hits, Verified hits.
• Fake bots lists requests claiming to be Googlebot, Bingbot or an AI crawler from an address that bot does not use: Claimed identity, Hits, Share of fake hits and Networks. Click a row to see which networks (AS numbers) the fake identity came from.

Many different bot names arriving from one network is itself a signal. The IP addresses of spoofed and unverified requests are never stored, so this report groups them by network instead.

PagesCopy Anchor Link

Crawled pagesCopy Anchor Link

Every URL a search or AI crawler fetched, Googlebot first.

The table covers up to 12 months, has no row cap, and is rebuilt nightly. If the nightly build runs more than 2 days behind, a note says so.

Click a URL to open its detail: hits, bots that fetched it, response time, bytes served, first and last fetch, every status code it returned, hits over time, and which bots hit it. Per-request detail is kept for about 7 days, so the page detail covers at most the last 7 days of your range. Older crawls still count in the reports you opened it from.

To get told when a page changes, click Watch in Page Monitoring (the eye icon) on its row. See Page Monitoring.

SectionsCopy Anchor Link

Crawl activity grouped by your section rules, so you can see where bot traffic lands across your site: Section, Hits, Share, Unique URLs, Bytes, Response time and Last seen. Hits that match no rule appear as (unsectioned).

Section rules apply to data already collected, so this page fills in as soon as you add a rule. Click a section to open it in Explore. If you have no rules yet, click Set up sections.

Crawl budgetCopy Anchor Link

Where the window's bot traffic went, and how much of it earned nothing back.

Crawl waste is the share of bot hits that went to a URL earning nothing back:

• a redirect (3xx)
• an error (4xx or 5xx)
• a URL carrying query parameters
• a URL your robots.txt disallows

A 304 Not Modified is a healthy answer and is not counted as waste.

The page then breaks the budget down:

• By response status: how the window's hits were answered.
• Where the budget goes: HTML share (hits that reached an HTML page), Parameter waste (hits that carried a query string) and Redirect hops (hits that got a redirect instead of content). A card reads Not available when your logs don't record content type or query strings.
• By content type: hits, share and bytes per content type.
• Top query parameters: which parameter names attract crawling, for Googlebot or All bots. Click a parameter to see the affected pages in Explore. Typical fixes are canonical tags, robots.txt rules, or removing the parameter from internal links.

Crawl budget ROI plots each section's share of crawl against its share of Search Console clicks. It needs section rules and a linked Search Console property.

Duplicate-variant crawl finds URLs crawled under more than one address that are almost certainly the same page, in three tabs: Trailing slash, Case variants and Repeat visits with query strings. Redundant hits is the crawl spent on the less-used variants. Treat case variants as candidates, not confirmed duplicates: most servers treat paths case-sensitively.

The Bot, Bot category and Section filters don't apply to Crawl budget ROI or Duplicate-variant crawl.

Crawl mapCopy Anchor Link

Which parts of your site a bot has reached, and which it has never touched. Switch between Folders, Treemap and Folder depth.

• Folders shows your site as a folder tree. Expand a folder to see the crawl coverage of everything beneath it, with bot hits, status mix, last visit and top bot.
• Treemap sizes each tile by how many URLs are known and colors it by Coverage or by Crawl share vs. URL share. Needs a larger screen.
• Folder depth shows how many folders deep Googlebot's crawl reaches, level by level.

Click a Not crawled by Googlebot count to list those URLs. For each one you see where VitalSentinel knows it from (Sitemap, Search Console, URL Inspection, Web Analytics or Server logs), and whether another verified crawler, or Googlebot before this window, has reached it.

"Not crawled" always means not crawled by verified Googlebot in this window.

ResponsesCopy Anchor Link

How bots are actually being answered.

• Tiles: Total hits, Failure rate (share answered 4xx or 5xx; red from 5%) and Redirect share (share answered 3xx, excluding 304; amber from 10%).
• Status codes over time: one line per Exact code or per class (By class), by hour, day, week or month. Errors only hides successful responses and Relative switches to shares. The hourly view covers up to 92 days.
• Bot by status: every distinct status code returned to each bot. Click a bot to filter the page to it.

URLs to fixCopy Anchor Link

Every URL crawlers got an error or a redirect for, one row per URL, in three tabs: Not found (4xx), Server errors (5xx) and Redirected (3xx). Switch between Search engines and All bots.

Columns: URL, Status, Googlebot hits, Other search/AI hits, First seen, Last seen, and In sitemap (shown when your sitemap is synced). The page filters don't apply to this list.

Start with the URLs Googlebot hits most, and with errors on URLs that are in your sitemap.

This table also has the Watch in Page Monitoring button on each row.

ExploreCopy Anchor Link

Find exactly which crawler hit which page with a given status code, and share the answer as a link. Every setting on the page is kept in the URL.

• Group by: Bot + page (default), Bot, Page, Network or IP prefix.
• Sort: Most hits or Most recent.
• Every view ends with Hits, Bytes, Response time and Last seen. Click a URL to open its page detail.

Explore reads individual requests going back up to 12 months, and supports every filter.

Unverified and spoofed requests have their IP redacted when they are stored, so they have no network to group by. Query-parameter names are recorded but their values never are.

RelatedCopy Anchor Link