Pages
The Pages page of Server Log Monitoring shows which URLs bots crawl, how much crawl budget is wasted, and which parts of your site bots never reach.
The Pages page of Server Log Monitoring shows what bots crawl on your site. It has three tabs: Crawled pages, Crawl budget and Crawl map. Use it to make sure that Google spends its crawl on the pages that matter.
Before you start
- Connect a log source first. See Connect server logs.
- Open a domain. In the sidebar, select Server Log Monitoring > Pages.
- Each tab uses only the filters that it supports. The Crawl map uses only the date range.
- Members can read this page.
Crawl budget is the number of pages a search engine is willing to fetch from your site in a period. Crawl waste is the part of that crawl that goes to URLs that earn nothing back.
Crawled pages tab
The Crawled pages tab of the Pages page lists every URL that a search engine or AI crawler fetched in the date range. Look for unimportant pages near the top of the list.
Pages and Files
The card has two tabs:
- Pages: different pages that any collected search or AI crawler fetched. The note shows the number of verified Googlebot hits.
- Files: different files (scripts, styles, images, feeds) that crawlers fetched.
Crawl waste warning on Crawled pages
If 10% or more of bot visits went to URLs that earn nothing, a warning shows. For example: 24% of bot visits went to URLs that earn nothing back. The detail names the number of hits and the kinds of URL (failing, redirecting, parameterised or robots-disallowed). The warning is amber from 10% and red from 30%.
Crawled pages columns
| Column | Meaning |
|---|---|
| URL | The page. Icons after the address show facts from other modules (see below). |
| Section | The site section of the page, if you set up sections. |
| Googlebot | Verified Googlebot hits. The Bots page counts every hit with Googlebot's name, so its figure is higher. |
| All crawlers | Every search engine and AI crawler that VitalSentinel collects, verified or not. |
| Last crawl | The most recent verified Googlebot visit. None in this window if there was none. |
| Status | The answer your server gave the most recent crawler, which can be any crawler. |
| Clicks | Google Search Console clicks. It needs Google Search Console. Files show no clicks. |
Facts after the URL
The icons after a URL show what other modules know about the page:
- Index status: if Google indexed the page, from Indexing Monitoring.
- Changed since the last crawl: Page Monitoring found a change after Googlebot last fetched the page. Google may still hold the old version.
- Read by AI: the AI crawlers that fetched the page in the date range.
- About this page: In your sitemap shows Yes or No, from your synced sitemaps.
Crawled pages row actions and filters
Each page row has a Watch in Page Monitoring action and an Open in menu. The menu opens the same page in other modules. Click a URL to open the page drawer.
The filters for this tab are URL, Section, Last status code, Last status class, Googlebot hits, Crawler hits, Last Googlebot crawl (days ago) and In sitemap. A URL that Googlebot never fetched counts as the longest ago.
Nightly build of Crawled pages
The list comes from a nightly build that is usually 1 to 2 days behind. If the build falls further behind, a note says, for example, Built through Oct 20, 2026 – the nightly build is running further behind than its usual 1–2 days. If no page was crawled, the table says No pages crawled in this window.
Page drawer
Click a URL on the Pages page, or in another Server Log Monitoring table, to open the page drawer. The header shows the URL, its section, and links to open the page or see it in Explore.
The page drawer has three tabs:
- Overview: one sentence about how the page answers bots, for example "The page loads normally for bots." Tiles show Verified Googlebot hits, Last Googlebot crawl and Error answers. A list shows what other modules know.
- Crawl activity: the tiles Hits, Verified Googlebot hits, Client errors and Server errors, and the Hits over time chart (one point per UTC day).
- Bots: the Bots that hit this page table with Bot, Category, Hits, Share, Status mix and Last seen, plus Response time and Bytes served.
What other modules know about the page
On the Overview tab, each row shows one module and a link to open the page there:
| Row | What it shows |
|---|---|
| Google index | Indexed by Google or Not indexed by Google, from Indexing Monitoring. |
| Search Console | Clicks and impressions for the page. |
| Web Analytics | Page views for the page. |
| AI crawlers | The AI crawlers that read the page. |
| Sitemap | If the page is in your sitemap. |
| Robots.txt | If robots.txt allows crawlers, with a link to the robots.txt tester. |
| Page Monitoring | If the page is watched, and its last change. |
| Site Health | Open issues on the page. |
Page drawer detail limits
Per-request detail is kept for about 7 days. The Bots tab shows Detail kept: About 7 days in its tooltip. For this reason, the Bots tab covers at most the last 7 days of your date range. Older crawls still count in the reports.
Crawl budget tab
The Crawl budget tab of the Pages page shows how much of the crawl goes to pages that matter and how much is wasted. Reduce the waste so that bots visit your good pages more often.
Crawl budget key numbers
| Tab | What it shows |
|---|---|
| Crawl waste | The share of bot hits on a failing, redirecting, parameterised or robots-disallowed URL. |
| Total hits | Every bot hit that VitalSentinel collects. |
| Parameter waste | The share of bot hits on a URL with a query string. Query strings are not recorded if your logs do not include them. |
| Redirected hits | The share of bot hits answered with a redirect instead of content. Only the first answer counts, not the chain behind it. |
Crawl waste uses the tones Fine (under 10%), Worth a look (10% to 30%) and Costly (30% and over). A 304 Not Modified is a healthy answer and does not count as waste.
Where the crawl went
The Where the crawl went card has four views: Sections, Content types, Query parameters and Duplicate URLs. Fix the biggest waste first.
The Crawl budget tab uses the Bot and Bot category filters. Duplicate URLs does not use them, and a note says so.
Sections view
The Sections view compares crawl with Google clicks, section by section. It needs site sections. If you have none, it says No sections configured yet. Click Set up sections to add them. Section rules apply to data already collected, so the view fills in right away.
| Column | Meaning |
|---|---|
| Section | The section. Hits that match no rule show as (unsectioned). |
| Crawl vs clicks | The group of the section (see the next table). |
| Verified crawler hits | Hits from crawlers whose identity was confirmed, over every verified bot. |
| Share of crawl | The section's part of all verified crawler hits. |
| Clicks | Google clicks on the section's pages. |
| Return on crawl | Share of clicks divided by share of crawl. Above 1x, the section earns more than it costs. |
| Change in all hits | The change against the earlier period, with comparison on. |
Open a row to see All collected hits, Unique URLs, Bytes, Response time, Last seen, Crawled URLs, Share of clicks and Impressions.
Crawl in, clicks out
The Crawl in, clicks out chart places each section by its share of crawl (across) and share of clicks (up). It shows only when there are enough sections to compare.
| Group | Meaning |
|---|---|
| Workhorse | Earns a bigger share of the clicks than it takes of the crawl. |
| Crawl drain | Takes a big share of the crawl and returns few clicks. |
| Underfed | Earns clicks on a small share of the crawl. |
| Long tail | Small share of both. Cheap, and not where the traffic is. |
If much of the crawl matches no section rule, a note shows the share of crawl and clicks outside all rules.
Content types view
The Content types view shows what the bots fetched: pages, images, scripts and the rest. The columns are Content type, Hits, Share and Bytes. Bytes are left out if your logs report response size for only some types.
Query parameters view
The Query parameters view lists the query parameters that bots fetched most. The columns are Parameter, Hits, Unique URLs and Share. Click a parameter to see the pages in Explore. Typical fixes are canonical tags, robots.txt rules, or the removal of the parameter from internal links. If no URL with a parameter was crawled, it says No parameterised URLs crawled.
Duplicate URLs view
The Duplicate URLs view (duplicate-variant crawl) finds URLs that crawlers fetch under more than one address, but that are almost certainly one page. It has three tabs:
- Trailing slash: the same path with and without a slash at the end.
- Case variants: the same path in different upper and lower case. These are candidates, not confirmed duplicates, because most servers treat paths case-sensitively.
- Repeat visits with query strings: URLs fetched both bare and with parameters.
The columns are Duplicate group, Serving most traffic, Total hits and Redundant hits. Redundant hits is the crawl spent on the less-used addresses. If none are found, it says No duplicate clusters in this window.
Crawl map tab
The Crawl map tab of the Pages page shows your site as folders, and which parts Googlebot reaches. A folder that no bot visits is unlikely to rank. The Crawl map uses only the date range, not the other filters.
Crawl map key numbers
| Tile | Meaning |
|---|---|
| Known URLs | URLs that VitalSentinel knows from your sitemap, Google Search Console (performance or URL Inspection) and Web Analytics. All-time, not only this date range. |
| Not crawled by Googlebot | Known URLs that Googlebot did not fetch in the date range. Click the tile to see the URLs. |
| Googlebot coverage | Known URLs that Googlebot fetched at least once. It shows Not measured, never 0%, if no day has log data. |
URLs only in logs are URLs that only your logs ever recorded. They show in the folders, but not in the coverage figures.
Folders view
Select Folders to see the Site structure list. Expand a folder to see the crawl coverage of everything beneath it. The columns are Depth, Known URLs, Crawled, Not crawled, Coverage and Bot hits. Hover a folder to see its Last verified visit and Top bot. Each level shows 20 folders per page.
Treemap view
Select Treemap to see each folder as a tile. The tile size is the number of known URLs. The color has two modes:
- Coverage: Under 20% reached, 20% to 60%, 60% and over or Not measured.
- Crawl share vs. URL share: from Much more site than crawl to More crawl than site.
The treemap needs a larger screen. On a small screen, it says The crawl-map treemap needs a larger display.
Not crawled by Googlebot list
Click a Not crawled count to open the list of URLs that Googlebot did not fetch. For each URL, Known from shows where VitalSentinel knows it from: Sitemap, Google Search Console, URL Inspection, Web Analytics or Server logs. Reachable by shows if Another bot reached it. "Not crawled" always means not crawled by verified Googlebot in this date range.
Troubleshooting
Why does the Crawled pages list not show today?
The list comes from a nightly build that is usually 1 to 2 days behind. If it is more than 2 days behind, a note says so. Wait for the next build.
Why are the Googlebot hits on Crawled pages lower than on the Bots page?
The Googlebot column counts verified Googlebot hits only. The Bots page counts every hit with the Googlebot name, fake hits included.
Why does the Sections view say "No sections configured yet"?
If you have no site sections, the Sections view and the Crawl in, clicks out chart cannot show. Click Set up sections, or open the domain settings. See Domain settings.
Why does Parameter waste say Query strings are not recorded?
If your logs do not record query strings, Server Log Monitoring cannot measure parameter waste. The tile says Query strings are not recorded.
Related
- Server Log Monitoring - how crawl waste and verification work.
- Responses - the URLs that give bots errors or redirects.
- Crawl vs results - crawl next to clicks, visits and your sitemap.
- Domain settings - set up site sections.
- Blocked pages - the pages your robots.txt keeps crawlers away from.
- Sitemap Monitoring - the URLs your sitemap lists.
