# Server Log Monitoring: Pages

The **Pages** page of Server Log Monitoring shows what bots crawl on your site. It has three tabs: **Crawled pages**, **Crawl budget** and **Crawl map**. Use it to make sure that Google spends its crawl on the pages that matter.

## Before you start [#before-you-start]

- Connect a log source first. See [Connect server logs](/connect/server-logs).
- Open a domain. In the sidebar, select **Server Log Monitoring** > **Pages**.
- Each tab uses only the filters that it supports. The Crawl map uses only the date range.
- Members can read this page.

**Crawl budget** is the number of pages a search engine is willing to fetch from your site in a period. **Crawl waste** is the part of that crawl that goes to URLs that earn nothing back.

## Crawled pages tab [#crawled-pages-tab]

The **Crawled pages** tab of the Pages page lists every URL that a search engine or AI crawler fetched in the date range. Look for unimportant pages near the top of the list.

### Pages and Files [#pages-and-files]

The card has two tabs:

- **Pages**: different pages that any collected search or AI crawler fetched. The note shows the number of verified Googlebot hits.
- **Files**: different files (scripts, styles, images, feeds) that crawlers fetched.

### Crawl waste warning on Crawled pages [#crawl-waste-warning-on-crawled-pages]

If 10% or more of bot visits went to URLs that earn nothing, a warning shows. For example: **24% of bot visits went to URLs that earn nothing back**. The detail names the number of hits and the kinds of URL (failing, redirecting, parameterised or robots-disallowed). The warning is amber from 10% and red from 30%.

### Crawled pages columns [#crawled-pages-columns]

| Column | Meaning |
|--------|---------|
| **URL** | The page. Icons after the address show facts from other modules (see below). |
| **Section** | The site section of the page, if you set up sections. |
| **Googlebot** | Verified Googlebot hits. The Bots page counts every hit with Googlebot's name, so its figure is higher. |
| **All crawlers** | Every search engine and AI crawler that VitalSentinel collects, verified or not. |
| **Last crawl** | The most recent verified Googlebot visit. **None in this window** if there was none. |
| **Status** | The answer your server gave the most recent crawler, which can be any crawler. |
| **Clicks** | [Google Search Console](/google-search-console/overview) clicks. It needs Google Search Console. Files show no clicks. |

### Facts after the URL [#facts-after-the-url]

The icons after a URL show what other modules know about the page:

- **Index status**: if Google indexed the page, from [Indexing Monitoring](/indexing-monitoring/overview).
- **Changed since the last crawl**: [Page Monitoring](/page-monitoring/overview) found a change after Googlebot last fetched the page. Google may still hold the old version.
- **Read by AI**: the AI crawlers that fetched the page in the date range.
- **About this page**: **In your sitemap** shows **Yes** or **No**, from your synced sitemaps.

### Crawled pages row actions and filters [#crawled-pages-row-actions-and-filters]

Each page row has a **Watch in Page Monitoring** action and an **Open in** menu. The menu opens the same page in other modules. Click a URL to open the page drawer.

The filters for this tab are **URL**, **Section**, **Last status code**, **Last status class**, **Googlebot hits**, **Crawler hits**, **Last Googlebot crawl (days ago)** and **In sitemap**. A URL that Googlebot never fetched counts as the longest ago.

### Nightly build of Crawled pages [#nightly-build-of-crawled-pages]

The list comes from a nightly build that is usually 1 to 2 days behind. If the build falls further behind, a note says, for example, **Built through Oct 20, 2026 – the nightly build is running further behind than its usual 1–2 days.** If no page was crawled, the table says **No pages crawled in this window**.

## Page drawer [#page-drawer]

Click a URL on the Pages page, or in another Server Log Monitoring table, to open the page drawer. The header shows the URL, its section, and links to open the page or see it in **Explore**.

The page drawer has three tabs:

- **Overview**: one sentence about how the page answers bots, for example "The page loads normally for bots." Tiles show **Verified Googlebot hits**, **Last Googlebot crawl** and **Error answers**. A list shows what other modules know.
- **Crawl activity**: the tiles **Hits**, **Verified Googlebot hits**, **Client errors** and **Server errors**, and the **Hits over time** chart (one point per UTC day).
- **Bots**: the **Bots that hit this page** table with **Bot**, **Category**, **Hits**, **Share**, **Status mix** and **Last seen**, plus **Response time** and **Bytes served**.

### What other modules know about the page [#what-other-modules-know-about-the-page]

On the **Overview** tab, each row shows one module and a link to open the page there:

| Row | What it shows |
|-----|---------------|
| **Google index** | **Indexed by Google** or **Not indexed by Google**, from Indexing Monitoring. |
| **Search Console** | Clicks and impressions for the page. |
| **Web Analytics** | Page views for the page. |
| **AI crawlers** | The AI crawlers that read the page. |
| **Sitemap** | If the page is in your sitemap. |
| **Robots.txt** | If robots.txt allows crawlers, with a link to the robots.txt tester. |
| **Page Monitoring** | If the page is watched, and its last change. |
| **Site Health** | Open issues on the page. |

### Page drawer detail limits [#page-drawer-detail-limits]

Per-request detail is kept for about 7 days. The **Bots** tab shows **Detail kept: About 7 days** in its tooltip. For this reason, the Bots tab covers at most the last 7 days of your date range. Older crawls still count in the reports.

## Crawl budget tab [#crawl-budget-tab]

The **Crawl budget** tab of the Pages page shows how much of the crawl goes to pages that matter and how much is wasted. Reduce the waste so that bots visit your good pages more often.

### Crawl budget key numbers [#crawl-budget-key-numbers]

| Tab | What it shows |
|-----|---------------|
| **Crawl waste** | The share of bot hits on a failing, redirecting, parameterised or robots-disallowed URL. |
| **Total hits** | Every bot hit that VitalSentinel collects. |
| **Parameter waste** | The share of bot hits on a URL with a query string. **Query strings are not recorded** if your logs do not include them. |
| **Redirected hits** | The share of bot hits answered with a redirect instead of content. Only the first answer counts, not the chain behind it. |

Crawl waste uses the tones **Fine** (under 10%), **Worth a look** (10% to 30%) and **Costly** (30% and over). A **304 Not Modified** is a healthy answer and does not count as waste.

### Where the crawl went [#where-the-crawl-went]

The **Where the crawl went** card has four views: **Sections**, **Content types**, **Query parameters** and **Duplicate URLs**. Fix the biggest waste first.

The Crawl budget tab uses the **Bot** and **Bot category** filters. **Duplicate URLs** does not use them, and a note says so.

### Sections view [#sections-view]

The **Sections** view compares crawl with Google clicks, section by section. It needs site sections. If you have none, it says **No sections configured yet**. Click **Set up sections** to add them. Section rules apply to data already collected, so the view fills in right away.

| Column | Meaning |
|--------|---------|
| **Section** | The section. Hits that match no rule show as **(unsectioned)**. |
| **Crawl vs clicks** | The group of the section (see the next table). |
| **Verified crawler hits** | Hits from crawlers whose identity was confirmed, over every verified bot. |
| **Share of crawl** | The section's part of all verified crawler hits. |
| **Clicks** | Google clicks on the section's pages. |
| **Return on crawl** | Share of clicks divided by share of crawl. Above 1x, the section earns more than it costs. |
| **Change in all hits** | The change against the earlier period, with comparison on. |

Open a row to see **All collected hits**, **Unique URLs**, **Bytes**, **Response time**, **Last seen**, **Crawled URLs**, **Share of clicks** and **Impressions**.

### Crawl in, clicks out [#crawl-in-clicks-out]

The **Crawl in, clicks out** chart places each section by its share of crawl (across) and share of clicks (up). It shows only when there are enough sections to compare.

| Group | Meaning |
|-------|---------|
| **Workhorse** | Earns a bigger share of the clicks than it takes of the crawl. |
| **Crawl drain** | Takes a big share of the crawl and returns few clicks. |
| **Underfed** | Earns clicks on a small share of the crawl. |
| **Long tail** | Small share of both. Cheap, and not where the traffic is. |

If much of the crawl matches no section rule, a note shows the share of crawl and clicks outside all rules.

### Content types view [#content-types-view]

The **Content types** view shows what the bots fetched: pages, images, scripts and the rest. The columns are **Content type**, **Hits**, **Share** and **Bytes**. Bytes are left out if your logs report response size for only some types.

### Query parameters view [#query-parameters-view]

The **Query parameters** view lists the query parameters that bots fetched most. The columns are **Parameter**, **Hits**, **Unique URLs** and **Share**. Click a parameter to see the pages in Explore. Typical fixes are canonical tags, robots.txt rules, or the removal of the parameter from internal links. If no URL with a parameter was crawled, it says **No parameterised URLs crawled**.

### Duplicate URLs view [#duplicate-urls-view]

The **Duplicate URLs** view (duplicate-variant crawl) finds URLs that crawlers fetch under more than one address, but that are almost certainly one page. It has three tabs:

- **Trailing slash**: the same path with and without a slash at the end.
- **Case variants**: the same path in different upper and lower case. These are candidates, not confirmed duplicates, because most servers treat paths case-sensitively.
- **Repeat visits with query strings**: URLs fetched both bare and with parameters.

The columns are **Duplicate group**, **Serving most traffic**, **Total hits** and **Redundant hits**. **Redundant hits** is the crawl spent on the less-used addresses. If none are found, it says **No duplicate clusters in this window**.

## Crawl map tab [#crawl-map-tab]

The **Crawl map** tab of the Pages page shows your site as folders, and which parts Googlebot reaches. A folder that no bot visits is unlikely to rank. The Crawl map uses only the date range, not the other filters.

### Crawl map key numbers [#crawl-map-key-numbers]

| Tile | Meaning |
|------|---------|
| **Known URLs** | URLs that VitalSentinel knows from your sitemap, Google Search Console (performance or URL Inspection) and Web Analytics. All-time, not only this date range. |
| **Not crawled by Googlebot** | Known URLs that Googlebot did not fetch in the date range. Click the tile to see the URLs. |
| **Googlebot coverage** | Known URLs that Googlebot fetched at least once. It shows **Not measured**, never 0%, if no day has log data. |

**URLs only in logs** are URLs that only your logs ever recorded. They show in the folders, but not in the coverage figures.

### Folders view [#folders-view]

Select **Folders** to see the **Site structure** list. Expand a folder to see the crawl coverage of everything beneath it. The columns are **Depth**, **Known URLs**, **Crawled**, **Not crawled**, **Coverage** and **Bot hits**. Hover a folder to see its **Last verified visit** and **Top bot**. Each level shows 20 folders per page.

### Treemap view [#treemap-view]

Select **Treemap** to see each folder as a tile. The tile size is the number of known URLs. The color has two modes:

- **Coverage**: **Under 20% reached**, **20% to 60%**, **60% and over** or **Not measured**.
- **Crawl share vs. URL share**: from **Much more site than crawl** to **More crawl than site**.

The treemap needs a larger screen. On a small screen, it says **The crawl-map treemap needs a larger display**.

### Not crawled by Googlebot list [#not-crawled-by-googlebot-list]

Click a **Not crawled** count to open the list of URLs that Googlebot did not fetch. For each URL, **Known from** shows where VitalSentinel knows it from: **Sitemap**, **Google Search Console**, **URL Inspection**, **Web Analytics** or **Server logs**. **Reachable by** shows if **Another bot** reached it. "Not crawled" always means not crawled by verified Googlebot in this date range.

## Troubleshooting [#troubleshooting]

### Why does the Crawled pages list not show today? [#why-does-the-crawled-pages-list-not-show-today]

The list comes from a nightly build that is usually 1 to 2 days behind. If it is more than 2 days behind, a note says so. Wait for the next build.

### Why are the Googlebot hits on Crawled pages lower than on the Bots page? [#why-are-the-googlebot-hits-on-crawled-pages-lower-than-on-the-bots-page]

The **Googlebot** column counts verified Googlebot hits only. The Bots page counts every hit with the Googlebot name, fake hits included.

### Why does the Sections view say "No sections configured yet"? [#why-does-the-sections-view-say-no-sections-configured-yet]

If you have no site sections, the Sections view and the Crawl in, clicks out chart cannot show. Click **Set up sections**, or open the domain settings. See [Domain settings](/domain/settings).

### Why does Parameter waste say Query strings are not recorded? [#why-does-parameter-waste-say-query-strings-are-not-recorded]

If your logs do not record query strings, Server Log Monitoring cannot measure parameter waste. The tile says **Query strings are not recorded**.

## Related [#related]

- [Server Log Monitoring](/server-logs/overview) - how crawl waste and verification work.
- [Responses](/server-logs/responses) - the URLs that give bots errors or redirects.
- [Crawl vs results](/server-logs/crawl-vs-results) - crawl next to clicks, visits and your sitemap.
- [Domain settings](/domain/settings) - set up site sections.
- [Blocked pages](/robots-txt-monitoring/blocked-pages) - the pages your robots.txt keeps crawlers away from.
- [Sitemap Monitoring](/sitemap-monitoring/overview) - the URLs your sitemap lists.