# Server Log Monitoring: Overview

Server Log Monitoring reads the log lines your server writes for each request. It shows which search engines and AI crawlers visit your site and what your server answered. It also shows how much of their crawl goes to pages that earn nothing.

## What Server Log Monitoring shows [#what-server-log-monitoring-shows]

Your server already writes one line for each request that it answers. Server Log Monitoring reads those lines and tells you:

- Which bots visit your site, how often, and if Googlebot gets errors or redirects.
- How much of the crawl goes to broken, redirected, parameterized or robots.txt-blocked URLs. This is **crawl waste**.
- Which AI crawlers read your site, and if your server lets them in.
- Which visitors use the name of a real bot, such as Googlebot, but come from a different address.
- If the pages Googlebot crawls also earn impressions, clicks and visits in other VitalSentinel modules.

A **crawler** (also called a bot) is a program that fetches web pages automatically. Search engines use crawlers to find and index pages. **Crawl budget** is the number of pages a search engine is willing to fetch from your site in a period. You want that budget spent on pages that matter.

### What Server Log Monitoring collects [#what-server-log-monitoring-collects]

Server Log Monitoring stores five kinds of automated traffic. The **Excluded traffic** card on the Settings page lists them as **What we collect**:

| Kind | What it is |
|------|------------|
| **Search engines** | Google, Bing and the other crawlers that decide what gets indexed. |
| **AI training crawlers** | Crawlers collecting pages to train AI models. |
| **AI search crawlers** | Crawlers building the indexes behind AI search answers. |
| **AI assistants** | Assistants fetching a page because someone asked about it. |
| **AI agents** | Automated agents browsing your site on a person's behalf. |

All other automated traffic (SEO tools, social link previews, feed readers, archives, ad verifiers, security scanners, uptime checkers and developer tools) is dropped on arrival. Server Log Monitoring counts the dropped traffic, so you can see how much it left out and why. Human visitors are never collected by this module. Query-parameter names are stored, but never their values.

## Bot verification: verified, cannot be checked and fake [#bot-verification-verified-cannot-be-checked-and-fake]

Anyone can put "Googlebot" in a request. Server Log Monitoring checks the address of each request against the IP ranges or reverse-DNS names that the bot's company publishes. A **reverse-DNS name** is the host name that an IP address points back to.

| Result | Meaning |
|--------|---------|
| **Verified** | The request came from an address that the bot really uses. The check uses the IP range, reverse DNS or a signature. |
| **Cannot be checked** (filter value **Unverified**) | The bot publishes no IP ranges or reverse-DNS names, so nothing can confirm it. This is not proof of fraud. |
| **Fake** (filter value **Spoofed**) | The request used the name of a known bot, but came from an address that the bot does not use. |

Several AI crawlers, including crawlers that Meta runs, publish no ranges. Server Log Monitoring shows **Not measured** for their verified share, not 0%. The tooltip says: "This bot publishes no IP ranges or reverse-DNS suffixes, so its traffic cannot be confirmed either way."

### Where Server Log Monitoring uses verified Googlebot only [#where-server-log-monitoring-uses-verified-googlebot-only]

The Googlebot figures across Server Log Monitoring count verified Googlebot only. These include the Googlebot tabs on the Overview, the **Googlebot** column on Crawled pages, the Crawl map and the Crawl vs results tabs. The Bots page counts every hit that carries a bot's name, fake hits included. For this reason, the Bots page can show a higher Googlebot figure.

## AI crawlers in Server Log Monitoring [#ai-crawlers-in-server-log-monitoring]

Server Log Monitoring sorts AI bots by what they do with your pages. The labels are the same on the Bots page and the Crawl vs results page:

| Label | What the bot does | Examples |
|-------|-------------------|----------|
| **Trains on you** (category **AI crawler**) | Copies pages to train a model. | GPTBot, ClaudeBot |
| **Indexes for answers** (category **AI search**) | Reads pages to find and cite them. | OAI-SearchBot, PerplexityBot |
| **Fetched live for a user** (category **AI assistant**) | Opens a page while a person waits for an answer. | ChatGPT-User, Claude-User |

The category **AI agent** is an automated agent that browses your site for a person. To see all AI bots, open **Bots** and select the **AI crawlers** tab. See [Bots](/server-logs/bots).

## How fast data arrives in Server Log Monitoring [#how-fast-data-arrives-in-server-log-monitoring]

The delay depends on the log source:

- **Cloudflare Worker**: the setup screen says "About 5 minutes with the Cloudflare Worker."
- **File sources** (SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage): the data arrives after the next scheduled run. A file source runs every 12 to 24 hours.
- **Manual upload**: the data arrives after you upload a file.

The **Crawled pages** list comes from a nightly build. It is usually 1 to 2 days behind today. Most charts count the days in your own time zone. When a chart can only count UTC days, its **?** says: **Days are UTC days: the log totals are counted by UTC date and cannot be shown in your time zone.**

### How far back Server Log Monitoring reads [#how-far-back-server-log-monitoring-reads]

Select the **?** next to the page title to see the limits of the page. It shows **Reads at most** (the longest date range the page reads at a time) and **Kept for** (how long the data is kept). Most pages read at most 366 days at a time. VitalSentinel keeps Server Log Monitoring data for 12 months (90 days on the Free plan). See [Plans and limits](/workspace/plans-and-limits).

If a page cannot show your full date range, an amber notice says so. For example: "Showing Jan 1, 2026 to Jun 30, 2026, not the full range you picked."

## Plans and the monthly hit allowance [#plans-and-the-monthly-hit-allowance]

Server Log Monitoring is on every plan. The setup screen says one of these:

- **On every plan. The Cloudflare Worker collects in real time; file sources collect on a schedule.**
- **On every plan. Free collects through the Cloudflare Worker; file sources (SFTP/FTPS, cloud storage, upload) need a paid plan.**

Each plan has a **monthly hit allowance**. This is the number of crawler hits that the workspace can store in one billing cycle. All domains in the workspace share it. The size of the allowance depends on your plan, see [Plans and limits](/workspace/plans-and-limits). The Settings page shows how much of the allowance you used, see [Settings](/server-logs/settings#monthly-hit-allowance).

## Before you start [#before-you-start]

- **Verify the domain.** VitalSentinel monitors nothing until you verify that you own the domain. See [Verify ownership](/getting-started/domain-setup/verify-ownership).
- **Connect a log source.** Each domain collects from one log source at a time. See [Connect server logs](/connect/server-logs).
- **Roles.** Workspace owners and admins can connect a source and change the settings. Members can read every report. The Settings page is for owners and admins only.
- **Where to find it.** Open a domain. In the sidebar, under **Crawling & availability**, select **Server Log Monitoring**. Until a source delivers its first request, the entry opens the setup screen. After that, it expands into the entries below.

### The Server Log Monitoring sidebar entries [#the-server-log-monitoring-sidebar-entries]

| Sidebar entry | What it shows | Page |
|---------------|---------------|------|
| **Overview** | How Googlebot is answered, bot traffic over time, and bots that need attention. | This page |
| **Bots** | Tabs **All bots**, **AI crawlers** and **Fake bots**. | [Bots](/server-logs/bots) |
| **Pages** | Tabs **Crawled pages**, **Crawl budget** and **Crawl map**. | [Pages](/server-logs/pages) |
| **Responses** | Tabs **All responses** and **Gave up waiting**. | [Responses](/server-logs/responses) |
| **Crawl vs results** | Your crawl next to [Google Search Console](/google-search-console/overview), [Web Analytics](/web-analytics/overview), [RUM Monitoring](/rum-monitoring/overview), [Sitemap Monitoring](/sitemap-monitoring/overview) and [Page Monitoring](/page-monitoring/overview). | [Crawl vs results](/server-logs/crawl-vs-results) |
| **Explore** | Individual bot requests, grouped the way you choose. | [Explore](/server-logs/explore) |
| **Settings** | Log source, monthly hit allowance, excluded traffic and sections. | [Settings](/server-logs/settings) |

## Server Log Monitoring setup states [#server-log-monitoring-setup-states]

### Not set up yet [#not-set-up-yet]

If the domain has no log source, every Server Log Monitoring page shows a setup screen with the chip **Not set up yet**. The Overview headline is "Connect your server logs to see which bots crawl your site". Owners and admins click **Connect a log source**. Members see **Not connected – ask an owner or admin**. The **Settings** entry is always open, so you can connect a source there too. If the domain is not verified yet, the chip says **Waits for verification**.

### Waiting for the first data [#waiting-for-the-first-data]

After you connect a source, the page says **Your log source is connected, and nothing has reached us from it yet.** It shows **Waiting for the first data** and the **Log source** card. For the Cloudflare Worker, the line says **Reports fill in as your server logs requests.** For a file source, it says **Log files are collected on the schedule shown below.**

### Your log source stopped delivering [#your-log-source-stopped-delivering]

If a source delivered data before and then stopped, a banner shows above the reports: **Your log source stopped delivering**. The text says "No new requests arrived recently. The source may be disconnected, or the credentials changed." It also says "Your history is kept." Click **Fix the log source** to open the Settings page.

### Server Log Monitoring has reached this month's limit [#server-log-monitoring-has-reached-this-months-limit]

If the workspace used its monthly hit allowance, a banner says **Server Log Monitoring has reached this month's limit**. The text says "Data collected so far stays. New data is not stored until the limit resets." The banner shows a usage meter and the reset date. Click **See your log sources** to open the Settings page.

### We could not check what this domain collects from [#we-could-not-check-what-this-domain-collects-from]

If VitalSentinel cannot load the state of the log source, the page shows **We could not check what this domain collects from.** This does not mean that you have no log source. Use the retry button on the card to try again.

## The Overview page [#the-overview-page]

The Overview page shows how your server answers Googlebot and the busiest problems with other bots. From top to bottom, it shows a verdict, an issues banner, the **Bot traffic** card and the **Who needs attention** card.

### Googlebot verdict [#googlebot-verdict]

The verdict is one sentence about verified Googlebot visits in the date range. Your filters do not apply to it.

| Verdict | When it shows |
|---------|---------------|
| "Googlebot got an error on X% of its visits" (red) | Errors (4xx or 5xx) are 5% or more of visits. |
| "Googlebot was sent through a redirect on X% of its visits" (amber) | Redirects are 10% or more of visits. |
| "Googlebot got a normal answer on X% of its visits" (green) | Neither limit is reached. |
| "Googlebot visited too rarely to judge how it was answered" | Fewer than 100 verified visits. |

If verified Googlebot made no request, the verdict does not show. If filters are active, the line under the verdict says **Not filtered**.

### Issues banner on the Overview [#issues-banner-on-the-overview]

Site Health Monitoring keeps the issues that your logs find, such as Googlebot errors or fake bots. If there are open issues, a banner says, for example, "3 open issues from Server Log Monitoring". Click it to open the issues in Site Health Monitoring. See [Site Health Monitoring](/site-health/overview).

### Bot traffic card [#bot-traffic-card]

The **Bot traffic** card has four tabs. Each tab shows a key number and a chart.

| Tab | What it shows |
|-----|---------------|
| **Bot hits** | Every bot hit in the date range. The chart stacks the hits in four groups: **Search engines**, **AI**, **Everything else** and **Unidentified**. |
| **Googlebot hits** | Verified Googlebot hits. Shows **No verified Googlebot hits** if there are none. |
| **Googlebot response time** | How long your server took to start its answer to verified Googlebot. |
| **Crawl waste** | The share of bot hits on URLs that earn nothing back. |

With comparison on, each tab shows the change against the earlier period, and the chart shows that period as a dashed line.

### Bot traffic chart controls [#bot-traffic-chart-controls]

Above the chart, select **Hour**, **Day**, **Week** or **Month** to group the hits. **Relative** shows each group as a share of all hits. It works on the **Bot hits** tab only. The Googlebot tabs have no hour view. The **Crawl waste** tab has a day view only.

The **Bot hits** chart shows markers. **robots.txt changed** comes from [Robots.txt Monitoring](/robots-txt-monitoring/overview). **Downtime** comes from [Uptime Monitoring](/uptime-monitoring/overview). The two Googlebot tabs show other markers: changes that [Page Monitoring](/page-monitoring/overview) found, and critical or high problems from Site Health Monitoring. Use the markers to see if a change in crawling happened at the same time.

On the **Googlebot response time** tab, a line under the chart shows the share **answered from cache** and **answered by your server**, with the time of each. If no source reports cache results, the line says "Server Log Monitoring has no log source that reports cache results".

### Crawl waste on the Overview [#crawl-waste-on-the-overview]

Crawl waste is the share of bot hits that went to a URL earning nothing back. Such a URL gives a redirect or an error, has query parameters, or is disallowed by your robots.txt. A **304 Not Modified** is a healthy answer and does not count as waste.

| Tone | Range |
|------|-------|
| **Fine** | Under 10% |
| **Worth a look** | 10% to 30% |
| **Costly** | 30% and over |

With an exact **Status code** filter, the tab shows **Not available with a status-code filter**. To see where the waste goes, open **Pages** and select the **Crawl budget** tab. See [Pages](/server-logs/pages#crawl-budget-tab).

### Who needs attention [#who-needs-attention]

The **Who needs attention** card lists the bots that your server treats badly. The columns are **Bot**, **What is wrong** and **Real hits**. **Real hits** are hits that are not fake.

| Chip | When it shows |
|------|---------------|
| "X% ended in an error" | 5% or more of the real hits got a 4xx or 5xx answer (at least 100 hits). |
| "X% answered 403" | An AI bot got **403 Forbidden** on 50% or more of its real hits. |
| "N fake hits – X% served" | At least 20 fake hits used the bot's name, and 50% or more of them got a normal page. |

Click a bot name to filter the page to that bot. If fake hits occurred, the header shows a **Fake bots** button that opens the **Fake bots** tab. If no bot has a problem, the card says **No bot needs attention**.

## Filters in Server Log Monitoring [#filters-in-server-log-monitoring]

The filter bar at the bottom of the page sets the date range and the filters. It stays the same when you move between pages. For how the filter bar works, see [Using the app](/getting-started/using-the-app).

| Filter | Narrows to |
|--------|-----------|
| **Bot** | One or more bots. |
| **Bot category** | Categories such as **Search engine** or **AI crawler**. |
| **Vendor** | The company behind the bot, for example Google or OpenAI. |
| **AI crawler** | AI bots only, or no AI bots. |
| **Status code** / **Status class** | Exact codes, or 2xx, 3xx, 4xx and 5xx. |
| **Verification** | **Verified (any method)**, **Verified (IP range)**, **Verified (reverse DNS)**, **Verified (signature)**, **Unverified**, **Spoofed**. |
| **Section** / **URL** | One of your site sections, or URLs that contain a text. |
| **Query parameter** / **Robots.txt** | URLs with a query-parameter name, or URLs that robots.txt **Disallowed** or **Allowed**. |
| **Network (ASN)** | On the Explore page only. |
| **Content type** / **Response time (ms)** | On the Explore page and the **Gave up waiting** tab only. |

### When a filter does not apply [#when-a-filter-does-not-apply]

Not every page can use every filter. If a page cannot use a filter, a notice says **Not applied here**, with the name of the filter. Click **Remove** to remove it. On the Overview, Bots, Fake bots and All responses pages, **Verification** cannot be combined with an exact **Status code**. A card that ignores a filter shows a line such as "Not filtered by Bot."

## Common figures in Server Log Monitoring [#common-figures-in-server-log-monitoring]

- **Response time** is the typical [time to first byte (TTFB)](/help/glossary#ttfb-time-to-first-byte). **TTFB** is how long your server took to start its answer. Three out of four hits were this fast or faster.
- **Slow responses** describes the slowest hits: 19 out of 20 hits were this fast or faster.
- **Status mix** is a bar of **2xx Success**, **3xx Redirect**, **4xx Client error** and **5xx Server error**.

### Missing data is never shown as zero [#missing-data-is-never-shown-as-zero]

Days with no log data show as gaps in charts, not as zero. If the last part of a chart covers only part of a period, it is marked as partial. A figure that cannot be calculated shows **Not measured**, **Not observed** or **Not measurable**, with the reason in the tooltip.

## Troubleshooting [#troubleshooting]

### Why does Server Log Monitoring show no data? [#why-does-server-log-monitoring-show-no-data]

If the page shows **Not set up yet**, connect a log source. See [Connect server logs](/connect/server-logs). If the page shows **Waiting for the first data**, wait about 5 minutes for the Cloudflare Worker, or until the next scheduled run of a file source.

### Why did my Server Log Monitoring charts stop? [#why-did-my-server-log-monitoring-charts-stop]

If the banner **Your log source stopped delivering** shows, click **Fix the log source** and check the source on the Settings page. If the banner **Server Log Monitoring has reached this month's limit** shows, the workspace used its monthly hit allowance. Collection starts again when the limit resets, or a few minutes after you upgrade.

### Why is the verified share of a bot "Not measured"? [#why-is-the-verified-share-of-a-bot-not-measured]

If a bot publishes no IP ranges or reverse-DNS names, Server Log Monitoring cannot confirm its hits. It shows **Not measured** for that bot. This is not a sign of fake traffic.

### Why is the Googlebot count on the Bots page higher than on the Overview? [#why-is-the-googlebot-count-on-the-bots-page-higher-than-on-the-overview]

The Overview counts verified Googlebot hits only. The Bots page counts every hit that carries the Googlebot name, fake hits included. Open the **Fake bots** tab to see the fake hits.

### Why does a page say "Not applied here"? [#why-does-a-page-say-not-applied-here]

If the page cannot use one of your filters, it ignores the filter and shows **Not applied here**. Click **Remove** to remove the filter, or open a page that supports it, such as [Explore](/server-logs/explore).

## Related [#related]

- [Connect server logs](/connect/server-logs) - set up the Cloudflare Worker or a file source.
- [Bots](/server-logs/bots) - every bot, AI crawlers and fake bots.
- [Pages](/server-logs/pages) - crawled pages, crawl budget and the crawl map.
- [Robots.txt Monitoring](/robots-txt-monitoring/overview) - the rules that let crawlers in or keep them out.
- [Uptime Monitoring](/uptime-monitoring/overview) - outages that bots can meet.
- [Glossary](/help/glossary#crawlers-and-bots) - crawl budget, crawl waste and bot verification.