Overview
Server Log Monitoring shows which search engine and AI crawlers fetch your pages, how your server answers them, and which bots are fake.
Server Log Monitoring reads the log lines your server writes for each request. It shows which search engines and AI crawlers visit your site and what your server answered. It also shows how much of their crawl goes to pages that earn nothing.
What Server Log Monitoring shows
Your server already writes one line for each request that it answers. Server Log Monitoring reads those lines and tells you:
- Which bots visit your site, how often, and if Googlebot gets errors or redirects.
- How much of the crawl goes to broken, redirected, parameterized or robots.txt-blocked URLs. This is crawl waste.
- Which AI crawlers read your site, and if your server lets them in.
- Which visitors use the name of a real bot, such as Googlebot, but come from a different address.
- If the pages Googlebot crawls also earn impressions, clicks and visits in other VitalSentinel modules.
A crawler (also called a bot) is a program that fetches web pages automatically. Search engines use crawlers to find and index pages. Crawl budget is the number of pages a search engine is willing to fetch from your site in a period. You want that budget spent on pages that matter.
What Server Log Monitoring collects
Server Log Monitoring stores five kinds of automated traffic. The Excluded traffic card on the Settings page lists them as What we collect:
| Kind | What it is |
|---|---|
| Search engines | Google, Bing and the other crawlers that decide what gets indexed. |
| AI training crawlers | Crawlers collecting pages to train AI models. |
| AI search crawlers | Crawlers building the indexes behind AI search answers. |
| AI assistants | Assistants fetching a page because someone asked about it. |
| AI agents | Automated agents browsing your site on a person's behalf. |
All other automated traffic (SEO tools, social link previews, feed readers, archives, ad verifiers, security scanners, uptime checkers and developer tools) is dropped on arrival. Server Log Monitoring counts the dropped traffic, so you can see how much it left out and why. Human visitors are never collected by this module. Query-parameter names are stored, but never their values.
Bot verification: verified, cannot be checked and fake
Anyone can put "Googlebot" in a request. Server Log Monitoring checks the address of each request against the IP ranges or reverse-DNS names that the bot's company publishes. A reverse-DNS name is the host name that an IP address points back to.
| Result | Meaning |
|---|---|
| Verified | The request came from an address that the bot really uses. The check uses the IP range, reverse DNS or a signature. |
| Cannot be checked (filter value Unverified) | The bot publishes no IP ranges or reverse-DNS names, so nothing can confirm it. This is not proof of fraud. |
| Fake (filter value Spoofed) | The request used the name of a known bot, but came from an address that the bot does not use. |
Several AI crawlers, including crawlers that Meta runs, publish no ranges. Server Log Monitoring shows Not measured for their verified share, not 0%. The tooltip says: "This bot publishes no IP ranges or reverse-DNS suffixes, so its traffic cannot be confirmed either way."
Where Server Log Monitoring uses verified Googlebot only
The Googlebot figures across Server Log Monitoring count verified Googlebot only. These include the Googlebot tabs on the Overview, the Googlebot column on Crawled pages, the Crawl map and the Crawl vs results tabs. The Bots page counts every hit that carries a bot's name, fake hits included. For this reason, the Bots page can show a higher Googlebot figure.
AI crawlers in Server Log Monitoring
Server Log Monitoring sorts AI bots by what they do with your pages. The labels are the same on the Bots page and the Crawl vs results page:
| Label | What the bot does | Examples |
|---|---|---|
| Trains on you (category AI crawler) | Copies pages to train a model. | GPTBot, ClaudeBot |
| Indexes for answers (category AI search) | Reads pages to find and cite them. | OAI-SearchBot, PerplexityBot |
| Fetched live for a user (category AI assistant) | Opens a page while a person waits for an answer. | ChatGPT-User, Claude-User |
The category AI agent is an automated agent that browses your site for a person. To see all AI bots, open Bots and select the AI crawlers tab. See Bots.
How fast data arrives in Server Log Monitoring
The delay depends on the log source:
- Cloudflare Worker: the setup screen says "About 5 minutes with the Cloudflare Worker."
- File sources (SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage): the data arrives after the next scheduled run. A file source runs every 12 to 24 hours.
- Manual upload: the data arrives after you upload a file.
The Crawled pages list comes from a nightly build. It is usually 1 to 2 days behind today. Most charts count the days in your own time zone. When a chart can only count UTC days, its ? says: Days are UTC days: the log totals are counted by UTC date and cannot be shown in your time zone.
How far back Server Log Monitoring reads
Select the ? next to the page title to see the limits of the page. It shows Reads at most (the longest date range the page reads at a time) and Kept for (how long the data is kept). Most pages read at most 366 days at a time. VitalSentinel keeps Server Log Monitoring data for 12 months (90 days on the Free plan). See Plans and limits.
If a page cannot show your full date range, an amber notice says so. For example: "Showing Jan 1, 2026 to Jun 30, 2026, not the full range you picked."
Plans and the monthly hit allowance
Server Log Monitoring is on every plan. The setup screen says one of these:
- On every plan. The Cloudflare Worker collects in real time; file sources collect on a schedule.
- On every plan. Free collects through the Cloudflare Worker; file sources (SFTP/FTPS, cloud storage, upload) need a paid plan.
Each plan has a monthly hit allowance. This is the number of crawler hits that the workspace can store in one billing cycle. All domains in the workspace share it. The size of the allowance depends on your plan, see Plans and limits. The Settings page shows how much of the allowance you used, see Settings.
Before you start
- Verify the domain. VitalSentinel monitors nothing until you verify that you own the domain. See Verify ownership.
- Connect a log source. Each domain collects from one log source at a time. See Connect server logs.
- Roles. Workspace owners and admins can connect a source and change the settings. Members can read every report. The Settings page is for owners and admins only.
- Where to find it. Open a domain. In the sidebar, under Crawling & availability, select Server Log Monitoring. Until a source delivers its first request, the entry opens the setup screen. After that, it expands into the entries below.
The Server Log Monitoring sidebar entries
| Sidebar entry | What it shows | Page |
|---|---|---|
| Overview | How Googlebot is answered, bot traffic over time, and bots that need attention. | This page |
| Bots | Tabs All bots, AI crawlers and Fake bots. | Bots |
| Pages | Tabs Crawled pages, Crawl budget and Crawl map. | Pages |
| Responses | Tabs All responses and Gave up waiting. | Responses |
| Crawl vs results | Your crawl next to Google Search Console, Web Analytics, RUM Monitoring, Sitemap Monitoring and Page Monitoring. | Crawl vs results |
| Explore | Individual bot requests, grouped the way you choose. | Explore |
| Settings | Log source, monthly hit allowance, excluded traffic and sections. | Settings |
Server Log Monitoring setup states
Not set up yet
If the domain has no log source, every Server Log Monitoring page shows a setup screen with the chip Not set up yet. The Overview headline is "Connect your server logs to see which bots crawl your site". Owners and admins click Connect a log source. Members see Not connected – ask an owner or admin. The Settings entry is always open, so you can connect a source there too. If the domain is not verified yet, the chip says Waits for verification.
Waiting for the first data
After you connect a source, the page says Your log source is connected, and nothing has reached us from it yet. It shows Waiting for the first data and the Log source card. For the Cloudflare Worker, the line says Reports fill in as your server logs requests. For a file source, it says Log files are collected on the schedule shown below.
Your log source stopped delivering
If a source delivered data before and then stopped, a banner shows above the reports: Your log source stopped delivering. The text says "No new requests arrived recently. The source may be disconnected, or the credentials changed." It also says "Your history is kept." Click Fix the log source to open the Settings page.
Server Log Monitoring has reached this month's limit
If the workspace used its monthly hit allowance, a banner says Server Log Monitoring has reached this month's limit. The text says "Data collected so far stays. New data is not stored until the limit resets." The banner shows a usage meter and the reset date. Click See your log sources to open the Settings page.
We could not check what this domain collects from
If VitalSentinel cannot load the state of the log source, the page shows We could not check what this domain collects from. This does not mean that you have no log source. Use the retry button on the card to try again.
The Overview page
The Overview page shows how your server answers Googlebot and the busiest problems with other bots. From top to bottom, it shows a verdict, an issues banner, the Bot traffic card and the Who needs attention card.
Googlebot verdict
The verdict is one sentence about verified Googlebot visits in the date range. Your filters do not apply to it.
| Verdict | When it shows |
|---|---|
| "Googlebot got an error on X% of its visits" (red) | Errors (4xx or 5xx) are 5% or more of visits. |
| "Googlebot was sent through a redirect on X% of its visits" (amber) | Redirects are 10% or more of visits. |
| "Googlebot got a normal answer on X% of its visits" (green) | Neither limit is reached. |
| "Googlebot visited too rarely to judge how it was answered" | Fewer than 100 verified visits. |
If verified Googlebot made no request, the verdict does not show. If filters are active, the line under the verdict says Not filtered.
Issues banner on the Overview
Site Health Monitoring keeps the issues that your logs find, such as Googlebot errors or fake bots. If there are open issues, a banner says, for example, "3 open issues from Server Log Monitoring". Click it to open the issues in Site Health Monitoring. See Site Health Monitoring.
Bot traffic card
The Bot traffic card has four tabs. Each tab shows a key number and a chart.
| Tab | What it shows |
|---|---|
| Bot hits | Every bot hit in the date range. The chart stacks the hits in four groups: Search engines, AI, Everything else and Unidentified. |
| Googlebot hits | Verified Googlebot hits. Shows No verified Googlebot hits if there are none. |
| Googlebot response time | How long your server took to start its answer to verified Googlebot. |
| Crawl waste | The share of bot hits on URLs that earn nothing back. |
With comparison on, each tab shows the change against the earlier period, and the chart shows that period as a dashed line.
Bot traffic chart controls
Above the chart, select Hour, Day, Week or Month to group the hits. Relative shows each group as a share of all hits. It works on the Bot hits tab only. The Googlebot tabs have no hour view. The Crawl waste tab has a day view only.
The Bot hits chart shows markers. robots.txt changed comes from Robots.txt Monitoring. Downtime comes from Uptime Monitoring. The two Googlebot tabs show other markers: changes that Page Monitoring found, and critical or high problems from Site Health Monitoring. Use the markers to see if a change in crawling happened at the same time.
On the Googlebot response time tab, a line under the chart shows the share answered from cache and answered by your server, with the time of each. If no source reports cache results, the line says "Server Log Monitoring has no log source that reports cache results".
Crawl waste on the Overview
Crawl waste is the share of bot hits that went to a URL earning nothing back. Such a URL gives a redirect or an error, has query parameters, or is disallowed by your robots.txt. A 304 Not Modified is a healthy answer and does not count as waste.
| Tone | Range |
|---|---|
| Fine | Under 10% |
| Worth a look | 10% to 30% |
| Costly | 30% and over |
With an exact Status code filter, the tab shows Not available with a status-code filter. To see where the waste goes, open Pages and select the Crawl budget tab. See Pages.
Who needs attention
The Who needs attention card lists the bots that your server treats badly. The columns are Bot, What is wrong and Real hits. Real hits are hits that are not fake.
| Chip | When it shows |
|---|---|
| "X% ended in an error" | 5% or more of the real hits got a 4xx or 5xx answer (at least 100 hits). |
| "X% answered 403" | An AI bot got 403 Forbidden on 50% or more of its real hits. |
| "N fake hits – X% served" | At least 20 fake hits used the bot's name, and 50% or more of them got a normal page. |
Click a bot name to filter the page to that bot. If fake hits occurred, the header shows a Fake bots button that opens the Fake bots tab. If no bot has a problem, the card says No bot needs attention.
Filters in Server Log Monitoring
The filter bar at the bottom of the page sets the date range and the filters. It stays the same when you move between pages. For how the filter bar works, see Using the app.
| Filter | Narrows to |
|---|---|
| Bot | One or more bots. |
| Bot category | Categories such as Search engine or AI crawler. |
| Vendor | The company behind the bot, for example Google or OpenAI. |
| AI crawler | AI bots only, or no AI bots. |
| Status code / Status class | Exact codes, or 2xx, 3xx, 4xx and 5xx. |
| Verification | Verified (any method), Verified (IP range), Verified (reverse DNS), Verified (signature), Unverified, Spoofed. |
| Section / URL | One of your site sections, or URLs that contain a text. |
| Query parameter / Robots.txt | URLs with a query-parameter name, or URLs that robots.txt Disallowed or Allowed. |
| Network (ASN) | On the Explore page only. |
| Content type / Response time (ms) | On the Explore page and the Gave up waiting tab only. |
When a filter does not apply
Not every page can use every filter. If a page cannot use a filter, a notice says Not applied here, with the name of the filter. Click Remove to remove it. On the Overview, Bots, Fake bots and All responses pages, Verification cannot be combined with an exact Status code. A card that ignores a filter shows a line such as "Not filtered by Bot."
Common figures in Server Log Monitoring
- Response time is the typical time to first byte (TTFB). TTFB is how long your server took to start its answer. Three out of four hits were this fast or faster.
- Slow responses describes the slowest hits: 19 out of 20 hits were this fast or faster.
- Status mix is a bar of 2xx Success, 3xx Redirect, 4xx Client error and 5xx Server error.
Missing data is never shown as zero
Days with no log data show as gaps in charts, not as zero. If the last part of a chart covers only part of a period, it is marked as partial. A figure that cannot be calculated shows Not measured, Not observed or Not measurable, with the reason in the tooltip.
Troubleshooting
Why does Server Log Monitoring show no data?
If the page shows Not set up yet, connect a log source. See Connect server logs. If the page shows Waiting for the first data, wait about 5 minutes for the Cloudflare Worker, or until the next scheduled run of a file source.
Why did my Server Log Monitoring charts stop?
If the banner Your log source stopped delivering shows, click Fix the log source and check the source on the Settings page. If the banner Server Log Monitoring has reached this month's limit shows, the workspace used its monthly hit allowance. Collection starts again when the limit resets, or a few minutes after you upgrade.
Why is the verified share of a bot "Not measured"?
If a bot publishes no IP ranges or reverse-DNS names, Server Log Monitoring cannot confirm its hits. It shows Not measured for that bot. This is not a sign of fake traffic.
Why is the Googlebot count on the Bots page higher than on the Overview?
The Overview counts verified Googlebot hits only. The Bots page counts every hit that carries the Googlebot name, fake hits included. Open the Fake bots tab to see the fake hits.
Why does a page say "Not applied here"?
If the page cannot use one of your filters, it ignores the filter and shows Not applied here. Click Remove to remove the filter, or open a page that supports it, such as Explore.
Related
- Connect server logs - set up the Cloudflare Worker or a file source.
- Bots - every bot, AI crawlers and fake bots.
- Pages - crawled pages, crawl budget and the crawl map.
- Robots.txt Monitoring - the rules that let crawlers in or keep them out.
- Uptime Monitoring - outages that bots can meet.
- Glossary - crawl budget, crawl waste and bot verification.
Trends & Anomalies
The Web Analytics Trends & Anomalies page shows the days your traffic jumped or dropped, lasting level shifts, and pages that lose or gain readers.
Bots
The Bots page of Server Log Monitoring lists every bot that visits your site, the AI crawlers, and the visitors that fake a bot's name.
