Server Log Monitoring
See which search engine and AI crawlers hit your server, how your server answered them, which bots are fake, and how much of your crawl budget is wasted.
Your server already writes a line for every request it answers. Server Log Monitoring reads those lines and tells you what crawled your site, what it was given, and what that cost you.
66.249.66.1 - - [14/Sep/2026:08:12:44 +0000] "GET /blog/technical-seo HTTP/1.1" 200 18432 "-" "Googlebot/2.1"From a line like this one, VitalSentinel works out:
- Which bot it was, checked against the IP ranges that crawler publishes, so a request only claiming to be Googlebot is named as fake.
- Which URL your crawl budget went to, and whether that URL earns anything back.
- What your server answered and how long it took, per bot and over time.
What it answers
- Is Googlebot crawling my site, how often, and is it getting errors or redirects?
- How much of my crawl budget goes to broken, redirected, parameterized or robots.txt-disallowed URLs?
- Which AI crawlers (training crawlers, AI search, AI assistants and agents) read my site, and do they respect robots.txt?
- Who is pretending to be Googlebot or another known bot?
- Which pages with search impressions has Google stopped re-crawling?
- Which sitemap URLs does Googlebot ignore, and which crawled pages are missing from the sitemap?
How it works
- Connect a source. Either the Cloudflare Worker, which collects in real time, or a log-file source (SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage or a one-off upload), which is collected on a schedule. A domain collects from one source at a time.
- Every request gets classified. Each hit is matched to VitalSentinel's bot registry, verified where the crawler publishes its IP ranges, and grouped into the sections of your site that you define.
- Reports fill as requests arrive. The Cloudflare Worker starts from the moment you connect it. A log-file source can also import past files, so your reports can start with history already in them.
What gets collected
Server Log Monitoring reports on how your site is discovered, so it stores five kinds of automated traffic and drops everything else before it is written down:
| Kind | What it is |
|---|---|
| Search engines | Google, Bing and the other crawlers that decide what gets indexed |
| AI training crawlers | Crawlers collecting pages to train AI models |
| AI search crawlers | Crawlers building the indexes behind AI search answers |
| AI assistants | Assistants fetching a page because someone asked about it |
| AI agents | Automated agents browsing your site on a person's behalf |
Everything else (SEO tools, social link previews, feed readers, archives, ad verifiers, security scanners, uptime and page-speed checkers) is dropped on arrival and never stored. So is any automated client that gives no name, URL or contact. Human visitors are never collected by this module.
Dropped traffic is still counted, so you can see how much was excluded and why. See Excluded traffic.
Query-parameter names are stored, never their values. The IP addresses of unverified and spoofed requests are never stored; those requests are grouped by network (AS number) instead.
Bot verification
Anyone can put "Googlebot" in a User-Agent header. VitalSentinel checks each request's IP address against the IP ranges or reverse-DNS suffixes the bot's vendor publishes, and gives every hit one of three verdicts:
| Verdict | Meaning |
|---|---|
| Verified | The request came from an address the bot really uses (by IP range, reverse DNS or signature). |
| Unverified | It could not be confirmed either way, usually because the bot publishes no IP ranges or reverse-DNS suffixes to check against. Not the same as spoofed. |
| Spoofed | It claimed to be a known bot but came from an address that bot doesn't use. Someone is using the bot's name without being the bot. |
A low verified share is not proof of fraud. Several AI crawlers, including ones Meta runs, publish neither IP ranges nor reverse-DNS suffixes, so their traffic can never be confirmed. Those bots show Not measured instead of 0%.
Googlebot-specific figures across the module (the Googlebot tiles, crawl map, stale-crawl and sitemap reports) count verified Googlebot only.
Missing is never shown as zero
When something was not measured, the module says so rather than drawing a zero:
- Days with no log data at all are drawn as gaps in charts, not as zero.
- The last bucket of a chart is shaded when it covers only part of a period. Its hits are a floor, not a fall.
- Figures that cannot be calculated read Not measured, Not observed or Not measurable, with a reason in the tooltip.
- When a report can only cover part of the date range you picked, an amber notice names the shorter window and why.
Plans and limits
Server Log Monitoring is available on every plan. Each plan has a monthly hit allowance: the number of crawler hits the workspace can store per billing cycle, shared by all of its domains.
| Free | Starter | Lite | Standard | Advanced | |
|---|---|---|---|---|---|
| Server log hits / month | 500K | 3M | 10M | 50M | 150M |
| Log sources | Cloudflare Worker only | Cloudflare, SFTP and others | Cloudflare, SFTP and others | Cloudflare, SFTP and others | Cloudflare, SFTP and others |
- Only stored hits count. Traffic the collection policy excludes does not use up the allowance.
- At the limit, collection stops until the next billing cycle starts (or within a few minutes of an upgrade). Nothing already collected is deleted, and new hits are counted as excluded rather than dropped silently.
- On the Free plan, log-file sources are not available. A single log-file job can backfill a year of history, which would spend a whole month's allowance at once. The Cloudflare Worker collects at the rate your site is actually crawled and is available on every plan.
See Monthly hit allowance for how usage is shown.
How far back the data goes
| Data | How far back |
|---|---|
| Overview, Bots, Pages, Responses and most reports | Up to 12 months |
| Explore (individual requests) | Up to 12 months. Every error and redirect is kept in full; successful (2xx) responses are kept as a 1% sample |
| Per-page, per-bot request detail (the page drill-down) | About 7 days |
| Intelligence reports | Up to 180 days per view |
Retention is the same on every plan. See Data & Privacy.
Where to find it
Open a domain and select Server Log Monitoring in the sidebar. Until a source has delivered its first request, the entry opens the setup screen. After that it expands into the module's reports:
| Group | Pages |
|---|---|
| Overview | The module's dashboard |
| Bots | All bots, AI crawlers, Security |
| Pages | Crawled pages, Sections, Crawl budget, Crawl map |
| Responses | Status codes over time, bot-by-status matrix, URLs to fix |
| Intelligence | Search Console, Web Analytics, RUM, Sitemap |
| Explore | Individual requests, grouped and filtered |
| Settings | Log source, hit allowance, collection policy, sections |
Server log data also shows up elsewhere:
- Indexing Monitoring and Robots.txt Monitoring gain cards built from your logs.
- 15 server log alerts warn you about crawl problems.
- The public API and the MCP server can read this data too.
Next steps
Settings
Set the Page Monitoring defaults for new pages, default labels, the weekly top pages list, and the kinds of change that VitalSentinel records but never flags.
Connecting a Log Source
Connect server logs to VitalSentinel with the Cloudflare Worker for real-time collection, or with SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage or a file upload.
