Server Log Settings
Manage your log source, track the monthly server log hit allowance, see what traffic is excluded and why, exclude your own agents, and group URLs into sections.
Open Settings in the Server Log Monitoring sidebar. Settings is always available, even before a source is connected. It has four parts:
- Log source: connect, check, reconnect or disconnect the domain's source
- Monthly hit allowance
- What gets collected: excluded traffic and collection settings
- Sections
The last three appear once a source exists.
Monthly hit allowance
Hits stored this billing cycle shows how many crawler hits the workspace has stored, against the plan's limit, with a progress bar:
- Green below 75%
- Amber from 75%
- Red from 90%
The allowance is shared by every domain in the workspace and counts the hits actually stored. Traffic your collection settings exclude does not use it up. See Plans and limits for each plan's allowance.
The same meter appears as Server Log Hits (monthly) in the workspace's Usage Overview. As with your other usage limits, you receive a warning as you approach the allowance.
When the allowance runs out
At 100%, Collection has stopped for this billing cycle:
- New crawler hits are no longer kept. They are counted as excluded with the reason Over your monthly hit limit, so the excluded-traffic numbers keep moving while your reports do not.
- Nothing already collected is deleted.
- Collection resumes when the next billing cycle starts, or within a few minutes of an upgrade.
A gap caused by a spent allowance is not a quiet site. If your reports flatline, check this card before assuming crawlers stopped visiting.
What gets collected
Excluded traffic
What was left out of collection, why, and how much of it VitalSentinel caused itself.
| Tile | What it shows |
|---|---|
| Excluded requests | Requests not stored, as a share of all automated traffic to the domain. Nothing is discarded without being counted here |
| Attributed to VitalSentinel | Traffic from VitalSentinel's own monitoring agents and scheduled Lighthouse runs |
| Third-party tooling | Tooling, scanners and your own excluded-agent list |
Attributed to VitalSentinel is an upper bound: it is matched by the page being one VitalSentinel runs Lighthouse against, so a Lighthouse run you started yourself on a monitored page is counted too. VitalSentinel's own Lighthouse hits are stored so you can see their cost, but they do not count against your hit allowance.
Click Show what was excluded, and why for the breakdown:
- Where it was excluded: Refused at the edge, Excluded on arrival, or Excluded while parsing a log file.
- Why it was excluded: for example SEO and backlink tools, Social media link previews, Feed and newsletter readers, Web archives, Ad verification crawlers, Security and vulnerability scanners, Uptime and availability checkers, Performance and dev tooling, Gave no name, address or contact, Your own excluded-agent list, VitalSentinel's own agents, and Over your monthly hit limit.
Requests refused by the Cloudflare Worker at the edge never reach VitalSentinel, so they are reported only as a total, with no page, bot or status code to break them down by.
Collection settings
Which kinds of bots are collected is fixed: search engines plus the four AI categories (AI training crawlers, AI search crawlers, AI assistants and AI agents). Everything else is dropped on arrival. There is no setting to bring other traffic into your charts or the crawl-waste figure. See What gets collected.
The one setting you control is Your own excluded agents: case-insensitive text matched against a bot's name and raw User-Agent. Use it for your own crawler, a staging health check or a load-test client that isn't already in VitalSentinel's registry.
- Type a pattern, for example
internal-healthcheck - Click Add pattern
- Click Save settings
By default, only VitalSentinel's own agents are excluded.
Sections
Sections group crawled URLs into named areas of your site (for example /blog/ or /products/), so every report can break traffic down by area: the Sections report, the Section filter and column, Crawl budget ROI and Crawl frequency by ranking position.
Sections apply when a report is read, not when traffic is collected. Adding, changing or deleting a rule regroups past traffic too, and every report updates immediately, including history.
Adding a rule
Click Add rule:
| Field | Notes |
|---|---|
| Match type | Prefix or Regex |
| Path pattern | For Prefix, any path starting with this, for example /blog/ (a trailing * means the same). For Regex, a regular expression matched against the path, for example ^/blog/\d{4}/ |
| Section name | The name shown in reports |
Changes save automatically.
Rule order
Your rules are checked in order, top to bottom, and the first pattern that matches a path wins. Use the arrows to reorder them. Hits that match no rule show as (unsectioned).
If a rule can never match because an earlier rule already claims every path it would, it is flagged with Never matches. Move it above the rule that shadows it.
Suggested sections
Suggested for this site proposes rules built from the folders bots actually crawled, plus patterns that are standard for your platform. Each suggestion shows its evidence (how many URLs and hits it would cover in the last 30 days). Click Add to accept one.
Suggestions appear within a day of log collection starting.
Related
Server Log Intelligence
Join server logs with Search Console, Web Analytics, RUM and your sitemap to find stale money pages, crawl-to-click drop-offs, orphan pages and slow pages Google crawls most.
Setting Up Alerts
Create alert rules for any metric, set the thresholds that matter, and get notified by email and in-app, with worked examples and best practices.
