Connecting a Log Source
Connect server logs to VitalSentinel with the Cloudflare Worker for real-time collection, or with SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage or a file upload.
Each domain collects from one log source. To connect it, open the domain, select Server Log Monitoring in the sidebar and click Connect a log source, or go to Settings inside the module.
Choosing a source
The Log source section offers two options. A domain collects from one source at a time, and you can change it later.
| Source | How it collects | Plans |
|---|---|---|
| Cloudflare Worker (Recommended) | Real time, through a Worker deployed on your own Cloudflare account. For sites that run on Cloudflare. | All plans |
| Your own server | SFTP, FTPS, Amazon S3, Cloudflare R2, Google Cloud Storage, or a one-off file upload, collected on a schedule (every 12 to 24 hours). Can import past log files. | Paid plans |
On the Free plan, Your own server is disabled and marked Not on your plan. See Plans and limits for why.
Cloudflare Worker
The Cloudflare Worker is deployed into your own Cloudflare account, on the zone of your domain. It forwards each request's log line to VitalSentinel as your server answers it. Your site keeps working if the Worker fails: the route fails open.
Before you start
Create a Cloudflare API token in Cloudflare's dashboard (My Profile → API Tokens → Create Token → Custom Token), scoped to exactly these three permissions:
| Scope | Permission |
|---|---|
| Account | Workers Scripts: Edit |
| Zone | Workers Routes: Edit |
| Zone | Zone: Read |
Your domain must be added as a zone on the Cloudflare account the token belongs to.
The four steps
The connect wizard shows Step N of 4:
- Connect an API token. Paste the token into Cloudflare API token and click Verify token. A token that is not active, or was pasted incompletely, is rejected with an explanation.
- Confirm the zone. VitalSentinel lists the Cloudflare zones on that account that match your domain. A single match is pre-selected. Click Use this zone.
- Review plan and cost. The wizard shows your Cloudflare plan and how many Workers the account already uses, then estimates what the Worker will cost (see below). If there is an estimate, tick I understand and accept this cost on my Cloudflare account and click Continue.
- Deploy the Worker. Click Deploy Worker. VitalSentinel deploys a Worker unique to your domain on the chosen zone, plus a route that sends the domain's traffic through it. No other Worker on the account is touched.
Nothing is saved until the deploy step succeeds. Every step re-sends the token rather than relying on a stored copy. After a successful deploy, the token is stored encrypted and only a prefix is ever shown back.
Cost on your Cloudflare bill
The Worker runs on your Cloudflare account, so Cloudflare charges you for it, not VitalSentinel. Cloudflare charges roughly $0.30 per million requests a Worker runs on.
- The wizard estimates the monthly cost from your zone's own traffic over the trailing 7 days. If the token cannot read zone analytics, it says so instead of guessing.
- Workers Free daily limit: on Cloudflare's Workers Free plan, Cloudflare stops running a Worker for the rest of the day once it crosses the daily request cap. Your site stays up, but log collection stops until the next day. The wizard warns you when your traffic is above that cap. Upgrade to Workers Paid, or use a log-file source, to avoid the gap.
- Workers per account: if the account is at or near its Workers limit, the wizard warns you. You can still continue. If the deploy fails, free up a script or upgrade Workers.
If you don't want this on your Cloudflare bill, connect an SFTP, S3, R2 or GCS log source instead: the same dashboards, on a daily rather than real-time cadence.
After connecting
Settings shows the Worker's status, a Refresh button and Integration details (deployment mode, Worker script, Worker version, last verified, last event received, consecutive failures).
| Status | Meaning |
|---|---|
| Pending | The Worker is deployed and VitalSentinel re-checks its route and token every few minutes. It turns green when the first request arrives. On a low-traffic site that can take hours or days. |
| Healthy | Verified bot traffic is flowing through this Worker. |
| No recent data | The connection is in place, but no hits have arrived for more than 24 hours. Check that the Worker route still matches your traffic. |
| Degraded / Unhealthy | Something is failing. VitalSentinel retries automatically in the background, and the panel shows the last error. |
| Disabled | Collection is stopped. |
| Disconnecting | The Worker and its route are being removed. |
Reconnecting with a new token
If VitalSentinel can no longer read the saved token, Settings shows Cloudflare needs reconnecting. Click Reconnect Cloudflare and paste a new token from the same Cloudflare account and zone, with the same three permissions. Your Worker and route stay as they are; only the saved token is replaced. To move the domain to a different zone, disconnect and connect again.
If you can't reconnect right away, Stop collection stops collection immediately. The Worker stays on your Cloudflare account until you reconnect with a fresh token, and is then removed.
Disconnecting
Click Disconnect to remove the Worker route, then the Worker script from your Cloudflare account, then VitalSentinel's encrypted copy of the API token. Collection stops immediately. Server logs already collected are kept for their retention window and stay visible in your reports.
Disconnecting does not revoke the token. It stays valid in Cloudflare until you revoke it yourself under My Profile → API Tokens.
Because a domain collects from one source at a time, disconnecting is also how you switch to a log-file source. You can reconnect Cloudflare at any time.
Log-file sources
Log-file sources collect your existing access logs on a schedule. They are available on paid plans.
Click Your own server, then choose a Source type in the Connect a log source dialog:
| Source type | Collects from |
|---|---|
| SFTP | Rotated log files on an SFTP server |
| FTPS | Rotated log files on an FTPS server |
| Amazon S3 | Log files in an Amazon S3 bucket |
| Cloudflare R2 | Log files in a Cloudflare R2 bucket |
| Google Cloud Storage | Log files in a Google Cloud Storage bucket |
| Manual upload | A log file you upload directly. No server or bucket to connect. |
Connection settings
SFTP and FTPS
| Field | Notes |
|---|---|
| Host | Required, for example logs.example.com |
| Port | Optional. Defaults to 22 for SFTP and 21 for FTPS |
| Username | Required |
| File path pattern | Required. Which files to collect, with wildcards, for example /var/log/nginx/access.log* or /logs/*.log.gz |
| Credentials | Choose Password or Private key (PEM) |
Amazon S3, Cloudflare R2 and Google Cloud Storage
| Field | Notes |
|---|---|
| Bucket | Required |
| Endpoint | Required for R2 and GCS. Leave blank for standard Amazon S3 |
| Region | Optional, for example auto |
| Prefix | Optional path prefix within the bucket |
| Credentials | Access key ID and Secret access key |
Sync interval (minutes) sets how often files are collected: between 720 and 1440 minutes (every 12 to 24 hours). The default is 720.
Testing the connection
Click Test connection before saving. Save stays disabled until a test succeeds, and changing any field after a passing test requires a new test. Manual uploads skip this step.
After saving, the source starts collecting on its configured schedule.
Supported log formats
The log format is detected automatically. There is nothing to pick. Supported formats:
- Combined (Apache/Nginx)
- Custom Nginx
log_format - IIS (W3C extended)
- Newline-delimited JSON
- Cloudflare log fields
- AWS Application Load Balancer
- AWS CloudFront
Your logs must include the User-Agent, because that is how a hit is matched to a bot. The plain Common Log Format has no User-Agent field and cannot be used.
If a custom Nginx format cannot be detected during the test, a Nginx log_format directive field appears. Paste the exact log_format directive from your Nginx config and test again.
Uploading a file
For a Manual upload source, click Upload file on the source card.
- Accepted files:
.log,.txt,.gz,.bz2,.zst,.zip,.jsonor.ndjson, optionally followed by a rotation number (for exampleaccess.log.1) - Maximum size: 8 GB per file
- The file is processed on the source's next scheduled sync
Importing history (backfill)
Log-file sources can import past log files, so your reports start with history already in them. To re-collect older files later, click Backfill on the source card. Re-collect historical logs re-lists files from the date you choose on the next scheduled sync. Leave Collect from blank to go as far back as your data retention allows.
Backfilled hits count toward your monthly hit allowance.
Some reports start later than the backfilled history. For example, New pages: Googlebot to first impression becomes available after 30 days of collection, because the first 30 days are mostly backfill, not new pages.
Managing a log-file source
The source card shows Sync status (last sync, last event received, consecutive failures, sync interval), the Connection details, the detected log format, and the source's Capabilities.
- Edit changes the connection settings. The source type cannot be changed. If you change the host, port, endpoint or username, you must re-enter the credentials: they are never sent to a new destination.
- Delete turns off collection from that source and permanently deletes its stored credentials.
Capabilities
Not every source records every field. The Capabilities line on a source tells you which fields it can't capture and which reports that affects. For example:
| Missing field | What becomes unavailable |
|---|---|
| Status code | The status-code and error reports |
| User-Agent | Matching hits to a specific bot at all |
| IP address | The Security report's network and spoofing detail |
| Content type | Crawl budget's content-type breakdown |
| Response bytes | Bytes-served figures |
| Response time | Response-time figures |
| Query string | Crawl budget's query-parameter waste |
Until a source has processed its first log file, its capabilities read as not yet determined.
Waiting for the first request
After you connect a source, the module shows Waiting for the first request until data arrives. This is an absence rather than a zero, which is why you see this page instead of empty charts.
- Cloudflare Worker: the first request is confirmed within about five minutes of arriving. Crawlers come on their own schedule, so on a quiet site the first visit can still be hours away.
- Log-file sources: files are collected on the source's schedule, so the first data lands after the next run finishes.
Click Check again to refresh. The reports open on their own once the first request arrives.
Troubleshooting
No Cloudflare zone found
The token's account has no zone matching your domain. Add the domain as a zone in Cloudflare, and make sure the token belongs to the right account.
Status shows No recent data
No hits have arrived for more than 24 hours (Cloudflare), or for more than twice the sync interval (log-file sources). For the Worker, check that the route still matches your traffic and that you have not hit the Workers Free daily limit. For log-file sources, check that new files are still being written to the path or bucket you configured.
My reports stopped moving
Check Monthly hit allowance in Settings. If the workspace has used its allowance, collection has stopped until the next billing cycle. See Monthly hit allowance.
Related
Server Log Monitoring
See which search engine and AI crawlers hit your server, how your server answered them, which bots are fake, and how much of your crawl budget is wasted.
Server Log Reports
Read the Server Log Monitoring reports - Overview, Bots, AI crawlers, Security, Crawled pages, Sections, Crawl budget, Crawl map, Responses and Explore.
