# Server logs

Server Log Monitoring reads the access logs that your server or CDN already writes. It shows every page that Googlebot and AI crawlers fetch, and what your server answered. It uses no script on your pages. You connect a log source instead.

## What Server Log Monitoring turns on [#what-server-log-monitoring-turns-on]

The **This lights up** card in the setup lists the parts of the module:

- **Bots**: who crawled you and how often, with AI crawlers broken out and fake bots flagged.
- **Pages**: which URLs and site sections take the crawl budget.
- **Responses**: what your server answered bots, and how that changed.
- **Crawl vs results**: crawling compared with [Google Search Console](/google-search-console/overview), [Web Analytics](/web-analytics/overview), [RUM Monitoring](/rum-monitoring/overview) and your sitemap.
- **Explore**: individual requests, filtered as you need.

See [Server Log Monitoring](/server-logs/overview).

## Before you start [#before-you-start]

- Only workspace owners and admins can connect a log source. A member sees **A workspace admin connects server logs** and the button **Ask a workspace admin**. Members cannot open the Server Log Monitoring **Settings** page.
- The Cloudflare Worker is on every plan. File sources (SFTP, FTPS, cloud storage buckets and uploads) are on paid plans. See [Plans and limits](/workspace/plans-and-limits).
- A domain collects from one log source at a time. You can change the source later.
- Every hit counts toward a monthly hit allowance. The allowance depends on your plan. See [Settings](/server-logs/settings).

## Choose how to send your logs [#choose-how-to-send-your-logs]

| Method | Use it when | How it collects | Plans |
|--------|-------------|-----------------|-------|
| **Cloudflare Worker** | Your site runs behind Cloudflare. | **Real time** | Every plan |
| **SFTP or FTPS** | Your own server (Nginx, Apache, IIS) writes log files. | **On a schedule**, every 12 to 24 hours, with history | Paid plans |
| **Cloud storage bucket** | Your logs go to Amazon S3, Cloudflare R2 or Google Cloud Storage. | **On a schedule**, every 12 to 24 hours, with history | Paid plans |
| **Upload a log file** | You want a first look with an exported access log. | **One-off** | Paid plans |

On the Free plan, the file methods show the chip **On paid plans**. The setup then says **File sources are on paid plans**: "On Free, connect through the Cloudflare Worker instead: it is real time and on every plan."

### Why file sources need a paid plan [#why-file-sources-need-a-paid-plan]

A file source can import a year of history in one job. On the Free plan, this would use a month of hit allowance at once. To add a file source, upgrade your plan.

### Which method is right for my site? [#which-method-is-right-for-my-site]

- **Your site uses Cloudflare (orange cloud)**: use the Cloudflare Worker. It is real time and sees the requests that the Cloudflare cache answers.
- **You have SSH access to your own server**: use SFTP. It is safer and simpler than FTPS.
- **Your server, load balancer or CDN already writes logs to a bucket**: connect the bucket.
- **You use shared hosting or a hosting panel (cPanel, Plesk)**: download the access log and use **Manual upload**. If your host gives SFTP access to the logs folder, use SFTP.
- **You only want to try the module**: use **Manual upload** with one access log file.

For the log locations of common servers and hosting panels, see [Server log formats](/connect/server-log-formats).

## Where to connect a log source [#where-to-connect-a-log-source]

- **Onboarding**: the fourth step, **Connect Server Log Monitoring** (marked **Optional**). The step asks **How should we get your logs?** Select a method, then click its button: **Connect Cloudflare**, **Set up SFTP or FTPS**, **Set up a bucket** or **Set up an upload**.
- **The module page**: if no source is connected, the Server Log Monitoring pages show **Connect server logs**.
- **Server Log Monitoring > Settings**: the **Log source** card offers **Cloudflare Worker** (**Recommended**) with **Connect Cloudflare**, and **Your own server** (**Beta**) with **Connect your own server**.

Each button opens the side panel **Connect Server Log Monitoring**. If you are not ready in onboarding, click **Skip for now**.

## Connect the Cloudflare Worker [#connect-the-cloudflare-worker]

The Cloudflare Worker is a small program that VitalSentinel deploys into your own Cloudflare account, on the zone of your domain. It sends the log line of each request to VitalSentinel when your server answers it. If the Worker fails, your site continues to work, because the route fails open.

### What you need in Cloudflare [#what-you-need-in-cloudflare]

- Your domain is a zone (a domain added to Cloudflare) in your Cloudflare account.
- Cloudflare proxies the traffic of the domain. In the Cloudflare DNS settings, the records of your site show the orange cloud (**Proxied**). The Worker does not see traffic that goes to your server directly.
- The Cloudflare account has room for one more Worker. Cloudflare allows 100 Workers on Workers Free and 500 on Workers Paid.
- You can create API tokens in the Cloudflare account.

You do not need a paid Cloudflare plan. The Worker runs on Workers Free and on Workers Paid. See [Cloudflare limits that the wizard shows](#cloudflare-limits-that-the-wizard-shows).

### Create a Cloudflare API token [#create-a-cloudflare-api-token]

Before you start the wizard, create an API token in the Cloudflare dashboard:

1. Go to **My Profile** > **API Tokens**.
2. Click **Create Token**.
3. At **Create Custom Token**, click **Get started**.
4. Enter a token name, for example `VitalSentinel`.
5. Under **Permissions**, add the three permissions in the table below.
6. Under **Account Resources**, select the account of your domain.
7. Under **Zone Resources**, select the zone of your domain.
8. Click **Continue to summary**, then click **Create Token**.
9. Copy the token. Cloudflare shows it only once.

| Scope | Permission |
|-------|------------|
| Account | Workers Scripts: Edit |
| Zone | Workers Routes: Edit |
| Zone | Zone: Read |

The wizard asks for exactly these three permissions: "Narrower is safer, and it's all this integration needs."

### The four steps of the Cloudflare wizard [#the-four-steps-of-the-cloudflare-wizard]

The wizard shows **Step 1 of 4** to **Step 4 of 4**:

1. **Connect an API token**: paste the token into **Cloudflare API token**, then click **Verify token**.
2. **Confirm the zone**: VitalSentinel lists the Cloudflare zones that match your domain. Each zone shows its account, its Cloudflare plan and its status. Click **Use this zone**.
3. **Review plan and cost**: read the cost estimate, then click **Continue**.
4. **Deploy the Worker**: click **Deploy Worker**.

A message "Worker deployed" confirms the deploy. The Worker is unique to your domain. VitalSentinel adds a route that sends the traffic of your domain through it, and does not touch other Workers.

### What happens to the API token [#what-happens-to-the-api-token]

VitalSentinel saves nothing until the deploy succeeds. Each step sends the token again and does not use a stored copy. After the deploy, VitalSentinel stores the token encrypted and shows only its first characters.

If the token is not active, the wizard shows its status, for example "This token is expired, not active."

If you pasted only part of the token, the wizard says "This token couldn't be verified. Double-check you pasted the full value."

### The Worker route [#the-worker-route]

VitalSentinel adds one Worker route with the pattern `*example.com/*`, with your domain in place of `example.com`. This pattern matches your domain and all its subdomains, for example `www.example.com`.

Cloudflare runs only one Worker for each request. When more than one route matches a request, the most specific route wins. If you already have a Worker on a more specific route, for example `www.example.com/*` or `example.com/api/*`, the VitalSentinel Worker does not see those requests.

If a route with the same pattern already points to a different Worker, the deploy fails. The wizard says "A Cloudflare Worker route for pattern '*example.com/*' already exists and points at a different script". VitalSentinel does not change the route of another Worker.

### Cost of the Worker on your Cloudflare bill [#cost-of-the-worker-on-your-cloudflare-bill]

<Callout type="warn">
  The Worker runs in your own Cloudflare account, so Cloudflare bills its usage, not VitalSentinel. Cloudflare charges roughly $0.30 per million requests that a Worker runs on.
</Callout>

In the **Review plan and cost** step, the wizard shows your Cloudflare plan and how many Workers the account uses, out of its limit. It estimates the monthly cost from the traffic of your zone in the last 7 days. If the token cannot read zone analytics, the wizard says so and gives no number.

If there is an estimate, tick **I understand and accept this cost on my Cloudflare account.** before you click **Continue**. If you do not want this cost, connect an SFTP, S3, R2 or GCS source instead. You get the same dashboards, once a day and not in real time.

### Cloudflare limits that the wizard shows [#cloudflare-limits-that-the-wizard-shows]

- **Workers Free daily limit**: on Workers Free, Cloudflare runs a Worker on at most 100,000 requests a day. After that, Cloudflare stops the Worker for the rest of the day. Your site stays up, but log collection stops until the next day. The wizard warns you if your traffic is above the limit.
- **Workers per account**: if the account is at or near its limit, the wizard warns you. You can still continue. If the deploy fails, free up a script or upgrade Workers.
- **Unknown subscription**: if the token cannot read your Workers subscription, the wizard assumes the Free limit of 100 Workers. It says that the real limit may be higher.

To prevent gaps in collection, upgrade to Workers Paid or use a file source.

### Cloudflare Worker status [#cloudflare-worker-status]

After the deploy, the **Log source** card in **Settings** shows the Worker status, a **Refresh** button and **Integration details**. The details are deployment mode, Worker script, Worker version, last verified, last event received and consecutive failures.

| Status | What it means |
|--------|---------------|
| **Pending** | The Worker is deployed. VitalSentinel checks its route and token every few minutes. On a low-traffic site, the first request can take hours or days. |
| **Healthy** | Bot traffic flows through the Worker. |
| **No recent data** | The connection is in place, but no hits arrived for more than 24 hours. |
| **Degraded** or **Unhealthy** | Something fails. VitalSentinel shows the last error. For **Unhealthy**, it also retries automatically. |
| **Disabled** | Collection is stopped. |
| **Disconnecting** | VitalSentinel removes the Worker and its route. |

### Reconnect Cloudflare with a new token [#reconnect-cloudflare-with-a-new-token]

If VitalSentinel cannot read the saved token, the card shows **Cloudflare needs reconnecting**.

1. Click **Reconnect Cloudflare**.
2. Create a new token on the same Cloudflare account and zone, with the same three permissions.
3. Paste it into **New Cloudflare API token**, then click **Reconnect**.

Your Worker and route stay as they are. VitalSentinel replaces only the saved token. To move the domain to a different zone, disconnect Cloudflare and connect it again.

If you cannot reconnect now, click **Stop collection**. Collection stops at once. The Worker stays in your Cloudflare account until you reconnect with a new token, and VitalSentinel then removes it.

### Disconnect Cloudflare [#disconnect-cloudflare]

1. On the **Log source** card, click **Disconnect**.
2. In the dialog **Disconnect Cloudflare?**, click **Disconnect**.

VitalSentinel removes the Worker route, then the Worker script, then its encrypted copy of the token. Collection stops at once. VitalSentinel keeps the server logs that it already collected for the server log history of your plan. See [Plans and limits](/workspace/plans-and-limits). To use a file source, disconnect Cloudflare first.

<Callout type="info">
  Disconnecting does not revoke the token. The token stays valid in Cloudflare until you revoke it in **My Profile** > **API Tokens**.
</Callout>

## Connect a file source [#connect-a-file-source]

File sources collect your access logs on a schedule. In the side panel, the form **Connect a log source** opens. Select the **Source type**:

| Source type | Collects from |
|-------------|---------------|
| **SFTP** | Rotated log files on an SFTP server |
| **FTPS** | Rotated log files on an FTPS server |
| **Amazon S3** | Log files in an Amazon S3 bucket |
| **Cloudflare R2** | Log files in a Cloudflare R2 bucket |
| **Google Cloud Storage** | Log files in a Google Cloud Storage bucket |
| **Manual upload** | A log file that you upload. There is no server or bucket to connect. |

If your plan does not include file sources, the panel shows **File-based sources are not on your plan** with "Your plan collects server logs through the Cloudflare Worker."

Before you connect a file source, make sure that your logs use a format that VitalSentinel reads. See [Server log formats](/connect/server-log-formats).

## Connect SFTP or FTPS [#connect-sftp-or-ftps]

SFTP and FTPS let VitalSentinel log in to your server and read the access log files. SFTP uses SSH (port 22 by default). FTPS is FTP with TLS encryption (port 21 by default). If your server has SSH, use SFTP.

### Prepare a user on your server [#prepare-a-user-on-your-server]

Create a separate user for VitalSentinel on your server:

- The user can only read. It does not need write access.
- The user can read the folder of the access logs and the log files in it.
- The user cannot read other folders, if your server allows this restriction.

VitalSentinel only lists and downloads files. It never changes or deletes files on your server.

Make sure that the user can also read new files. Log rotation creates new files, sometimes with different permissions. If the user cannot read the rotated files, collection fails.

### Choose the file path pattern [#choose-the-file-path-pattern]

The **File path pattern** tells VitalSentinel which files to collect, for example `/var/log/nginx/access.log*`. Use the path as the SFTP or FTPS user sees it. If your server locks the user into a folder, the path starts at that folder.

- Use a wildcard (`*`), so that the pattern also matches the rotated files, for example `access.log.1` and `access.log.2.gz`.
- For FTPS, put the wildcard only in the file name. The pattern cannot search more than one folder.
- Make sure that the pattern matches only access logs. Do not match error logs.

VitalSentinel reads gzip (`.gz`) files, for example `/logs/*.log.gz`. It also reads bzip2, zstd and zip files.

### How VitalSentinel reads rotated log files [#how-vitalsentinel-reads-rotated-log-files]

Most servers rotate their access log: they rename the current file, for example to `access.log.1`, and start a new `access.log`. VitalSentinel reads each rotated file one time.

VitalSentinel does not read the file that the server still writes to, if rotated copies of it exist. It also waits until the server stopped writing to a file for some minutes. This prevents double counting.

Thus, the data of a day arrives after your server rotates the log and after the next scheduled sync. With daily rotation and a 12-hour sync interval, the data is usually one to two days old. If your server never rotates its log, set up log rotation.

### Allow VitalSentinel through your firewall [#allow-vitalsentinel-through-your-firewall]

VitalSentinel connects to your server from the IP addresses on [Monitoring IPs and user agents](/help/monitoring-ips). If your server has a firewall or an IP allowlist, allow these addresses:

- **SFTP**: the SFTP port, usually 22.
- **FTPS**: the control port, usually 21, and the passive data ports of your FTP server.

If a security tool such as fail2ban blocks IP addresses after failed logins, add the VitalSentinel addresses to its ignore list.

### SFTP with a password or a private key [#sftp-with-a-password-or-a-private-key]

In **Credentials**, select **Password** or **Private key**:

- **Password**: the password of the SFTP user.
- **Private key**: paste the whole private key, from the `-----BEGIN` line to the `-----END` line, into **Private key (PEM)**. Put the matching public key into the `~/.ssh/authorized_keys` file of the user on your server.

The private key must not have a passphrase. The form has no field for a passphrase. Create a separate key pair for VitalSentinel.

The first time VitalSentinel connects, it records the host key of your SFTP server. If the host key changes later, VitalSentinel refuses to connect, because the change can mean an attack. If you replaced the server on purpose, delete the source and add it again.

### FTPS requirements [#ftps-requirements]

- The server must support explicit TLS (`AUTH TLS`) with TLS 1.2 or later. VitalSentinel does not connect with plain FTP.
- VitalSentinel uses passive mode for file transfers.
- FTPS logs in with a password. A private key works only with SFTP.

The first time VitalSentinel connects, it records the TLS certificate of your FTPS server. If the certificate changes, for example when the server renews it, the connection fails. **Test connection** then says "The server's TLS certificate does not match the one recorded for this source. Nothing was sent. If the certificate was changed deliberately, remove and re-add the source."

If your server renews its certificate often, use SFTP to prevent these failures.

### Connect SFTP or FTPS step by step [#connect-sftp-or-ftps-step-by-step]

1. In **Source type**, select **SFTP** or **FTPS**.
2. Enter the **Host**, for example `logs.example.com`.
3. Optional: enter the **Port**. The default is 22 for SFTP and 21 for FTPS.
4. Enter the **Username**.
5. In **File path pattern**, enter the files to collect, for example `/var/log/nginx/access.log*`.
6. In **Credentials**, enter the password. For SFTP, you can select **Private key** and paste the key instead.
7. Set the **Sync interval (minutes)**.
8. Click **Test connection**.
9. When the test succeeds, click **Save**.

### What Test connection checks for SFTP and FTPS [#what-test-connection-checks-for-sftp-and-ftps]

For **SFTP**, the test makes sure that the host answers and that an SSH server runs on the port. It does not log in. If it passes, it says "TCP reachable and an SSH server responded. Credentials are verified on the first scheduled run."

Thus, a wrong password, key or path shows only after the first collection run. Look at the **Log source** card after the first run.

For **FTPS**, the test logs in and lists the folder. If it passes, it says "Connected, authenticated, and listed the directory."

| Message | What it means |
|---------|---------------|
| "could not connect to the configured host and port" | The host or port is wrong, or a firewall blocks VitalSentinel. |
| "connected, but the peer does not speak SSH" | The port is not an SFTP port, for example an FTP or web port. |
| "connection failed" | FTPS failed: wrong host, port, user or password, no TLS support, or a firewall. |

## Connect a cloud storage bucket [#connect-a-cloud-storage-bucket]

VitalSentinel reads log files from Amazon S3, Cloudflare R2 and Google Cloud Storage. It uses the S3-compatible API of each service, with an access key ID and a secret access key. It needs permission to list the objects in the bucket and to read them. It does not need write or delete permission.

1. In **Source type**, select **Amazon S3**, **Cloudflare R2** or **Google Cloud Storage**.
2. Enter the **Bucket** name.
3. For R2 and Google Cloud Storage, enter the **Endpoint**. For Amazon S3, leave it blank.
4. Optional: enter the **Region** and a **Prefix** (the folder of the logs in the bucket).
5. In **Credentials**, enter the **Access key ID** and the **Secret access key**.
6. Set the **Sync interval (minutes)**.
7. Click **Test connection**.
8. When the test succeeds, click **Save**.

A message "Log source connected" confirms the save. The panel then shows **Source connected**, with "It will start collecting on its configured schedule."

VitalSentinel reads all objects under the **Prefix**. Make sure that the prefix holds only access logs.

### Amazon S3 access [#amazon-s3-access]

Create an IAM user for VitalSentinel and give it this policy. Replace `my-log-bucket` with your bucket and `logs/` with your prefix. Then create an access key for the user.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-log-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["logs/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-log-bucket/logs/*"
    }
  ]
}
```

In VitalSentinel, enter `logs/` as the **Prefix**, with the slash at the end. Enter the region of your bucket as the **Region**, for example `eu-west-1`. If you leave the region blank, VitalSentinel uses `us-east-1`.

If the bucket encrypts its objects with an AWS KMS key, the user also needs `kms:Decrypt` on that key.

### Cloudflare R2 access [#cloudflare-r2-access]

Create an R2 API token that can only read:

1. In the Cloudflare dashboard, open the **R2 object storage** overview.
2. Next to **API Tokens**, click **Manage**.
3. Create an account API token or a user API token.
4. Under **Permissions**, select **Object Read only**.
5. Scope the token to the bucket of your logs, then click **Create**.
6. Copy the **Access Key ID** and the **Secret Access Key**. Cloudflare shows the secret only once.

In VitalSentinel, enter the **Endpoint** `https://<ACCOUNT_ID>.r2.cloudflarestorage.com`, with your Cloudflare account ID. For a bucket in the EU jurisdiction, use `https://<ACCOUNT_ID>.eu.r2.cloudflarestorage.com`. Leave the **Region** blank or enter `auto`.

### Google Cloud Storage access [#google-cloud-storage-access]

Google Cloud Storage uses HMAC keys for the S3-compatible API:

1. Create a service account for VitalSentinel in your Google Cloud project.
2. On the log bucket, give the service account the role **Storage Object Viewer** (`roles/storage.objectViewer`).
3. In the Cloud Storage **Settings** page, open the **Interoperability** tab.
4. Click **Create a key for a service account**.
5. Select the service account, then click **Create key**.
6. Copy the access ID and the secret.

In VitalSentinel, enter the access ID as the **Access key ID** and the secret as the **Secret access key**. Enter the **Endpoint** `https://storage.googleapis.com`.

The Storage Object Viewer role can list and read objects, but it cannot change or delete them.

### What Test connection checks for a bucket [#what-test-connection-checks-for-a-bucket]

The test lists the bucket under your **Prefix**. If it passes, it says "Bucket is reachable with the given credentials."

| Message | What it means |
|---------|---------------|
| "could not reach the configured endpoint" | The endpoint is wrong or does not answer. Look for a typo in **Endpoint**. |
| "could not list the bucket with the given credentials" | The key is wrong, or it has no permission to list the bucket or the prefix. The bucket name or the region can also be wrong. |

The test does not read a file. If the key can list but not read the objects, the first collection run fails. Make sure that the key has read permission (`s3:GetObject` on S3).

## Sync interval and connection test [#sync-interval-and-connection-test]

The **Sync interval (minutes)** sets how often VitalSentinel collects new files: "Between 12 and 24 hours (720-1440)". The default is 720 minutes.

If you change a field after a test, run the test again. The form says "Test the connection before saving – Save stays disabled until it succeeds."

A successful test shows the message "Connection works". A failed test shows "The connection test didn't pass" and the reason.

## Upload a log file [#upload-a-log-file]

1. In **Source type**, select **Manual upload**.
2. In **Log file**, drag the file into the area, or click **Choose a log file**.
3. Click **Save**.

Save adds the source and uploads the file. VitalSentinel reads the file once, then deletes it from its storage. To upload more files later, use **Upload a log file** on the **Log source** card. A manual upload has no schedule and no connection test.

The file can be `.log`, `.txt`, `.gz`, `.json` or `.ndjson`, with an optional rotation number, such as `access.log.1`. For other file types, the form shows an error. The maximum size is 8 GB per file.

The upload area says "Drag a log file here, or choose one. Plain text or .gz, up to 8 GB."

## Log formats and fields [#log-formats-and-fields]

VitalSentinel finds the log format automatically. It reads the Combined format of Apache and Nginx, custom Nginx formats and IIS (W3C extended) logs. It also reads newline-delimited JSON, Cloudflare Logpush, AWS Application Load Balancer and AWS CloudFront logs.

<Callout type="warn">
  Your logs must include the User-Agent, because VitalSentinel uses it to match each hit to a bot. VitalSentinel rejects the Common log format (Apache `common`), which has no User-Agent. Change to the Combined format.
</Callout>

If your server is behind a load balancer or CDN, the log must have the real client IP. If not, every hit seems to come from your proxy, and VitalSentinel cannot verify bots.

For the full list of formats, the fields that VitalSentinel needs, and example Nginx and Apache settings, see [Server log formats](/connect/server-log-formats).

## Manage a file source [#manage-a-file-source]

The **Log source** card in Server Log Monitoring **Settings** shows the source type and a status chip. It has two lists:

- **Sync status**: **Last sync**, **Last event received**, **Consecutive failures** and **Sync interval**. For a manual upload, **Last imported** replaces **Last sync**, and there is no **Sync interval**.
- **Connection**: **Host**, **Username** and **Path**, or **Bucket**, with **Log format** and **Credentials**. **Log format** shows **Auto-detect** until VitalSentinel confirms the format.

The card has the buttons **Test connection**, **Backfill**, **Edit** and **Delete**. A manual upload has only **Delete** and the upload area. If a sync fails, the card shows the last error.

### Edit a file source [#edit-a-file-source]

Click **Edit** to change the connection settings. You cannot change the source type.

If you leave a credential field blank, VitalSentinel keeps the current credentials. But if you change the host, port, endpoint or username, enter the credentials again. The form then says "Re-enter credentials – they are never sent to a new destination."

### Import history (backfill) [#import-history-backfill]

File sources can import old log files, so that your reports start with history. To collect old files again later, click **Backfill**.

1. In the dialog **Re-collect historical logs**, enter a date in **Collect from**. Leave it blank to go as far back as the server log history of your plan allows. See [Plans and limits](/workspace/plans-and-limits).
2. Click **Start backfill**.

VitalSentinel lists the files again from that date on the next scheduled sync. Backfilled hits count toward your monthly hit allowance.

### Delete a file source [#delete-a-file-source]

1. Click **Delete**.
2. In the dialog **Remove this log source?**, click **Remove source**.

This turns off collection from the source and permanently deletes its stored credentials. You cannot undo this. A message "Log source removed" confirms the delete.

### Capabilities of a source [#capabilities-of-a-source]

Not every source records every field. If a source cannot record a field, the card shows **Capabilities**, for example: "This source can't capture response time. That leaves response-time figures unavailable for it."

| Missing field | What becomes unavailable |
|---------------|--------------------------|
| Status | The status-code and error reports |
| User-Agent | Matching hits to a specific bot |
| IP | The network and spoofing detail of the security report |
| Content type | The content-type breakdown of crawl budget |
| Response bytes | Bytes-served figures |
| Response time | Response-time figures |
| Query | The query-parameter waste of crawl budget |

Until a source reads its first log file, VitalSentinel does not know what it can capture. The card then says "Not yet determined".

## Wait for the first request [#wait-for-the-first-request]

After you connect a source, the setup shows **Waiting for the first data**. The sidebar shows "Connected – waiting for the first request". Reports fill in as your server logs requests.

- **Cloudflare Worker**: about 5 minutes after the first request arrives. Crawlers come on their own schedule, so on a quiet site the first visit can take hours.
- **File sources**: the first data comes after the next scheduled run. VitalSentinel does not read the log file that the server still writes to, so the newest data can be one to two days old.

The reports open on their own when the first request arrives. When data flows, the setup shows **Your log source is delivering**.

## Log source status in Settings [#log-source-status-in-settings]

The top of Server Log Monitoring **Settings** tells you in one sentence if the log source delivers:

| Message | What it means |
|---------|---------------|
| **Logs are arriving** | Hits arrive. The line shows the date of the newest hit. |
| **Connected – waiting for the first hit** | The source works, but no hit arrived yet. |
| **No new logs since** a date | No new hits arrived recently. |
| **Log collection is failing** | The source failed one or more times in a row. The line shows how often. |
| **Your monthly hit allowance is used up** | VitalSentinel stores no new hits until the next billing cycle. |
| **Log collection is switched off** | Collection is stopped. |

## When a log source stops delivering [#when-a-log-source-stops-delivering]

If no new requests arrive, the module shows **Your log source stopped delivering**. "No new requests arrived recently. The source may be disconnected, or the credentials changed." Click **Fix the log source** to open **Settings**.

The source status chip then shows **No recent data**. For the Cloudflare Worker, this means no hits for more than 24 hours. For a file source, it means no hits for more than two sync intervals. A manual upload never shows **No recent data**.

For the Cloudflare Worker, the card says "The connection is still in place, but no hits have arrived since" the date. "Reports cover only the days that were collected."

## Request a connector for another platform [#request-a-connector-for-another-platform]

If your site runs on another platform, click **Request a connector** below the methods. In the dialog **Request a real-time connector**, select the **Platform**. The choices are Fastly, Akamai, AWS CloudFront, Vercel, Netlify, Azure Front Door, Google Cloud CDN, Bunny CDN and Other. Then click **Send request**.

VitalSentinel builds a new connector when several customers ask for the same platform. It sends you an email when the connector is ready. Until then, use the Cloudflare Worker or a file source.

## Troubleshooting [#troubleshooting]

### Why does Server Log Monitoring show no data after I connected a source? [#why-does-server-log-monitoring-show-no-data-after-i-connected-a-source]

- **Cloudflare Worker**: if the status is **Pending**, no bot request has arrived yet. On a quiet site, this can take hours or days. Make sure that Cloudflare proxies your traffic (orange cloud).
- **SFTP, FTPS or bucket**: the first collection runs on the schedule. Wait for the next sync, then look at **Last sync** and the last error on the **Log source** card.
- **File sources**: VitalSentinel does not read the current log file while rotated copies of it exist. If your server rotates its log once a day, today's requests arrive after tomorrow's rotation.
- If the card shows an error about the log format, see [Server log formats](/connect/server-log-formats).

### Why does the Cloudflare wizard find no zone? [#why-does-the-cloudflare-wizard-find-no-zone]

If the wizard says "We found no Cloudflare zone for" your domain, the account of the token has no matching zone. Add the domain as a zone in Cloudflare. Make sure that the token belongs to the right Cloudflare account, and that its **Zone Resources** include the zone.

### Why does my Cloudflare Worker stay Pending? [#why-does-my-cloudflare-worker-stay-pending]

The Worker turns **Healthy** when the first bot request arrives. On a low-traffic site, this can take hours or days. Make sure that Cloudflare proxies the traffic of your domain. DNS records with a grey cloud (**DNS only**) send traffic past the Worker.

### Why does my log source show No recent data or Log collection is failing? [#why-does-my-log-source-show-no-recent-data-or-log-collection-is-failing]

- **Cloudflare Worker**: make sure that the route still exists and that no more specific route of another Worker took over. Also make sure that you did not reach the Workers Free limit of 100,000 requests a day.
- **Cloudflare token**: if the card shows **Cloudflare needs reconnecting**, click **Reconnect Cloudflare** and paste a new token.
- **File source**: make sure that your server still writes and rotates files in the path or bucket that you set. Make sure that the password, key or access key did not change.
- Click **Test connection** on the **Log source** card to check the connection again.

### Why do hits show at the wrong time or on the wrong day? [#why-do-hits-show-at-the-wrong-time-or-on-the-wrong-day]

VitalSentinel reads the time zone from each log line, for example `+0200` in the Combined format. If the times are wrong, the clock or the time zone of your server is wrong. Correct the server clock, for example with NTP time synchronization.

Daily charts count days by UTC date. Thus, a day in a chart can start some hours before or after your local midnight. The chart help says "Days are UTC days: the log totals are counted by UTC date and cannot be shown in your time zone."

For JSON logs, each timestamp must have a time zone, for example `Z` or `+02:00`, or be a Unix timestamp.

### Why are the user agent or IP fields missing? [#why-are-the-user-agent-or-ip-fields-missing]

The **Capabilities** list on the **Log source** card can say that the source cannot capture the User-Agent or the IP. Then the log lines do not have these fields. Common causes:

- The server uses the Common log format. Change it to Combined.
- A custom Nginx format does not include `$http_user_agent` or `$remote_addr`.
- A JSON log uses field names that VitalSentinel does not know.

Change the log format of your server. New files then have the fields. See [Server log formats](/connect/server-log-formats).

### Why do all hits come from the same IP address? [#why-do-all-hits-come-from-the-same-ip-address]

If a CDN, load balancer or reverse proxy is in front of your server, your server logs the IP address of the proxy. Then VitalSentinel cannot verify bots. It can mark real Googlebot requests as fake, because the IP address is not a Google address.

To fix it, configure your server to log the real client IP from the `X-Forwarded-For` or `CF-Connecting-IP` header. VitalSentinel reads the client address field of the log line, not an extra `X-Forwarded-For` column. See [Server log formats](/connect/server-log-formats) for the Nginx and Apache settings.

If your site is on Cloudflare, the Cloudflare Worker records the real client IP automatically.

### Why do I see only part of my bot traffic? [#why-do-i-see-only-part-of-my-bot-traffic]

- **CDN cache**: if a CDN answers a request from its cache, the request does not reach your server, and your server logs do not show it. Use the Cloudflare Worker, or the logs of your CDN, to see all requests.
- **More than one server**: if your site runs on several servers, each one writes its own log. Collect the logs of all servers, for example in one bucket.
- **HTTP and HTTPS logs**: some servers and panels write HTTPS requests to a separate file, for example `access_ssl_log` in Plesk. Make sure that the **File path pattern** includes it.
- **Excluded traffic**: Server Log Monitoring **Settings** can exclude some traffic. See [Settings](/server-logs/settings).

### Why are compressed or rotated files not collected? [#why-are-compressed-or-rotated-files-not-collected]

- The **File path pattern** does not match the rotated names. Use a wildcard, for example `access.log*`, to match `access.log.1` and `access.log.2.gz`.
- For FTPS, the wildcard is in a folder name. Put the wildcard only in the file name.
- The SFTP or FTPS user cannot read the rotated files. Log rotation can create files with other permissions.
- The file uses a compression that VitalSentinel does not read. VitalSentinel reads gzip, bzip2, zstd and zip.
- A file that the server still writes to is not read until the server rotates it.

### Why does the SFTP or FTPS login fail? [#why-does-the-sftp-or-ftps-login-fail]

- **Firewall or IP allowlist**: allow the addresses on [Monitoring IPs and user agents](/help/monitoring-ips) on the SFTP or FTPS port.
- **Password or key**: for SFTP with a key, the public key must be in `~/.ssh/authorized_keys` of the user, and the private key must have no passphrase.
- **Key with FTPS**: FTPS logs in with a password. A private key works only with SFTP.
- **Host key or certificate changed**: if you replaced the server or renewed the FTPS certificate, delete the source and add it again.
- **Blocked after failed logins**: a tool such as fail2ban can block the VitalSentinel addresses. Unblock them and add them to its ignore list.

The SFTP connection test does not log in. A login error shows only after the first scheduled run, on the **Log source** card.

### Why does the bucket connection fail with access denied? [#why-does-the-bucket-connection-fail-with-access-denied]

- The access key ID or the secret is wrong, or the key was deleted.
- The key has no permission to list the bucket. On S3, it needs `s3:ListBucket` for the bucket and the prefix.
- The key can list but cannot read. On S3, it needs `s3:GetObject`. On Google Cloud Storage, use the role **Storage Object Viewer**. On R2, use **Object Read only** for this bucket.
- The **Prefix** does not match the prefix in the S3 policy. Enter it with a slash at the end, for example `logs/`.
- The **Region** of an S3 bucket is wrong. Enter the region of the bucket.
- The **Endpoint** of R2 or Google Cloud Storage is wrong.

### Why is Save disabled in the file source form? [#why-is-save-disabled-in-the-file-source-form]

Save stays disabled until **Test connection** succeeds. If you changed a field after the test, run the test again. For a manual upload, choose a log file first.

### Why does VitalSentinel reject my log file? [#why-does-vitalsentinel-reject-my-log-file]

If the file uses the Common log format, it has no User-Agent. Change your server to the Combined format.

If you use a custom Nginx format, VitalSentinel can fail to detect it. Click **Edit** on the **Log source** card, then click **Test connection**. The form then shows the field **Nginx log_format directive**. Paste your `log_format` line, then test again. See [Server log formats](/connect/server-log-formats).

### Why did my reports stop moving? [#why-did-my-reports-stop-moving]

If the workspace used its monthly hit allowance, Settings shows **Your monthly hit allowance is used up**. VitalSentinel stores no new hits until the allowance resets: "We delete nothing that is already stored." Check **Monthly hit allowance** in Server Log Monitoring **Settings**.

To get more hits, upgrade your plan. See [Plans and limits](/workspace/plans-and-limits).

## Related [#related]

- [Server log formats](/connect/server-log-formats) - formats, fields, Nginx and Apache settings, log locations
- [Server Log Monitoring](/server-logs/overview) - what the module reports
- [Settings](/server-logs/settings) - hit allowance, excluded traffic and sections
- [Bots](/server-logs/bots) - the crawlers that the logs show
- [Monitoring IPs and user agents](/help/monitoring-ips) - the IP addresses to allow on your server
- [Plans and limits](/workspace/plans-and-limits) - the log hits of each plan