Server log formats
The access log formats that VitalSentinel Server Log Monitoring reads, the fields it needs, and how to configure Nginx, Apache and CDN logs.
Server Log Monitoring reads the access logs of your web server, load balancer or CDN. This page lists the log formats that VitalSentinel reads, the fields that each log line must have, and example server settings. Use it before you connect an SFTP, FTPS, bucket or upload source. To connect a source, see Server logs.
The Cloudflare Worker does not use log files. If your site runs behind Cloudflare, the Worker collects the requests directly, and this page does not apply to it.
How VitalSentinel finds the log format
You do not select a log format. VitalSentinel examines the first lines of each file and finds the format automatically. Until it confirms the format, the Log source card in Server Log Monitoring Settings shows Log format as Auto-detect. After the first file, the card shows the format name, for example Combined (Apache/Nginx).
VitalSentinel also detects compression from the file content, not from the file name. It reads plain text files and gzip (.gz) files. For SFTP, FTPS and bucket sources, it also reads bzip2, zstd and zip files.
If VitalSentinel cannot detect the format of a custom Nginx log, the app asks for your Nginx log_format line. See Custom Nginx log formats.
Log formats that VitalSentinel reads
The Log format row of the Log source card uses these names:
| Log format | Written by |
|---|---|
| Combined (Apache/Nginx) | Apache and Nginx with the default combined format |
| Custom Nginx log_format | Nginx with your own log_format, when you give VitalSentinel the format line |
| IIS (W3C extended) | Microsoft IIS, with a #Fields: header line in each file |
| Newline-delimited JSON | Servers and tools that write one JSON object for each request, on one line |
| Cloudflare log fields | Cloudflare Logpush, with the HTTP requests dataset |
| AWS Application Load Balancer | AWS Application Load Balancer access logs |
| AWS CloudFront | AWS CloudFront standard logs in the W3C format |
VitalSentinel does not read other formats, for example the Common log format, Parquet files or CSV exports.
The Common log format does not work
The Common log format (Apache common) has no User-Agent field. VitalSentinel uses the User-Agent to find the bot that sent each request, so it rejects Common logs. The error on the Log source card starts with "this log has no User-Agent or Referer field (Apache/Nginx Common format)."
To fix it, change your server to the Combined format. The next files that the server writes then work. Old files in the Common format stay unusable.
Fields that VitalSentinel needs
Each log line must have these fields:
- Time of the request, with a time zone.
- Request path, the URL that the client asked for.
- Status code, the HTTP answer of your server, for example
200or404. - User-Agent, the name that the client sends. VitalSentinel uses it to identify Googlebot, AI crawlers and other bots.
- Client IP address. VitalSentinel uses it to verify that a bot is real and not a fake bot.
These fields are optional, but each one unlocks more reports: request method, query string, referrer, response size and response time. Content type, cache status, host name, country and network (ASN) are also optional.
If a source does not record a field, the Log source card shows Capabilities, with the reports that the missing field disables. See Server logs.
Apache and Nginx combined format
The Combined format is the default format of Apache and Nginx. In most cases, you do not change anything. The Nginx combined format is:
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';The Apache combined format is:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog "logs/access_log" combinedVitalSentinel also detects a Combined line with more quoted fields at the end. An example is the Nginx main format, which adds "$http_x_forwarded_for". VitalSentinel ignores these extra fields.
Make sure your server uses the combined format
Look at your server configuration:
- Nginx: find the
access_logline. If it ends withcombined,mainor no format name, the format works. - Apache: find the
CustomLogline. If it ends withcombined, the format works. If it ends withcommon, change it tocombined.
Reload the server after a change. The new lines use the new format.
Custom Nginx log formats
You can use your own Nginx log_format, for example to record the response time. VitalSentinel cannot always detect a custom format automatically. If it cannot, the next collection run fails, and the Log source card shows the error.
To give VitalSentinel your format:
- On the Log source card, click Edit.
- Click Test connection. The form shows a warning and the field Nginx log_format directive.
- Paste the exact
log_formatline from your Nginx configuration. - Click Test connection again, then click Save changes.
The warning starts with "The connection works, but the last collection run could not process this source's log format".
A custom format must include $time_local or $time_iso8601. Without a time variable, VitalSentinel cannot read the lines.
Example format with response time
This format adds the response time and the content type to the combined format:
log_format vitalsentinel '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time "$sent_http_content_type"';
access_log /var/log/nginx/access.log vitalsentinel;When VitalSentinel asks for the format line, paste the whole log_format vitalsentinel ...; statement.
Nginx variables that VitalSentinel reads
VitalSentinel reads these Nginx variables in a custom format. It ignores other variables.
| Data | Variables |
|---|---|
| Time | $time_local, $time_iso8601 |
| Client IP | $remote_addr, $http_x_forwarded_for |
| Request | $request, $request_method, $request_uri, $uri, $document_uri, $args, $query_string |
| Answer | $status, $body_bytes_sent, $bytes_sent, $sent_http_content_type, $upstream_cache_status |
| Time taken | $request_time, $upstream_response_time |
| Client | $http_user_agent, $http_referer |
| Host | $host, $http_host |
VitalSentinel uses $http_x_forwarded_for only when the format has no $remote_addr. Any client can write a false value into this header.
Log the real client IP behind a proxy or CDN
If a CDN, a load balancer or a reverse proxy is in front of your server, the server sees the address of the proxy. Then every log line shows the same few IP addresses. VitalSentinel cannot verify the bots, and it can mark real Googlebot requests as fake.
VitalSentinel reads the client address field of the log line ($remote_addr in Nginx, %h or %a in Apache). It does not read an extra X-Forwarded-For column of a Combined line. Thus, configure the server to put the real client IP into the client address field.
Trust only the IP addresses of your own proxy or CDN. If you trust all addresses, any client can send a false IP address in the header.
Nginx real client IP
Nginx sets the real client IP with the realip module. Nginx must include this module (--with-http_realip_module). To make sure, run nginx -V and look for the module in the output.
For a load balancer or proxy that sends X-Forwarded-For, add this to the http block:
set_real_ip_from 10.0.0.0/8; # the address range of your proxy
real_ip_header X-Forwarded-For;
real_ip_recursive on;For Cloudflare, use the CF-Connecting-IP header. Add one set_real_ip_from line for each Cloudflare IP range. Cloudflare publishes the ranges at https://www.cloudflare.com/ips/ and changes them from time to time.
set_real_ip_from 192.0.2.0/24; # repeat for each Cloudflare range
real_ip_header CF-Connecting-IP;After the change, $remote_addr in the combined format contains the real client IP.
Apache real client IP
Apache sets the real client IP with the mod_remoteip module.
- Turn on the module. On Debian and Ubuntu, run
sudo a2enmod remoteip. - Add
RemoteIPHeaderand oneRemoteIPTrustedProxyline for each address range of your proxy. - In the
combinedformat, replace%hwith%a. - Restart Apache.
RemoteIPHeader X-Forwarded-For
RemoteIPTrustedProxy 10.0.0.0/8
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combinedFor Cloudflare, use RemoteIPHeader CF-Connecting-IP and add each range from https://www.cloudflare.com/ips/.
IIS, JSON and other formats
IIS (W3C extended)
Use the W3C log file format of IIS. Each file must start with a #Fields: header line, because the header tells VitalSentinel the order of the columns. In the IIS logging fields, select at least date, time, cs-uri-stem, sc-status, c-ip and cs(User-Agent).
Newline-delimited JSON
Each line must be one JSON object for one request. VitalSentinel finds the fields by their names. These are some of the names that it reads:
| Data | Field names |
|---|---|
| Time | timestamp, time, ts, @timestamp, datetime |
| Path | path, uri, url, request_path, request_uri |
| Status | status, status_code, response_status |
| User-Agent | user_agent, ua, useragent, http_user_agent |
| Client IP | ip, remote_addr, client_ip, remote_ip |
| Method | method, http_method, request_method |
The time must be an ISO 8601 (RFC 3339) value with a time zone, for example 2026-10-24T08:15:00Z, or a Unix timestamp in seconds or milliseconds. Each line must have a status field.
CDN and cloud provider logs
Cloudflare Logpush
If your site is on Cloudflare, the Cloudflare Worker is the simpler way to connect. As an alternative, Cloudflare Logpush can send the HTTP requests dataset to an R2, Amazon S3 or Google Cloud Storage bucket. Connect that bucket as a bucket source.
VitalSentinel reads the native Logpush field names, for example EdgeStartTimestamp, ClientRequestHost, ClientRequestURI, EdgeResponseStatus, ClientRequestUserAgent and ClientIP. Include these fields in the Logpush job. EdgeStartTimestamp can use the ISO 8601 or the Unix nanoseconds format.
Logpush availability and cost depend on your Cloudflare plan. Check the Cloudflare documentation.
AWS Application Load Balancer
AWS writes Application Load Balancer access logs to an S3 bucket. VitalSentinel reads these files as AWS writes them. Connect the bucket as an Amazon S3 source, with the log folder as the Prefix.
AWS CloudFront
CloudFront standard logging can write access logs to an S3 bucket. Use the W3C output format with tab-separated fields. Each file must have its #Fields: header line.
Include at least the fields date, time, cs-uri-stem, sc-status, c-ip and cs(User-Agent). Also include x-edge-location, because VitalSentinel uses it to tell CloudFront logs from IIS logs. VitalSentinel does not read the JSON or Parquet output of CloudFront.
Other CDNs
VitalSentinel has no format for other CDNs, for example Fastly, Akamai or Bunny CDN. If your CDN can write the Combined format or newline-delimited JSON to a bucket, connect that bucket. If not, click Request a connector in the setup. See Server logs.
Where servers and hosting panels keep access logs
Servers that you manage
| Server | Usual location of the access log |
|---|---|
| Nginx | /var/log/nginx/access.log, with rotated copies such as access.log.1 and access.log.2.gz |
| Apache on Debian and Ubuntu | /var/log/apache2/access.log |
| Apache on RHEL and CentOS | /var/log/httpd/access_log |
| IIS | %SystemDrive%\inetpub\logs\LogFiles\W3SVC<site id>\ |
Your server can use a different path. The access_log line (Nginx) or the CustomLog line (Apache) shows the real path.
cPanel
In cPanel, go to Metrics > Raw Access. There you can download the access log of each domain as a .gz file. Upload this file with Manual upload. Your hosting provider can turn off this feature.
In Configure Logs, you can archive the logs to your home directory. cPanel then keeps them in /home/username/logs. If your host gives you SFTP access, you can connect this folder as an SFTP source.
Plesk
In Plesk, go to Websites and Domains, select the domain, then click Logs > Manage Log Files. Click the download icon next to a log file. Upload the file with Manual upload.
On Linux, Plesk keeps these access logs for each domain: access_log and access_ssl_log (Apache), and proxy_access_log and proxy_access_ssl_log (Nginx). Use the SSL logs for HTTPS traffic. In Log Rotation, Plesk can compress and delete old logs.
Managed hosting
Many managed hosts give a log download in their dashboard or SFTP access to a logs folder. The log format of a managed host can be different from the Combined format. Upload one file first, with Manual upload, to make sure that VitalSentinel can read it.
Related
- Server logs - connect a log source and fix connection problems
- Server Log Monitoring - what the module reports
- Settings - hit allowance, excluded traffic and sections
- Monitoring IPs and user agents - the IP addresses to allow on your server
