Explore

The Explore page of Server Log Monitoring lets you look up exactly which bot fetched which page, and what your server answered. Use it to follow up on something you saw on another page.

Before you startCopy Anchor Link

• Connect a log source first. See Connect server logs.
• Open a domain. In the sidebar, select Server Log Monitoring > Explore.
• Explore uses every filter of the filter bar. It also has three filters of its own: Network (ASN), Content type and Response time (ms).
• Explore reads at most 366 days at a time. VitalSentinel keeps Server Log Monitoring data for 12 months (90 days on the Free plan). See Plans and limits.
• Members can read this page.

Many tables in Server Log Monitoring have an Explore link, for example in the bot drawer and the page drawer. The link opens Explore with the filters of the row.

Questions to start fromCopy Anchor Link

The top card of the Explore page has four ready-made questions. Select one to run it. Each question shows how many pages or bots match.

Grouped hitsCopy Anchor Link

The Grouped hits card of the Explore page lists the bot requests that match your filters. The ? next to the title says what the list shows and how many hits match.

Group hits byCopy Anchor Link

Select how to group the hits:

Grouped hits columnsCopy Anchor Link

Depending on the grouping, the table shows Bot, Category, Vendor, URL, Section, Network or IP prefix. Every view ends with Hits, Bytes, Response time and Last seen. Click the Hits or Last seen column header to sort. Click a URL to open the page drawer. See Pages.

Some cells show a reason instead of a value. Not resolved yet means the URL is not ready yet. (unsectioned) means the URL matches no section rule. Redacted at ingest means the IP address was not stored. No ASN recorded means the network is not known.

How Explore counts hitsCopy Anchor Link

Explore keeps every error and redirect (non-2xx) request in full. It keeps successful (2xx) requests as a 1% sample. When your filters include successful requests, the count shows At least in front of the number. The real figures are higher.

To get exact counts, filter to a specific error status code in the filter bar. Every non-2xx hit is kept in full, so these counts are exact.

Privacy in ExploreCopy Anchor Link

Server Log Monitoring does not store the IP address of unchecked and fake requests. Those requests have no IP prefix to group by. Query-parameter names are stored, but never their values.

TasksCopy Anchor Link

See every page that one bot fetchedCopy Anchor Link

1. Open Explore.
2. In the filter bar, add a Bot filter and select the bot.
3. In Grouped hits, select Page.
4. Click the Hits column header to sort by hits.

Find where fake Googlebot traffic comes fromCopy Anchor Link

1. Open Explore.
2. Add a Bot filter for Googlebot.
3. Add a Verification filter for Spoofed.
4. Select Network to group the hits by network.

TroubleshootingCopy Anchor Link

Why does Explore say "At least" before the number of hits?Copy Anchor Link

If your filters include successful (2xx) requests, Explore counts from a 1% sample of them. The number is a lower bound. Filter to an error status code to get exact counts.

Why does Explore show "Nothing matches these filters"?Copy Anchor Link

If no request matches your filters in the date range, the table is empty. Remove a filter, or pick a longer date range.

Why can I not group fake requests by IP prefix?Copy Anchor Link

If a request is unchecked or fake, Server Log Monitoring does not store its IP address. Group by Network instead.

RelatedCopy Anchor Link

• Server Log Monitoring - filters and verification.
• Bots - every bot and its drawer.
• Responses - status codes and URLs to fix.
• Pages - crawled pages and crawl budget.
• Glossary - the crawler and bot terms.