# Bots

The **Bots** page of Server Log Monitoring shows who crawls your site and if you can trust them. It has three tabs: **All bots**, **AI crawlers** and **Fake bots**.

## Before you start [#before-you-start]

- Connect a log source first. See [Connect server logs](/connect/server-logs).
- Open a domain. In the sidebar, select **Server Log Monitoring** > **Bots**.
- The tabs use the filters **Bot**, **Bot category**, **Vendor**, **AI crawler**, **Status code** and **Status class**. **All bots** and **Fake bots** also use **Verification**, but not together with an exact **Status code**.
- Members can read this page. Only owners and admins can change settings.

If Site Health Monitoring has open issues about fake bots or AI crawlers, a banner at the top of the Bots page links to them. See [Site Health Monitoring](/site-health/overview).

## All bots tab [#all-bots-tab]

The **All bots** tab of the Bots page lists every bot that visited your site in the date range. A bot that you do not know and that visits a lot can be worth blocking.

### All bots key numbers [#all-bots-key-numbers]

The card at the top has four tabs. Select a tab to change the chart below it.

| Tab | What it shows |
|-----|---------------|
| **Bot hits** | Every bot hit in the date range. The note shows the number of bots, or the change against the earlier period. The chart stacks the hits by bot category. |
| **Search engine share** | The share of bot hits that came from search engines. |
| **AI crawler share** | The share of bot hits that came from AI bots. |
| **Verified** | The share of hits that Server Log Monitoring confirmed came from the bot they name. It counts only bots that publish IP ranges or reverse-DNS names. |

With an exact **Status code** filter, **Verified** shows **Not available with a status-code filter**.

### All bots table [#all-bots-table]

The table lists one row per bot, busiest first. Click a column header to sort.

| Column | Meaning |
|--------|---------|
| **Bot** | The bot name. Hover to see the company (vendor) and the category. |
| **Hits** | Every hit that carried the bot's name. |
| **Trend** | The change against the earlier period. It shows when comparison is on. |
| **Share** | The bot's share of all bot hits. |
| **Verified** | The share of the bot's hits that came from its published addresses. **Not measured** if the bot publishes none. |
| **Response time** | The typical [time to first byte (TTFB)](/help/glossary#ttfb-time-to-first-byte). Three out of four hits were this fast or faster. |
| **Slow responses** | The TTFB of the slowest hits: 19 out of 20 hits were this fast or faster. |
| **Status mix** | The answers the bot got, by class. Sort by it to put the bots with the most errors first. |
| **Last seen** | The bot's most recent hit. |

If no bot visited, the table says **No bot hits in this window**.

## Bot drawer [#bot-drawer]

Click a bot in any Bots table to open the bot drawer. The header shows the bot name and an **Open in** menu. The menu links to **Explore** (every request from this bot) and **Robots.txt Monitoring**. For an AI bot, the Robots.txt link opens the AI bots page. For Googlebot, it also links to **Indexing Monitoring** and **Google Search Console**.

The drawer has three tabs:

- **Overview**: the tiles **Hits**, **Share of bot hits**, **Verified** and **Response time**, how the bot's requests were answered, and the **Hits over time** chart.
- **Status codes**: every exact status code the bot got, with **Answer**, **Hits** and **Share**.
- **Pages**: the pages the bot asked for. Click a page to open the page drawer. See [Pages](/server-logs/pages#page-drawer).

### Bot drawer for an AI bot [#bot-drawer-for-an-ai-bot]

For an AI bot, the Overview tab also shows the company as a whole. **Visitors sent** is the number of visitors that came from the company's AI products. **Crawls per visitor** divides all the company's crawls by those visitors. Both need [Web Analytics](/web-analytics/overview). Without it, they show **Needs Web Analytics data** or **No Web Analytics data in this period**.

If your robots.txt changed, the AI bot drawer can also show what changed for the company since that day. For example: **OpenAI since robots.txt changed on Oct 5, 2026: crawls -98%, visitors about the same**.

## AI crawlers tab [#ai-crawlers-tab]

The **AI crawlers** tab of the Bots page shows the AI bots that visit your site, and if your server lets them in. AI bots decide if AI tools can read and mention your pages.

### AI bots blocked by your server [#ai-bots-blocked-by-your-server]

If robots.txt allows an AI bot but your server refuses 50% or more of its hits with **403 Forbidden**, a warning shows. For example: **2 AI bots allowed by robots.txt are blocked by your server (403)**. A firewall or CDN rule usually causes this. If you want these bots to read your site, change that rule.

### AI crawlers key numbers [#ai-crawlers-key-numbers]

| Tab | What it shows |
|-----|---------------|
| **AI crawler hits** | Every hit from a bot of an AI company. Fake hits are included. The chart stacks the hits by AI bot category. |
| **AI crawler share** | The share of all bot hits that came from an AI bot. |
| **Answered 403** | The share of AI crawler hits that your server refused with **403 Forbidden**. |

With a status filter, **Answered 403** shows **n/a with a status filter**.

### What each AI company takes and sends back [#what-each-ai-company-takes-and-sends-back]

This card (the AI ledger) shows one row per AI company. Read what a company takes next to what it sends back before you decide who may read your site.

| Column | Meaning |
|--------|---------|
| **Company** | The AI company, for example OpenAI. **Other** groups all other AI companies. |
| **Trains on you** | Hits from bots that copy pages to train an AI model. |
| **Indexes for answers** | Hits from bots that read pages so an AI product can find and cite them. |
| **Fetched live for a user** | Hits from bots that opened a page because a person asked an AI product. |
| **Visitors sent (at least)** | Visitors from the company's AI product. This needs Web Analytics. |
| **Crawls per visitor** | All the company's crawls divided by the visitors it sent. |
| **Robots.txt** | What robots.txt says to the company's bots: **Allowed**, **Blocked** or, for example, **Blocks training**. |

### More about the AI ledger [#more-about-the-ai-ledger]

The crawl columns leave out fake hits. Hover a count to see **Verified** and **Claimed by user agent** hits. Claimed hits come from bots whose company publishes nothing to check them against.

For Google, Googlebot is not counted, and visitors are from Gemini only. AI Overviews and AI Mode send visits that look like ordinary Google search. Many AI assistants send no referrer, so the real number of visitors is higher.

To change what robots.txt allows, open **Robots.txt Monitoring** > **AI bots**. See [AI bots](/robots-txt-monitoring/ai-bots).

### Every AI crawler table [#every-ai-crawler-table]

The **Every AI crawler** card has four tabs that narrow the table by purpose:

- **All AI bots**: every AI bot with activity. The note can show how many AI bots robots.txt blocks, for example "3 of 12 blocked in robots.txt".
- **Training**: training crawlers. A training crawl is a licensing question.
- **Live fetch**: AI assistants. A live fetch means a person is asking a question right now.
- **Other**: AI search indexers and agents.

### Every AI crawler columns [#every-ai-crawler-columns]

| Column | Meaning |
|--------|---------|
| **Bot** | The AI bot. Click a row to open the bot drawer. |
| **Hits** | Every hit that carried the bot's name, fake hits included. |
| **Trend** | The change against the earlier period, with comparison on. |
| **Answered 403** | The share of the bot's hits that your server refused with **403 Forbidden**. |
| **Share** | The bot's share of the hits. |
| **Verified** | The share of hits from the bot's published addresses. |
| **Robots.txt** | What your robots.txt says to this bot at the site root: **Allowed** or **Blocked**. |
| **Follows robots.txt** | The share of the bot's hits that stayed within what your robots.txt allows. |
| **Bytes served** | The response bytes across hits that reported a size. It can be lower than the real amount. |
| **Last seen** | The bot's most recent hit. |

If your site sends more hits per day than VitalSentinel keeps in full, the **Follows robots.txt** column is hidden. A share measured on part of the data would read too high. If no AI bot visited, the table says **No AI crawler hits in this period**.

### Pages no AI crawler fetched [#pages-no-ai-crawler-fetched]

This card lists pages that Googlebot crawled without an error in the date range, but that no AI crawler fetched. Three tiles show **Read by Googlebot**, **Also read by AI** and **Missed by AI crawlers**. The table shows **URL**, **Googlebot hits** and **Last Googlebot crawl**, ordered by Googlebot hits.

Use this list to find content that Google sees but AI tools do not. Link to these pages from pages that AI crawlers read. The card needs one full day of data first. Until then, it shows **Not measurable yet**. This card does not use the filters.

## Fake bots tab [#fake-bots-tab]

The **Fake bots** tab of the Bots page shows visitors that use the name of a real bot, such as Googlebot, from a different address. They are usually scrapers or attackers. Server Log Monitoring never blocks them. You block them at your firewall or CDN.

### Fake bots key numbers [#fake-bots-key-numbers]

| Tab | What it shows |
|-----|---------------|
| **Fake hits** | Hits that used the name of a known bot but came from an address that the bot does not use. |
| **Failed the check** | Hits that name a bot that publishes ranges, but came from outside them. They are not flagged fake, because the ranges may be out of date. |
| **Cannot be checked** | Hits from bots that publish no IP ranges or reverse-DNS names. |
| **Verified** | The share of hits confirmed as the real bot, among bots that can be checked. |

With an exact status code filter, these tiles show **Not available with a status-code filter**.

### Faked identities [#faked-identities]

The **Faked identities** table lists each bot name that was faked. It shows hits that claimed to be Googlebot, Bingbot or an AI crawler from an address that the bot does not use.

| Column | Meaning |
|--------|---------|
| **Claimed identity** | The bot name that the fake hits used. |
| **Fake hits** | The number of fake hits with that name. |
| **Share of fakes** | The share of all fake hits. |
| **Networks** | How many networks the fake hits came from. |

Click a row to open the fake bot drawer. It has two tabs: **Networks** (the networks this name came from) and **Pages** (the pages the fake hits asked for). If no fake hits occurred, the table says **No fake bots detected**.

### Networks [#networks]

The **Networks** table lists every network that sent bot hits in the date range. A **network** is the operator that routed the request, shown by its AS number (ASN). For example, AS15169 is Google.

| Column | Meaning |
|--------|---------|
| **Network** | The network name and AS number. |
| **Hits** / **Share** | Bot hits from this network, and their share. |
| **Bot names** | How many different bot names sent hits from this network. |
| **Fake** | The share of this network's hits that were fake. |
| **Not confirmed** | The share of hits that could not be confirmed as the real bot. |
| **Last seen** | The most recent hit from this network. |

Many different bot names from one network is a warning sign. A network that you do not know and that fakes a bot is a good one to block.

## Tasks [#tasks]

### Find out why an AI bot cannot read your site [#find-out-why-an-ai-bot-cannot-read-your-site]

1. Open **Bots** and select the **AI crawlers** tab.
2. In **Every AI crawler**, find the bot.
3. Compare the **Robots.txt** column with the **Answered 403** column.
4. If **Robots.txt** shows **Allowed** and **Answered 403** is high, check your firewall or CDN rules.
5. If **Robots.txt** shows **Blocked**, change your robots.txt in Robots.txt Monitoring.

### Block a fake Googlebot [#block-a-fake-googlebot]

1. Open **Bots** and select the **Fake bots** tab.
2. In **Faked identities**, click the row for Googlebot.
3. In the drawer, select the **Networks** tab.
4. Block the busiest networks at your firewall or CDN.

## Troubleshooting [#troubleshooting]

### Why does a bot show "Not measured" in the Verified column? [#why-does-a-bot-show-not-measured-in-the-verified-column]

If the bot publishes no IP ranges or reverse-DNS names, Server Log Monitoring cannot confirm its hits. This is normal for several AI crawlers. It does not mean the hits are fake.

### Why are AI crawlers blocked by my server? [#why-are-ai-crawlers-blocked-by-my-server]

If the **Answered 403** share of an AI bot is high, your server refuses it with **403 Forbidden**. A firewall, a bot-protection feature or a CDN rule usually does this. Allow the bot in that tool if you want AI products to read your site.

### Why is the Follows robots.txt column missing? [#why-is-the-follows-robotstxt-column-missing]

If your site sends more hits per day than VitalSentinel keeps in full, Server Log Monitoring hides the column. A share measured on part of the data would be too high.

### Why do the AI ledger and the Every AI crawler table show different hits? [#why-do-the-ai-ledger-and-the-every-ai-crawler-table-show-different-hits]

The **What each AI company takes and sends back** card leaves out fake hits. The **Every AI crawler** table counts every hit that carried the bot's name, fake hits included.

## Related [#related]

- [Server Log Monitoring](/server-logs/overview) - how verification and AI categories work.
- [AI bots](/robots-txt-monitoring/ai-bots) - set which AI bots may read your site.
- [Explore](/server-logs/explore) - every request from one bot.
- [Pages](/server-logs/pages) - the pages bots crawl.
- [AI readability](/site-health/ai-readability) - if AI crawlers can get in and read your pages.
- [Glossary](/help/glossary#verified-bot) - verified, unverified and spoofed bots.