Responses
The Responses page of Server Log Monitoring shows how your server answers bots, which URLs give errors or redirects, and when bots give up waiting.
The Responses page of Server Log Monitoring shows how your server answers bots. A healthy answer is 200 OK. Redirects and errors waste a bot's visit, so fix the biggest group first. The page has two tabs: All responses and Gave up waiting.
Before you start
- Connect a log source first. See Connect server logs.
- Open a domain. In the sidebar, select Server Log Monitoring > Responses.
- All responses uses the Bot, Bot category, Status code, Status class and Verification filters. Verification cannot be combined with an exact Status code.
- Gave up waiting uses the Bot, Bot category, Verification, Section, Content type and Response time (ms) filters.
- Members can read this page.
A status code is the three-digit answer your server gives to each request. 2xx means success, 3xx a redirect, 4xx a client error (for example 404 Not found) and 5xx a server error.
All responses tab
The All responses tab of the Responses page shows every answer that bots got. From top to bottom, it shows a verdict, a warning about gave-up requests and an issues banner. Then it shows the card with the answers that bots got, and the cards URLs with errors and redirects and How bots were served.
Where the problem comes from
If errors are 5% or more of bot hits, or redirects are 10% or more, a verdict names the bots behind them. For example: "62% of the errors come from Bingbot". If most errors come from hits that only claim to be a bot, the verdict says Most errors go to unverified hits that use a bot's name. The line under the verdict can add, for example, "Real Googlebot got no error on 99% of visits".
Requests that ended with a 499
If bots closed the connection before your server answered, a red warning shows. For example: 120 requests ended with a 499. Click Open Gave up waiting to see the details. See Gave up waiting tab.
Bot answers card
The card with the answers that bots got has three tabs:
| Tab | What it shows |
|---|---|
| Failure rate | The share of hits answered with a client error (4xx) or a server error (5xx). Fine under 5%, High at 5% and over. |
| Redirect share | The share of hits answered with a redirect, not counting 304. Fine under 10%, Worth a look at 10% and over. |
| Bot hits | Every bot hit, with one line per exact status code. |
A 304 Not Modified is a healthy answer. It does not count as a redirect in the Redirect share tile. In the By class chart, 3xx includes 304.
Bot answers chart controls
Above the chart, select Exact codes for one line per status code, or By class for one line per class. Select Hour, Day, Week or Month to group the hits. The hour view covers up to 92 days. Relative shows each line as a share of all hits.
With a Verification filter, the chart cannot show hours. It says Verification is only known per day. Group by Day, Week or Month to see this filter on the chart.
URLs with errors and redirects
The URLs with errors and redirects card lists every URL that crawlers got an error or a redirect for, one row per URL. It has four tabs:
| Tab | What it lists |
|---|---|
| Not found / Blocked (4xx) | URLs that got a client error, such as 404 Not found or 410 Gone. |
| Server errors (5xx) | URLs where your server failed to answer, such as 500 or 503. |
| Redirected (3xx) | URLs that sent crawlers somewhere else. A 304 Not Modified does not count. |
| Fixed | URLs that gave Googlebot an error and now give it a normal page. |
Start with the URLs that Googlebot hits most, and with errors on URLs that are in your sitemap.
URLs with errors and redirects columns
| Column | Meaning |
|---|---|
| URL | The page. The chip In sitemap shows if the URL is in your sitemap. |
| Status | The status code. |
| Googlebot | Verified Googlebot hits. |
| Other bots | Hits from other crawlers in the scope. |
| Last seen | The most recent hit with this answer. |
| Impressions | Google Search Console impressions. It needs Google Search Console. |
| Visits | Web Analytics visits. It needs Web Analytics. |
Click Filter the URLs to narrow the list by Bot, Status code or Status class. These filters apply to this card only, in addition to the filters of the filter bar. Each row has a Watch in Page Monitoring action and an Open in menu. Click a URL to open the page drawer. See Pages.
Fixed tab
The Fixed tab looks at the last 14 days of verified Googlebot visits. It lists URLs that gave Googlebot an error (4xx or 5xx) and now give it a normal page. A redirect that became a normal page is not listed. The columns are Googlebot got, Now it gets, Fetches since and Search impressions. The date range sets only the search impressions column.
If no URL was fixed, the tab says No fixed URLs in the last 14 days. If there are no Googlebot logs, it says No Googlebot logs yet.
How bots were served
The How bots were served card shows if each bot got its answer from your cache (CDN) or from your server, and how long it took. A CDN is a network of servers that keeps copies of your pages close to visitors.
| Tab | What it shows |
|---|---|
| Answered from cache | Answers the cache gave without asking your server. |
| Answered by your server | Answers your server had to build or check. |
| Time when cached vs not | The response time from the cache and from your server. |
The table shows Bot, Served how, Answers, From cache, By your server, Time from cache and Time from your server. A share needs at least 30 answers with a cache result. Only log sources that report a cache result count. If none does, the card says Server Log Monitoring has no log source that reports cache results.
Cache results in How bots were served
The cache results are your CDN's own words:
| Result | Meaning |
|---|---|
| Hit | Answered straight from the cache. |
| Old copy / Old copy while refreshing | The cache sent a copy that had expired. |
| Hit at the shield | Answered from the CDN's origin shield cache, not from your server. |
| Miss | The cache did not have the page, so your server built the answer. |
| Expired | The cache had the page but it was out of date, so your server was asked again. |
| Rechecked | The cache asked your server if its copy was still good. |
| Bypass | The cache was skipped, for example because of a cookie, a header or a rule. |
| Dynamic | The CDN does not cache this kind of page, so your server answers every time. |
Gave up waiting tab
The Gave up waiting tab of the Responses page shows requests where the bot closed the connection before your server answered. Logs record these requests with status 499. This almost always means that your server was too slow.
For Googlebot, this means that Google gave up waiting. Google did not fetch the page, so the crawl was wasted. Google also slows its crawl rate when a server stays slow.
Gave up waiting verdict
The verdict shows when verified Googlebot gave up on at least 20 requests and 1% of its visits. It also shows when bots gave up on 5% of all requests. For example: "Googlebot gave up on 45 requests" (red) or "Bots gave up on 6% of all requests" (amber).
If your log source cannot record a gave-up request, a note says This log source cannot record a gave-up request, with the reason.
Gave up waiting key numbers
| Tab | What it shows |
|---|---|
| Gave up | Requests closed before an answer. |
| Share of requests | Gave-up requests as a share of every bot request. |
| Googlebot gave up | Gave-up requests from verified Googlebot only. |
| Typical wait | The median time bots waited before they hung up. Your logs record no duration if your logs have no timing. |
The chart splits the requests into Googlebot, Claims to be Googlebot, Other search engines, AI crawlers and Other bots.
Which pages they gave up on
The Which pages they gave up on table lists the pages, most often first. Speed up the pages with the most clicks first.
| Column | Meaning |
|---|---|
| URL | The page. |
| Gave up | Gave-up requests, split into verified Googlebot and everyone else. |
| Of its requests | Gave-up requests as a share of every request for this page. |
| Typical wait | The median time bots waited before they hung up. |
| Answers in | The median time the page took when it did answer (2xx). Detail is kept for about a week. |
| Last gave up | The most recent gave-up request. |
| Clicks | Google Search Console clicks. It needs Google Search Console. |
Who gave up
The Who gave up table lists every bot that hung up before your server answered. The columns are Bot, Verified (Verified or Not verified), Gave up, Of its requests, Typical wait and Last seen. A real search bot on this list matters most. Click a bot to open the bot drawer.
When they gave up
The When they gave up chart shows gave-up requests by hour of the day, for Googlebot and Everyone else. A spike at the same hour every day usually means that a scheduled job, a backup or a traffic peak slows the server. The tooltip can show No cached copy and Outages from Uptime Monitoring. Without Uptime Monitoring, a link says Set up Uptime Monitoring.
Tasks
Fix the URLs that give Googlebot errors
- Open Responses and stay on the All responses tab.
- In URLs with errors and redirects, select Not found / Blocked (4xx).
- Sort by Googlebot.
- Fix or redirect the top URLs.
- After a few days, select the Fixed tab to confirm that Googlebot now gets a normal page.
Find out why bots give up waiting
- Open Responses and select the Gave up waiting tab.
- In When they gave up, look for an hour with a spike.
- If the spike is at the same hour each day, check your server for a scheduled job at that time.
- In Which pages they gave up on, speed up the pages with the most clicks.
Troubleshooting
Why is the Gave up waiting tab empty?
If no bot closed a connection early, the tab says No bot gave up waiting in this period. If the note This log source cannot record a gave-up request shows, your log source does not record status 499.
Why does the Bot answers chart not show hours?
If you use the Verification filter, the chart can show only days, weeks or months. If your date range is longer than 92 days, the hour view is not available.
Why does How bots were served show "Not measured"?
If your log source does not report a cache result, the card cannot split cache and server answers. If a bot has fewer than 30 answers with a cache result, it shows Too few answers with a cache result.
Why does the Redirect share not match the By class chart?
The Redirect share tile does not count 304 Not Modified, because it is a healthy answer. The By class chart counts 304 in 3xx.
Related
- Server Log Monitoring - status mix, response time and verification.
- Pages - crawl waste and the page drawer.
- Explore - every request with a given status code.
- Uptime Monitoring - outages that can make bots give up.
- Page Monitoring - watch a URL that gives errors.
Pages
The Pages page of Server Log Monitoring shows which URLs bots crawl, how much crawl budget is wasted, and which parts of your site bots never reach.
Crawl vs results
Crawl vs results in Server Log Monitoring puts your bot crawl next to Google Search Console, Web Analytics, RUM Monitoring, Sitemap Monitoring and Page Monitoring.
