# Settings



To open the settings, select **Site Health Monitoring** > **Settings** in the sidebar. The settings have three tabs:

| Tab                                   | What you set                                                                                         |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [**Pages to check**](#pages-to-check) | How many pages VitalSentinel checks, which pages come first, page rules, pinned pages and robots.txt |
| [**Exceptions**](#exceptions)         | Issue types that are normal for your site, and AI bots that you block or allow on purpose            |
| [**Crawler access**](#crawler-access) | Whether VitalSentinel can reach your site, and how to allowlist the crawler                          |

Only workspace owners and admins can change the settings. Other members see "Ask a workspace owner or admin to change this."

## Pages to check [#pages-to-check]

### How many pages we check [#how-many-pages-we-check]

The subtitle of this card tells you how VitalSentinel checks your site now. For example, "We check all 1,200 pages in your sitemap every week."

* If your sitemap has the same number of pages as the limit or fewer, VitalSentinel checks every page in it.
* If your sitemap has more pages, VitalSentinel checks a selection. You decide which pages come first.

VitalSentinel checks the pages at the speed of your plan. If you check fewer pages, each page gets a check more often.

To change the settings:

1. In **Pages to check**, enter the number of pages. The minimum is 100. The maximum is the limit of your plan (see [Plans and limits](/site-health/introduction#plans-and-limits)). The hint under the field shows how often each page will get a check.

2. In **Which pages come first**, select an option:

   | Option                      | Pages that come first                                                                                            |
   | --------------------------- | ---------------------------------------------------------------------------------------------------------------- |
   | **Balanced**                | A mix of your important pages and pages from every section of your site                                          |
   | **Most clicks from Google** | The pages that get the most visitors from Google. Needs [Google Search Console](/features/google-search-console) |
   | **Most visited**            | The pages that your visitors view most. Needs [Web Analytics](/features/web-analytics)                           |
   | **Pages in your sitemap**   | The pages that your sitemap lists                                                                                |

3. Click **Save**.

**Which pages come first** has an effect only when your sitemap has more pages than VitalSentinel checks.

### Page rules [#page-rules]

Page rules include or exclude pages by their address. There are three types of rule:

| Rule             | What it does                                                                                                                                                                         |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Only check**   | VitalSentinel checks only the pages that match. Without this rule, it selects from your whole site.                                                                                  |
| **Always check** | The pages that match are always in the selection.                                                                                                                                    |
| **Never check**  | VitalSentinel never selects or opens the pages that match. It also never adds them to your watched pages automatically. A page that you watch in Page Monitoring still gets a check. |

To add a rule:

1. Click **Add rule**.
2. In **What should happen**, select the type of rule.
3. In **Pages matching**, enter a pattern.
4. Click **Add rule**.

The rule saves immediately. To remove a rule, click the remove button on its row.

**Patterns:**

* A pattern matches the path and the query from the start. For example, `/blog/` matches `/blog/a?x=1`.
* `*` matches any characters. End the pattern with `$` to match exactly.
* Patterns are case-sensitive and cannot contain spaces.

| Pattern     | Matches                   |
| ----------- | ------------------------- |
| `/blog/*`   | Every page under `/blog/` |
| `/pricing$` | Only `/pricing`           |
| `/*?sort=`  | Sorted listings           |

You can add up to 50 patterns for each type of rule.

### Pinned pages [#pinned-pages]

Pinned pages are always in the selection, and they get a check more often than the other pages. Pin the pages that you can least afford to break, for example your homepage and checkout.

To pin a page:

1. Click **Pin a page**.
2. In **Page address**, enter the full address of a page on this site.
3. Click **Pin page**.

You can also pin a page from the **Pages** card on the Overview. Click **Check this page every day** on its row.

The header shows how many pins you used, for example "3 of 50 pinned". The number of pins depends on your plan. To remove a pin, click **Unpin**.

<Callout type="info">
  A pin does not send an alert when the page changes. To get an alert when a page changes, watch it in [Page Monitoring](/page-monitoring/introduction).
</Callout>

### robots.txt [#robotstxt]

The VitalSentinel crawler always obeys your robots.txt. It never opens a page that robots.txt blocks for it. You cannot change this.

Turn on **Include pages robots.txt blocks** to list and check pages that robots.txt hides from Google, for example a page in a blocked folder. Pages that robots.txt blocks for the VitalSentinel crawler stay in the list, but the crawler never opens them. For these pages, VitalSentinel can only tell you that they are blocked. The change takes effect in one day or less.

### Phone site [#phone-site]

This card shows only when your homepage sends phone visitors to a different host, for example `m.example.com`. Use it to also check that host.

## Exceptions [#exceptions]

### Normal for this site [#normal-for-this-site]

This card lists the issue types that you told VitalSentinel are normal. VitalSentinel does not raise them again.

A rule comes from one of two places:

* You selected **This is normal…** > **On every page, now and later** on an issue.
* You added a rule on this card.

To add a rule:

1. Click **Add rule**.
2. In **Issue type**, select an issue type, or **Any issue type**.
3. Optional: in **Only on pages matching**, enter a pattern, for example `/category/*`. Leave it empty for every page. `*` matches any characters.
4. In **How long**, select **Never ends**, **For 7 days**, **For 30 days** or **For 90 days**.
5. Click **Add rule**.

To change a rule, click the edit button on its row, change the values, then click **Save changes**. To delete a rule, click **Delete**. VitalSentinel then raises the matching issues again.

### Skip robots.txt alerts for AI bots [#skip-robotstxt-alerts-for-ai-bots]

If you block or allow some AI bots on purpose, turn them on here. VitalSentinel then does not raise an issue when your robots.txt changes for them. Their verdicts stay on the [AI readability](/site-health/ai-readability) page.

| Group                     | Bots in the group                                        |
| ------------------------- | -------------------------------------------------------- |
| **AI training bots**      | Bots that collect pages to train AI models               |
| **AI search bots**        | Bots that build the indexes behind AI answers            |
| **AI assistant fetchers** | Bots that fetch a page when a person asks an AI about it |

To select single bots, open **Choose single bots** on a group, then turn on each bot.

## Crawler access [#crawler-access]

### Crawler access card [#crawler-access-card]

This card shows whether VitalSentinel can reach your site now:

| State               | Meaning                                                                                              |
| ------------------- | ---------------------------------------------------------------------------------------------------- |
| **Healthy**         | VitalSentinel can reach your site and checks it at the normal speed.                                 |
| **Unreachable**     | Your site does not answer. The checks stop, and VitalSentinel tries again on a schedule.             |
| **Blocked**         | Your site or its firewall turns the crawler away. The checks stop until the crawler can get through. |
| **Not checked yet** | VitalSentinel did not try your site yet. The first check runs soon.                                  |

When the state is not **Healthy**, the card shows **Next check**, **Last check** and **Last success**. When the state is **Blocked**, it also shows **Blocked by** (the firewall, if VitalSentinel knows it) and **Where** (**The whole site** or **Some pages**).

To check now, click **Check now**. When the crawler is blocked, the button is **I’ve allowlisted you – check again**. The result shows in about 30 seconds.

### Allowlisting our crawler [#allowlisting-our-crawler]

If a firewall, CDN or security plugin blocks the crawler, allow it:

|                      |                                                              |
| -------------------- | ------------------------------------------------------------ |
| **User agent**       | `VitalSentinel Bot/1.2 (+https://www.vitalsentinel.com/bot)` |
| **robots.txt name**  | `VitalSentinel-Bot`                                          |
| **Our IP addresses** | See [Monitoring IPs](/others/monitoring-ips)                 |

Match the user agent on the text `VitalSentinel Bot`, without the version number. Then your rule works for future versions too.

When the crawler is blocked, the card also shows numbered steps for your firewall, if VitalSentinel knows it. For example, Cloudflare, Akamai, Imperva, AWS WAF, Sucuri, Wordfence, Vercel or Azure WAF. For any other firewall or bot filter:

1. Allow the requests whose user agent contains `VitalSentinel Bot`.
2. Also allow the VitalSentinel IP addresses, so that nobody can copy the rule with a fake user agent.
3. Save the rule. The next check finds the change.

For more steps, see [How to allowlist](/others/monitoring-ips#how-to-allowlist).

## Related [#related]

<Cards>
  <Card title="Overview" href="/site-health/overview">
    The Health score, issues and pages
  </Card>

  <Card title="Monitoring IPs" href="/others/monitoring-ips">
    VitalSentinel IP addresses and user agents
  </Card>
</Cards>