Overview
Sitemap Monitoring reads your XML sitemaps every day, shows how many URLs they list, and tells you which URLs were added, removed or updated.
Sitemap Monitoring reads your XML sitemaps every day and tells you what changed in them. A sitemap is the list of pages that you want search engines to find. If a deploy removes half of your sitemap URLs, or a sitemap file stops loading, Google can stop finding your pages.
What Sitemap Monitoring does
Sitemap Monitoring needs no setup. It starts after you verify that you own the domain.
- VitalSentinel finds your sitemaps. See How Sitemap Monitoring finds your sitemaps.
- VitalSentinel reads every sitemap file once a day.
- VitalSentinel compares each daily check with the check before it. It records the URLs that are new, removed, back in the sitemap or updated.
- VitalSentinel checks each file against the official sitemap limits and rules. See Files.
Sitemap Monitoring has no settings. It has one page with three tabs: Overview, URLs and Files. Open it from Crawling & availability > Sitemap Monitoring in the sidebar.
| Tab | What it shows |
|---|---|
| Overview | The number of URLs over time and what changed between two checks. This page. |
| URLs | Every URL in your sitemaps, with what other modules know about it. See URLs. |
| Files | Every sitemap file, its status and its limit findings. See Files. |
How Sitemap Monitoring finds your sitemaps
VitalSentinel finds your sitemaps automatically. The Found in column of the Files tab shows where:
| Found in | Meaning |
|---|---|
| robots.txt | A Sitemap: line in your robots.txt. |
| Google Search Console | A sitemap that you submitted in Google Search Console. |
| Common location | A standard address, for example /sitemap.xml. |
| Listed in … | A sitemap that a sitemap index lists. |
You cannot add a sitemap by hand in VitalSentinel. To add a sitemap, add a Sitemap: line to your robots.txt, or submit the sitemap in Google Search Console. VitalSentinel looks again every day.
If a sitemap file returns "not found" for 7 days in a row, VitalSentinel stops checking it. Its status then shows No longer checked.
How Sitemap Monitoring protects you from false removals
Sitemap Monitoring does not report a URL as removed by mistake:
- If a sitemap file cannot be downloaded, VitalSentinel keeps its URLs. They show Held – file not read and do not count as removed until the file downloads again.
- If a file suddenly loses many URLs, VitalSentinel downloads it a second time. If the second download does not confirm the loss, the check shows Listing shrank and the URLs are kept.
- If the counts of a check do not add up with the check before it, the check shows Counts did not add up. Nothing is counted for that check.
Data from other modules
Sitemap Monitoring does not open your pages. It uses data from other modules to show more about each URL:
| Data | Source |
|---|---|
| HTTP status, redirects, noindex and canonical | Site Health Monitoring checks |
| Whether Google indexed the URL | Indexing Monitoring |
| Googlebot visits and sitemap reads | Server Log Monitoring |
| Google clicks | Google Search Console |
A column or a tab that needs a source that you did not connect shows what it needs, for example Needs Server Log Monitoring.
First check and module states
While the first check runs, Sitemap Monitoring shows Checking your sitemaps for the first time. On a large site this can take a while. The page updates by itself.
After the first check, the page can show Monitoring starts with the next check. The Files tab already shows the files. Changes show from the next check. Until then, the What changed card shows "Changes show here after the next check".
If the daily check is late, a note at the top says "The daily check is running late. Your last check was …".
No sitemap found
If VitalSentinel finds no sitemap, the page shows Publish a sitemap to see what changes in it. Add a Sitemap: line to robots.txt, or put a sitemap at /sitemap.xml. The Open Robots.txt Monitoring button shows what VitalSentinel read from your robots.txt.
We couldn't download your sitemap
If the first check fails, the page shows We couldn't download your sitemap with the reason, for example "returned 403 (access denied)". Make sure that the sitemap opens in a browser. VitalSentinel tries again and shows when.
If your server or firewall refuses the VitalSentinel crawler, the page shows Let our crawler in to see what changes in your sitemap. Follow the steps in the red banner at the top of the page. See Monitoring IPs.
Sitemap size
The top card of the Sitemap Monitoring Overview has four tiles. They show the newest check in the selected date range:
| Tile | What it shows |
|---|---|
| URLs listed | How many URLs your sitemaps list. The note compares it with the previous check, for example "+120 vs previous check". |
| New | URLs listed for the first time at the check. |
| Removed | URLs that left every sitemap at the check. A URL whose file could not be downloaded is never counted here. |
| Files not read | Sitemap files that the check could not download or read. Their URLs are kept. |
If files were not read, the card shows a button such as See the 2 files not read. It opens the panel Sitemap files not read. See Files.
The size chart
The chart shows the selected tiles over the checks in the date range. Click a tile to add it to the chart or to remove it. You can show up to two. The second one uses the axis on the right, so the heights of the two lines are not comparable.
A mark with ! shows a check in which some files could not be read. Click a point of the chart to compare that check with the one before it in What changed. The chart shows after the second check.
What changed
The What changed card of Sitemap Monitoring lists the URLs that changed between two checks. By default, it compares the checks at the start and the end of the date range. Hover the ? to see the two dates.
To compare one check with the check before it, click a point of the chart or a row of Checks. To go back, click Back to the date range.
Change tabs
The tabs above the list show the totals for each type of change. Click a tab to filter the list. The totals ignore your filters. The list follows them.
| Tab | Meaning |
|---|---|
| All changes | Every URL that changed between the two checks, counted once. |
| New | Listed in your sitemaps for the first time. |
| Removed | No longer listed in any sitemap. |
| Updated | Its last modified date (lastmod) moved forward. |
| Other changes | Back in sitemap, In a different sitemap, Date went backwards, Date now given or Date no longer given. |
If your sitemaps give no dates, Updated shows No dates in your sitemaps. The Updated tab also shows a Google column: whether verified Googlebot fetched the page after its date moved. This needs Server Log Monitoring.
Columns and chips
The list shows URL, Change, Sitemap file, Last modified, Status and Clicks (Google clicks in the date range). A column with the same value on every row is left out. Click a row to open the URL panel. See URLs.
Some URLs show a chip:
| Chip | Meaning |
|---|---|
| Held – file not read | Its sitemap file did not download. VitalSentinel keeps the URL and does not count it as removed. |
| Sitemaps disagree on the date | Two sitemaps give different dates. It counts as updated only when they agree. |
| Date in the future | Its last modified date is more than a day after the check. |
| Keeps coming and going | It left and came back, or its date changed back and forth, 3 or more times in 14 days. |
Still getting clicks
Click Still getting clicks to see the removed URLs that Google still sends people to. The clicks are for the date range. Put a URL back in the sitemap if the removal was a mistake, or redirect it to the closest page.
This list needs Google Search Console. If no removed URL gets clicks, the card shows "No removed URLs still get search clicks". Click Back to all changes to see every change again.
Checks and filters
Lower on the Sitemap Monitoring Overview, the Checks table lists every daily check in the date range. The columns are Checked, URLs listed, New, Removed, Back in sitemap, Updated and Files not read. A dash means that the value was not measured. A check that was not counted shows a chip, for example Files failed. Click a row to compare that check with the one before it.
The filter bar at the bottom of the Sitemap Monitoring Overview has the date range and the change filters. The date range is the last 28 days by default. You can select up to 12 months. The date range changes the chart, the tiles and What changed. The change filters narrow only the list of changes. For example, Keeps coming and going can hide those URLs or show only them.
Issues in Site Health Monitoring
Sitemap Monitoring does not keep its own list of problems. Problems such as unreadable files, a large drop in URLs, broken sitemap URLs and limit findings open as issues in Site Health Monitoring. A banner at the top of the page shows the open sitemap issues and links to them.
Troubleshooting
How often does Sitemap Monitoring check my sitemaps?
Once a day. Each daily check adds one point to the chart.
How do I add a sitemap to Sitemap Monitoring?
Add a Sitemap: line to your robots.txt, or submit the sitemap in Google Search Console. VitalSentinel looks for new sitemaps every day. You cannot add a sitemap by hand.
Why does Sitemap Monitoring show no sitemap?
If the page shows Publish a sitemap to see what changes in it, VitalSentinel found no sitemap. Add a Sitemap: line to robots.txt, or put a sitemap at /sitemap.xml. If the page asks you to let the crawler in, allow the VitalSentinel crawler in your firewall. See Monitoring IPs.
Why are URLs not counted as removed?
If a sitemap file could not be downloaded, its URLs show Held – file not read. They count as removed only if the file downloads again without them.
Why does What changed show "The date range holds fewer than two checks"?
What changed compares two checks. Select a longer date range in the filter bar.
Related
- URLs - every URL in your sitemaps and its panel
- Files - every sitemap file, its status and its limits
- Robots.txt Monitoring - the
Sitemap:lines and the rules that block sitemap URLs - Indexing Monitoring - which sitemap pages Google indexed
- Server Log Monitoring - Googlebot's reads of your sitemaps and pages
- Site Health Monitoring issue types - the sitemap issues and how to fix them
Settings
The Settings page of Server Log Monitoring manages the log source, shows the monthly hit allowance and excluded traffic, and links to site sections.
URLs
The Sitemap Monitoring URLs tab lists every URL in your sitemaps, with broken, not indexed and not visited URLs, and a panel with each URL's history.
