Skip to content
SEOComet

Hear about a broken or shrinking sitemap before search engines do

SEOComet reads your sitemaps and sitemap indexes every hour on Free, down to every 30 minutes on Business, counts their URLs and lastmod dates, and alerts you when one goes missing, stops parsing, loses URLs or stops updating. Google News sitemaps are read more often: every hour on Free, down to every 10 minutes on Business.

Mock-up of a sitemap check in SEOComet: the sitemap index now lists 10,740 URLs, down from 18,204, across 12 child sitemaps, and the latest check raised a warning that it lost 41% of its URLs.

What it catches

Sitemaps fail quietly. A plugin update writes broken XML, a cache serves an HTML page in the sitemap’s place, or a deploy drops half the URLs. These are the issues SEOComet raises.

Issues the sitemap check raises
Sitemap missing Critical The sitemap was there before, or robots.txt lists it, and now the address answers 404 or 410, so search engines can’t read it either. One that never existed raises nothing.
Sitemap can’t be read Critical The file isn’t valid XML, or its root is neither urlset nor sitemapindex. A plugin update, or a cache serving an HTML page instead, are the usual causes.
Sitemap is empty Critical The sitemap was read correctly but lists no URLs at all, which usually means a build or plugin failed.
Fewer URLs in the sitemap WarningCritical Critical past twice your threshold The sitemap lists far fewer URLs than on the last complete check. The threshold starts at a 30% drop and you can change it. Not raised for news sitemaps.
A child sitemap failed Warning A sitemap index lists other sitemaps, and at least one of them couldn’t be fetched or read. Their URLs may drop out of search.
Couldn’t check the address Warning The sitemap answered with an error other than 404 or 410, or the request timed out or failed.
Sitemap looks stale Info The newest lastmod date is older than your limit (30 days to start with), so the sitemap may have stopped updating. Not raised for news sitemaps.
Index listed inside an index Suggestion A sitemap index lists another index, which Google skips there. Nothing is lost if the inner index is also in robots.txt or Search Console. Shown on the check, with no incident or alert.

How it reads a sitemap

SEOComet fetches the sitemap, checks the XML, and counts the URLs and the newest lastmod. It doesn’t visit the URLs, except the newest articles in a news sitemap, and it doesn’t store the file, only the counts and dates.

For a sitemap index it fetches the child sitemaps one at a time, up to 200 per check. Child sitemaps on another site aren’t fetched and count as failed. If a child fails, or the check runs out of time or reaches the cap, the count is marked incomplete. Only complete counts are compared, so a slow night doesn’t look like a lost half of your site.

Sitemaps can be up to 50 MB each and get 30 seconds to load.

Where sitemaps come from
The Sitemap: lines in robots.txt for your domain and its subdomains, up to 50, or /sitemap.xml if there are none, watched quietly until it exists. Add any other sitemap on your domain or its subdomains by hand.
URL count drop
A warning past 30%, critical past 60%. Set your own threshold on each sitemap’s Settings tab.
Stale after
30 days without a newer lastmod. Raise it for a site that rarely changes.
Check interval
Every hour on Free and Pro and every 30 minutes on Business.

Google News sitemaps

A sitemap with Google News tags is checked as a news sitemap, with no setup. Google News only reads articles from the last 48 hours, so the checks are about freshness, not size.

Issues the news sitemap check raises
News sitemap stopped updating Critical The newest article is older than your limit (6 hours to start with). Google says to add articles to the news sitemap as they're published, so new stories may not be reaching Google News. Google: News sitemap best practices
New article not indexable Warning One of the 5 newest articles didn’t load, is set to noindex, or has a canonical that points to another address, on two checks in a row. Google: Block indexing with noindex , Canonical URLs
Articles older than 48 hours Warning Google says a news sitemap should only list articles from the last two days. Older ones should come out, or lose their news tags. Google: News sitemap best practices
Too many URLs for a news sitemap Warning Google allows at most 1,000 news entries in one news sitemap and says to split a bigger one into several. Google: News sitemap reference
News tags missing Warning Some URLs lack a tag Google requires: the publication’s name or language, the title, or a valid publication date. Google: News sitemap reference
Stale after
6 hours without a new article to start with. Set 1 to 72 on the check’s Settings tab.
Newest articles
Each check opens the 5 newest articles to make sure they load, aren’t noindex and don’t point their canonical somewhere else.
Check interval
Every hour on Free, every 15 minutes on Pro and every 10 minutes on Business.

A news sitemap shrinks every weekend as old articles leave it, so the URL count drop and lastmod checks don’t apply to it.

Join the SEOComet waitlist

SEOComet isn’t open to the public yet. Leave your email and we’ll tell you when you can add your sites. The SEO Toolbar Chrome extension is free and works today.