Catch the robots.txt change that takes your site out of search
SEOComet fetches your robots.txt every hour, keeps every version, and alerts you when a change blocks Googlebot, a path you care about or a crawler on your watch list.
What it catches
Each problem opens an incident with a severity, so your alert rules can treat a blocked Googlebot differently from a harmless edit.
| Issue | Severity | What it means |
|---|---|---|
Search engines blocked robots.disallow_root_added | Critical | robots.txt now tells every crawler, or Googlebot or Bingbot, not to crawl any page. The site can drop out of search results. |
Important page blocked robots.important_path_blocked | Critical | A path you marked as important is now disallowed for every crawler or for Googlebot. |
Crawler blocked robots.watched_agent_blocked | Warning | A crawler on your watch list, such as GPTBot or ClaudeBot, is now blocked from the whole site. Crawlers that were already blocked on the first check are taken as deliberate. |
Sitemap line removed robots.sitemap_line_removed | Warning | A Sitemap: line that robots.txt used to list is gone, so crawlers may stop finding that sitemap. |
robots.txt missing robots.not_found | Warning | robots.txt answers 404 or 410, or a web page instead of a text file, so crawlers see no rules at all. |
robots.txt answers with an error fetch.failed | CriticalWarning Critical for a server error or 429 | Google treats a robots.txt that answers with a server error, or 429, as blocking the whole site, so that is raised as critical. Other failures are warnings. |
robots.txt changed robots.changed | Info | The file is different from the last check. Open the diff to see exactly which lines moved. |
How it decides a block is a problem
Plenty of sites block AI crawlers on purpose, so SEOComet doesn’t alarm you about blocks you already had. On the
first successful check it notes which crawlers are blocked from /, and
only a new block after that raises an issue.
Googlebot and Bingbot are the exception. They are never taken as deliberate, whether or not
they’re on your watch list, and losing either is critical. A * block
is critical too when it takes a real search engine with it. If you keep explicit allow groups for Googlebot and
Bingbot while blocking everyone else, that’s treated as a choice and handled like any other watched crawler.
A blocking issue is raised on every check while the block lasts, so its incident stays open until the file is fixed. A temporary 404 in between doesn’t reset what SEOComet knows about the file.
- Watch list
- Starts with Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Add up to 20 crawler names.
- Important paths
-
Up to 50 paths, such as
/products/. If one becomes disallowed for every crawler or for Googlebot, that’s critical. - Check interval
- Every hour by default. As often as every hour on Free, every 5 minutes on Pro and every minute on Agency.
- Server errors
- Google treats a robots.txt that answers 5xx or 429 as blocking the whole site, so SEOComet raises that as critical.
Every version, with the diff
SEOComet stores each version of robots.txt it fetches. Open any two side by side to see which lines were added and removed, and when. Slack and email alerts for a change carry the first 10 changed lines, so you often know what happened before you click through.
Sitemaps come from robots.txt too
When you add a site, SEOComet reads the Sitemap: lines in robots.txt
for your domain and its subdomains and adds a sitemap monitor for each one, falling back to
/sitemap.xml when there are none. If a line later disappears, you get
a warning, because crawlers may stop finding that sitemap.
Add your first site free
The Free plan covers 1 site with 6 monitors and email alerts. No card needed.