Catch the robots.txt change that takes your site out of search
SEOComet fetches your robots.txt every hour on Free, down to every 5 minutes on Business, keeps every version, and alerts you when a change blocks Googlebot, a path you care about or a crawler on your watch list.
What it catches
Each problem opens an incident with a severity, so you can be alerted about a blocked Googlebot and not about a harmless edit.
| Issue | Severity | What it means |
|---|---|---|
| Search engines blocked | Critical | robots.txt now blocks Googlebot or Bingbot from every page, on their own or through the * group. The site can drop out of search results. |
| Important page blocked | Critical | A path you marked as important is now disallowed for every crawler or for Googlebot. |
| Crawler blocked | Warning | A crawler on your watch list, such as GPTBot or ClaudeBot, is now blocked from the whole site. Crawlers that were already blocked on the first check are taken as deliberate. |
| Sitemap line removed | Warning | A Sitemap: line that robots.txt used to list is gone, so crawlers may stop finding that sitemap. |
| robots.txt missing | Warning | robots.txt was there before and now answers 404 or 410, or a web page instead of a text file, so crawlers see no rules at all. A robots.txt that never existed raises nothing. |
| Couldn’t check the address | CriticalWarning Critical for a server error or 429 | Google treats a robots.txt that answers with a server error, or 429, as blocking the whole site, so that is raised as critical. Other failures, like a timeout, are warnings. |
| robots.txt changed | Info | The file is different from the last check. Open the diff to see exactly which lines moved. |
How it decides a block is a problem
Plenty of sites block AI crawlers on purpose, so SEOComet doesn’t alarm you about blocks you already had. On the
first successful check it notes which crawlers are blocked from /, and
only a new block after that raises an issue.
Googlebot and Bingbot are the exception. They are never taken as deliberate, whether or not
they’re on your watch list, and losing either is critical. A * block
is critical too when it takes a real search engine with it. If you keep explicit allow groups for Googlebot and
Bingbot while blocking everyone else, that’s treated as a choice and handled like any other watched crawler.
A blocking issue is raised on every check while the block lasts, so its incident stays open until the file is fixed. A temporary 404 in between doesn’t reset what SEOComet knows about the file.
- Watch list
- Starts with Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Keep up to 20 crawlers on the list, on the check’s Settings tab.
- Important paths
-
Up to 50 paths, such as
/products/, added on the check’s Settings tab. If one becomes disallowed for every crawler or for Googlebot, that’s critical. - Check interval
- Every hour on Free and every 5 minutes on Pro and Business.
- Server errors
- Google treats a robots.txt that answers 5xx or 429 as blocking the whole site, so SEOComet raises that as critical.
Every version, with the diff
SEOComet stores each version of robots.txt it fetches. Open any two side by side to see which lines were added and removed, and when. Slack and email alerts for a change carry the first 10 changed lines, so you often know what happened before you click through.
Sitemaps come from robots.txt too
When you add a site, SEOComet reads the Sitemap: lines in robots.txt
for your domain and its subdomains and adds a sitemap check for each one, up to 50. If there are none, it
watches /sitemap.xml instead. If a line later disappears, you
get a warning, because crawlers may stop finding that sitemap.
A site with no robots.txt still gets a robots.txt check. It shows as not found and sends no alerts until the file appears, then it’s checked in full. robots.txt is included on every plan, so it never uses up your plan’s watched pages.
Join the SEOComet waitlist
SEOComet isn’t open to the public yet. Leave your email and we’ll tell you when you can add your sites. The SEO Toolbar Chrome extension is free and works today.