Skip to content
SEOComet

Catch the robots.txt change that takes your site out of search

SEOComet fetches your robots.txt every hour on Free, down to every 5 minutes on Business, keeps every version, and alerts you when a change blocks Googlebot, a path you care about or a crawler on your watch list.

Mock-up of a robots.txt diff in SEOComet: Disallow: /checkout/ became Disallow: /, which SEOComet flags as critical, search engines blocked, and a Sitemap line was removed, flagged as a warning.

What it catches

Each problem opens an incident with a severity, so you can be alerted about a blocked Googlebot and not about a harmless edit.

Issues the robots.txt check raises
Search engines blocked Critical robots.txt now blocks Googlebot or Bingbot from every page, on their own or through the * group. The site can drop out of search results.
Important page blocked Critical A path you marked as important is now disallowed for every crawler or for Googlebot.
Crawler blocked Warning A crawler on your watch list, such as GPTBot or ClaudeBot, is now blocked from the whole site. Crawlers that were already blocked on the first check are taken as deliberate.
Sitemap line removed Warning A Sitemap: line that robots.txt used to list is gone, so crawlers may stop finding that sitemap.
robots.txt missing Warning robots.txt was there before and now answers 404 or 410, or a web page instead of a text file, so crawlers see no rules at all. A robots.txt that never existed raises nothing.
Couldn’t check the address CriticalWarning Critical for a server error or 429 Google treats a robots.txt that answers with a server error, or 429, as blocking the whole site, so that is raised as critical. Other failures, like a timeout, are warnings.
robots.txt changed Info The file is different from the last check. Open the diff to see exactly which lines moved.

How it decides a block is a problem

Plenty of sites block AI crawlers on purpose, so SEOComet doesn’t alarm you about blocks you already had. On the first successful check it notes which crawlers are blocked from /, and only a new block after that raises an issue.

Googlebot and Bingbot are the exception. They are never taken as deliberate, whether or not they’re on your watch list, and losing either is critical. A * block is critical too when it takes a real search engine with it. If you keep explicit allow groups for Googlebot and Bingbot while blocking everyone else, that’s treated as a choice and handled like any other watched crawler.

A blocking issue is raised on every check while the block lasts, so its incident stays open until the file is fixed. A temporary 404 in between doesn’t reset what SEOComet knows about the file.

Watch list
Starts with Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Keep up to 20 crawlers on the list, on the check’s Settings tab.
Important paths
Up to 50 paths, such as /products/, added on the check’s Settings tab. If one becomes disallowed for every crawler or for Googlebot, that’s critical.
Check interval
Every hour on Free and every 5 minutes on Pro and Business.
Server errors
Google treats a robots.txt that answers 5xx or 429 as blocking the whole site, so SEOComet raises that as critical.

Every version, with the diff

SEOComet stores each version of robots.txt it fetches. Open any two side by side to see which lines were added and removed, and when. Slack and email alerts for a change carry the first 10 changed lines, so you often know what happened before you click through.

Mock-up of a SEOComet alert in Slack: "New issue: robots.txt on Shop", saying robots.txt now blocks every crawler from the whole site, with the changed lines and a View incident button.

Sitemaps come from robots.txt too

When you add a site, SEOComet reads the Sitemap: lines in robots.txt for your domain and its subdomains and adds a sitemap check for each one, up to 50. If there are none, it watches /sitemap.xml instead. If a line later disappears, you get a warning, because crawlers may stop finding that sitemap.

A site with no robots.txt still gets a robots.txt check. It shows as not found and sends no alerts until the file appears, then it’s checked in full. robots.txt is included on every plan, so it never uses up your plan’s watched pages.

How sitemap monitoring works

Join the SEOComet waitlist

SEOComet isn’t open to the public yet. Leave your email and we’ll tell you when you can add your sites. The SEO Toolbar Chrome extension is free and works today.