robots.txt checks
What the robots.txt check looks for: blocked search engines, blocked important paths, watched crawlers, removed Sitemap lines and every change.
Updated
The robots.txt check reads the file at your site root the way Google reads it, and tells you when a change blocks search engines, a page you care about or a crawler you watch. It also keeps every version of the file, so you can see exactly what changed.
Each site has one robots.txt check. SEOComet adds it when you add the site, even when the site has no robots.txt. Until one appears, the check shows Not found and sends no alerts. See Missing files.
How often
Every hour on Free and every 5 minutes on Pro and Business. See How often checks run.
What it can find
| Issue | Severity | What it means |
|---|---|---|
| Search engines blocked | Critical | robots.txt now blocks Googlebot or Bingbot from every page, on their own or through the * group. The site can drop out of search results. |
| Important page blocked | Critical | A path you marked as important is now disallowed for every crawler or for Googlebot. |
| Crawler blocked | Warning | A crawler on your watch list, such as GPTBot or ClaudeBot, is now blocked from the whole site. Crawlers that were already blocked on the first check are taken as deliberate. |
| Sitemap line removed | Warning | A Sitemap: line that robots.txt used to list is gone, so crawlers may stop finding that sitemap. |
| robots.txt missing | Warning | robots.txt was there before and now answers 404 or 410, or a web page instead of a text file, so crawlers see no rules at all. A robots.txt that never existed raises nothing. |
| Couldn’t check the address | CriticalWarning Critical for a server error or 429 | Google treats a robots.txt that answers with a server error, or 429, as blocking the whole site, so that is raised as critical. Other failures, like a timeout, are warnings. |
| robots.txt changed | Info | The file is different from the last check. Open the diff to see exactly which lines moved. |
A few details on how these are judged:
- SEOComet reads the rules as Google does: each crawler follows the one group that names it, or the
*group if none does, and the longest matching rule wins. - Googlebot and Bingbot are always checked, even if you remove them from the crawlers you watch. Blocking either from the whole site is critical, even if it was already blocked on the first check.
- Blocking
*(every crawler) is critical when Googlebot or Bingbot is blocked with it. If the file keeps them allowed in their own groups while blocking everyone else, that’s usually on purpose, so*is treated like a watched crawler instead. - The blocking issues are raised on every check while the block lasts, so their incident stays open until the file is fixed.
- Sitemap line removed and robots.txt changed compare one check with the one before. Their incidents close on the next check that gets an answer, whether or not the line comes back.
- A robots.txt that answers 404 or 410, or sends a web page in place of the file, counts as missing. Crawlers then treat the site as having no rules at all.
Important paths
Important paths are the parts of the site you can’t afford to have blocked, such as /products/ or /blog/. Add them in the check’s Settings › Important paths, one per line, each starting with /. There are none until you add some.
On every check, SEOComet tests each path against the rules for every crawler (*) and for Googlebot. If either can’t crawl it, that’s critical, and the issue names the path and who can’t crawl it.
Watched crawlers
Crawlers to watch in the check’s Settings is a list of user agents, one per line. When robots.txt newly blocks one of them from the whole site, you get a warning. The list starts as Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
Many sites block AI crawlers on purpose, so SEOComet doesn’t warn about crawlers that were already blocked on the check’s first run. The same goes for a crawler you add to the list while it’s already blocked. If a crawler is unblocked and later blocked again, that’s new, and you hear about it.
Only the owner and people with Can edit on the site can change a check’s settings. See Check settings and thresholds.
Versions
SEOComet keeps a copy of the file from every check. Each different copy is listed on the check’s Versions tab, and Compare versions shows the lines that changed between any two of them. The robots.txt changed issue has a View diff button for the latest change.
If the file goes missing for a while, SEOComet remembers the last copy it read, so when the file comes back it’s compared with what was there before the gap.
Versions are kept for as long as your plan’s check history, and the latest one is always kept. See How SEOComet works.