Missing files

When you add a site, SEOComet looks for its robots.txt, sitemaps and ads.txt, and adds a check for each one, whether it found the file or not. A file that isn’t there is watched quietly: no alerts, no incidents. When the file appears, SEOComet starts checking it in full.

Files a site doesn’t have

These are the files SEOComet looks for:

• robots.txt
• /sitemap.xml, when robots.txt doesn’t list a sitemap
• ads.txt

app-ads.txt is different: it’s only for app publishers, so SEOComet adds its check only when the file is there. If you publish apps and haven’t made one yet, add the check yourself with Add check.

Many sites don’t need every file. A site without ads has no use for ads.txt. You don’t have to do anything about a file you don’t need. It never alerts you, and it doesn’t count against your plan.

Not found

A check for a file that has never been there shows Not found in grey on the site’s Checks page. It isn’t a problem, so it isn’t counted under Needs attention and it never sends an alert. It counts as not found when the address answers 404 or 410. For robots.txt, ads.txt and app-ads.txt, a web page answering in the file’s place counts as not found too.

Any other error is reported like on any other check. A robots.txt that answers with a server error is critical, because Google then stops crawling the site.

The check keeps running at your plan’s speed for its kind of file. See How often checks run.

When the file appears

The first check that finds the file starts checking it in full, the same as a file that was there when you added the site. From then on, if the file goes missing again, that’s a real problem and SEOComet tells you:

• robots.txt: a warning that robots.txt is missing, which opens an incident.
• Sitemap: a critical issue that the sitemap is missing, which opens an incident.
• ads.txt and app-ads.txt: a critical issue, since buyers may stop bidding.

A sitemap that robots.txt lists is expected to be there from the start. If it answers 404, that’s critical straight away.

If you change a check’s address, it starts again as not found until the file is found at the new address.

Plan limits

robots.txt, sitemaps, ads.txt and app-ads.txt are included on every plan, found or not. They never use up your plan’s watched pages. A site can have up to 50 sitemaps.

If you don’t want a check at all, you can pause or delete it from the Checks page.