Skip to content
SEOComet

SEOCometBot

SEOCometBot is the crawler behind SEOComet. SEOComet watches a site’s robots.txt, sitemaps, ads.txt, uptime and SSL certificate, and tells its users when one of them changes or breaks.

If it shows up in your logs, someone added your site to SEOComet. That’s usually you, someone on your team or an agency you work with. We don’t ask people to prove they own the sites they add, so if nobody on your side did, write to [email protected] and we’ll stop.

How to recognise it

Every request it makes sends this user agent:

SEOCometBot/1.0 (+https://www.seocomet.com/bot)

It asks for any content type and accepts gzip. It sends no cookies and never logs in.

What it fetches

It only visits sites that SEOComet users have added, and on those sites only these addresses:

  • /robots.txt
  • Sitemaps: the ones listed in robots.txt on your domain or its subdomains, or /sitemap.xml when robots.txt lists none, plus any sitemap a user added by hand. For a sitemap index it also fetches the child sitemaps, one at a time, up to 200 per check unless the user changes that. It counts the URLs in a sitemap but never visits them.
  • /ads.txt and /app-ads.txt
  • For uptime checks, your home page and any other address on the same host and port that a user added.
  • Once, when a site is added: /favicon.ico, and if that isn’t an image, the home page, to find the icon it links to.

Every one of these is a plain GET. It doesn’t follow links, run JavaScript, load images or stylesheets, or submit forms.

SSL checks open a TLS connection to read your certificate and then close it. They send no HTTP request, so they won’t appear in your access logs with the user agent above.

SEOComet also sends alert webhooks with the same user agent, as POST requests to addresses its users set up. If you receive one of those, someone has pointed their SEOComet alerts at your server.

How often

Each address is checked on its own schedule. These are the defaults:

  • robots.txt: every hour
  • Sitemap: every 6 hours
  • ads.txt: every hour
  • app-ads.txt: every hour
  • Uptime: every 5 minutes
  • SSL certificate: every 12 hours

Users can change the interval, but never to less than their plan allows: every hour on Free, every 5 minutes on Pro and every minute on Agency. On a plan with a longer minimum, the defaults above are raised to it.

Users can also start a check by hand. That’s limited to 10 a minute per person and 30 a minute per account. A check that fails isn’t retried straight away; the next try comes at the next scheduled time.

If several SEOComet accounts watch the same site, each account’s checks run separately.

Limits on each request

  • It gives up on connecting after 5 seconds, and on the whole request after 10 seconds (30 seconds for sitemaps). Redirects count toward the same time.
  • It stops reading after 5 MB (50 MB for sitemaps, 100 KB for a favicon and 1 MB for the home page it reads to find one). Gzipped responses are held to the same limit once unpacked.
  • It follows up to 5 redirects.
  • It only connects to public IP addresses, and it checks every redirect against that rule too.

robots.txt

SEOCometBot doesn’t apply your robots.txt rules to its own requests, so a Disallow line for SEOCometBot won’t stop it. It reads robots.txt to report on it, for example to warn a user when a change blocks Googlebot.

What it fetches is limited to the files above, which sites publish for machines to read, and to the addresses a user chose to watch for uptime. It fetches them for the people monitoring the site, not to index or copy its content.

Asking us to stop

Email [email protected] with the domain you’d like us to stop checking. If you also run the SEOComet account that added it, you can pause or delete the site there yourself.