SEOCometBot
SEOCometBot is the crawler behind SEOComet. SEOComet watches a site’s robots.txt, sitemaps, ads.txt, uptime, key pages and SSL certificate, and tells its users when one of them changes or breaks.
If it shows up in your logs, someone added your site to SEOComet. That’s usually you, someone on your team or an agency you work with. We don’t ask people to prove they own the sites they add, so if nobody on your side did, write to hello@seocomet.com and we’ll stop.
How to recognise it
Every request it makes sends this user agent:
SEOCometBot/1.0 (+https://www.seocomet.com/bot) It asks for any content type and accepts gzip. It sends no cookies and never logs in.
What it fetches
It only visits sites that SEOComet users have added, and on those sites only these addresses:
/robots.txt-
Sitemaps: the ones listed in robots.txt on your domain or its subdomains, or
/sitemap.xmlwhen robots.txt lists none, plus any sitemap a user added by hand. For a sitemap index it also fetches the child sitemaps, one at a time, up to 200 per check. It counts the URLs in a sitemap but never visits them, with one exception: for a Google News sitemap it fetches the 5 newest articles on your site, to check that Google News could index them. /ads.txtand/app-ads.txt- For uptime checks, your home page and any other address on the same host and port that a user added.
- At most once every hour, the other addresses of your home page (http or https, with or without www), to check that they redirect to it.
-
Once, when a site is added: the home page, to see where it redirects;
/sitemap.xml,/ads.txtand/app-ads.txt, to see which exist; and/favicon.ico, and if that isn’t an image, the home page again, to find the icon it links to.
Every one of these is a plain GET. It doesn’t follow links, run JavaScript, load images or stylesheets, or submit forms.
SSL checks open a TLS connection to read your certificate and then close it. They send no HTTP request, so they won’t appear in your access logs with the user agent above.
SEOComet also sends alert webhooks with the same user agent, as POST requests to addresses its users set up. If you receive one of those, someone has pointed their SEOComet alerts at your server.
How often
How often SEOCometBot visits depends on the site owner’s plan. It fetches the homepage every 5 minutes on Free and every minute on Pro and Business, robots.txt every hour on Free and every 5 minutes on Pro and Business, sitemaps every hour on Free and Pro and every 30 minutes on Business and Google News sitemaps every hour on Free, every 15 minutes on Pro and every 10 minutes on Business. Other pages the owner watches are fetched every hour on Free, every 30 minutes on Pro and every 15 minutes on Business, ads.txt and app-ads.txt every hour on every plan, and the SSL certificate is read once a day.
Users can also start a check by hand. That’s limited to 10 a minute per person and 30 a minute per account. A request that times out, can’t connect or gets a server error or 429 is tried once more 2 seconds later. If that fails too, the next try comes at the next scheduled time.
If several SEOComet accounts watch the same site, each account’s checks run separately.
Limits on each request
- It gives up on connecting after 5 seconds, and on the whole request after 10 seconds (30 seconds for sitemaps, 5 seconds for the other addresses of your home page and for the first look at a new site). Redirects count toward the same time.
- It stops reading after 5 MB (50 MB for sitemaps, 100 KB for a favicon and 1 MB for the home page it reads to find one). Gzipped responses are held to the same limit once unpacked.
- It follows up to 5 redirects.
- It only connects to public IP addresses, and it checks every redirect against that rule too.
robots.txt
SEOCometBot doesn’t apply your robots.txt rules to its own requests, so a Disallow line for SEOCometBot won’t stop it. It reads robots.txt to report on it, for example to warn a user when a change blocks Googlebot.
What it fetches is limited to the files above, which sites publish for machines to read, and to the addresses a user chose to watch for uptime. It fetches them for the people monitoring the site, not to index or copy its content.
Asking us to stop
Email hello@seocomet.com with the domain you’d like us to stop checking. If you also run the SEOComet account that added it, you can pause or delete the site there yourself.