SEMRush, Ahrefs, Moz, Majestic SEO, and there are many other tools, all basically do the same thing. Allow you to see how your website is ranking, see where your competitors are ranking, and other opportunities to improve your website’s SEO.
Why Block Them?
Do you pay for any of these tools? If not why are you letting them collect data about your website that your competitors can see?
They use up your server resources (one tool may not use very much, but if there are 100 tools then that adds up over time), no website host will give you unlimited bandwidth (unless it is a provider that is either very cheap or very expensive) and each of these SEO tools uses a little bit of bandwidth so stop them in their tracks especially if you have limited bandwidth.
Even though they are SEO tools, they don’t affect your website’s SEO. The number’s that these tools give are either numbers you can get yourself for free, or are numbers made up by the company (like domain authority). If you block the SEO tools it doesn’t make your website rank lower (unless you block the search engine’s bots).
How To Block Them?
There isn’t a 100% fool-proof method to block all the SEO tools. This is meant to talk about the most common methods.
Most of the major tools use a specific user agent, which is a way to uniquely identify them and can be one way to block them. Depending on your website there could be multiple ways to block a user agent, from using the robots.txt to disallow that user agent, to using robot meta tags, to using security software that blocks the user agent. Each SEO tool will have their own user agent which may require you to have a ton of blocking lines.
This may block the well-known ones like some of the ones that was mentioned in this post, but what about the other ones? You may want look through your 404 results to see what pages those other SEO tools go to. They are pages that you currently don’t have (or aren’t made public) that the tools look for and may report in their own portal for their customers.
How do you find your 404 results? Some website analytics like Google Analytics can allow to see them. Your website log can show them (you may not want to do this if you aren’t technical, but if you are curious you can ask your website host how to see your website log). Google Search Console is also a great tool that can show you this among many other things, both Backlinko and Moz has great guides if you are new to that. Depending on what your website is running, there are also plugins you can install that can track that. If you are on WordPress, one of the more popular plugins that do this is Redirection.
Once you know the user agent of the tools that repeatedly show up in your 404 results, you can block them like you previously did (using robots.txt, robot meta tags, or security software).
Should I Block The IP Address Of The Tool?
Some suggest blocking the ip address of the SEO tools, in addition to the user agent, to be sure that it is blocked and won’t come to your website again.
That advice is outdated as ip addresses can be changed, and servers that run the tools can be moved (which would change the ip address). The reputable companies will keep the same user agent so you can block them.
Even SEMRush, one of the most popular tools on this list says not to block the ip address.
How Long Does It Take Before The SEO Tool Stops Showing Up?
Most SEO companies say if you block their tools it will take anywhere from one to twenty-four hours before their tools stop appearing on your site. They give a range because it depends on how often the tools checks your website.
