Kasomi
JT Founder
How to control AhrefsBot on your website?
Since AhrefsBot strictly follows the robots.txt file on your website, you're in full control of what it does.To change the frequency at which AhrefsBot visits your site, just specify the minimum acceptable delay between two consecutive requests in your robots.txt file:
User-agent: AhrefsBot
Crawl-Delay: [value]
(Where Crawl-Delay value is time in seconds.)
If you want to stop AhrefsBot from visiting your site, add these two lines to the robots.txt file on your server:
User-agent: AhrefsBot
Disallow: /
Please note that AhrefsBot may need some time to pick up the changes in your robots.txt file. This is done before the next scheduled crawl.
Also, if your robots.txt contains errors, AhrefsBot won’t be able to recognize your commands and it'll continue crawling your website the way it did before.
Read more about robots.txt at www.robotstxt.org.
If you think that AhrefsBot is misbehaving on your website or if you have any questions, please reach out to support@ahrefs.com and we'll do our best to help
