ahrefsbot

How to control AhrefsBot on your website? Since AhrefsBot strictly follows the robots.txt file on your website, you're in full control of what it does. To change the frequency at which AhrefsBot visits your site, just specify the minimum acceptable delay between two consecutive requests in your...

What is AhrefsBot? AhrefsBot is a web crawler that powers the database for both Ahrefs, an online data toolset, and Yep, a revenue-sharing web search engine. It’s the second most active crawler after Google, visiting over 8 billion web pages every 24 hours and updating its index every 15–30...