Banning web access by IP
Posted: Mon Apr 12, 2021 11:19 am
I've had to ban an address forcefully today for loading the site with tens of request per second with what looked like a site crawler. This slowed down the forum to snail pace.
It is perfectly allowed to crawl the site, and search bots to this regularly. But the general agreement is that you should then abide by what is specified in http://www.makehumancommunity.org/robots.txt, particularly the crawl-delay setting.
It is perfectly allowed to crawl the site, and search bots to this regularly. But the general agreement is that you should then abide by what is specified in http://www.makehumancommunity.org/robots.txt, particularly the crawl-delay setting.