A year of fighting scrapers on my 1.5 million-page website
Hacker News
Read full postThe PatronView website, hosting 1.5 million profile pages, experienced overwhelming bot traffic, with 214 bot page loads for every human visit. The site owner shares detailed firewall rules and observations, including blocking Amazon's AI crawler and noting extremely low CAPTCHA solve rates by bots.




