Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

appreciate the sharing of info here.

I have recently been discovering and combating some similar, albeit much smaller issues.

I've been finding that a bunch of my recent 'resource sucks' have been constant spidering from petal-bot, semrush bot, alibiba-bot and a few others.

Using the wordpress plugin stop-bad-bots and it's logs has been eye-opening for me recently.

I understand many of these are not directly dark-seo related, but their aggressive nature is hurting the cpu and memory limits of some of my servers and sites so it's a big issue regardless of the intents behind them.

(kind of) glad someone else has dealt with these issues, and glad to see some of the 'how' for handling, identifying, and some actual real numbers for the impacts, as I've been guessing some of these things in my small projects, indeed it's a real thing. As well as a practical issue to pay attention to and work on.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: