Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I agree - I actually built the thread pool stuff for work while I was testing, since I had the code from a previous project.

If I was more worried about Safeway catching on I'd probably do something as you suggested (at the very least I'd add user agent headers and the other cookies expected from a real session, as right now it's trivial to detect my requests).

Sneaker bots do this exceedingly well - it's a constant cat and mouse game to make the requests look as human as possible, very interesting space right now.



I highly recommend not making more then say 5req/s - and preferably do it at like 2am.

These projects are fun IMO, but it's best to not hammer anyone's servers if it isn't necessary.


But at 2AM your traffic will create a more noticeable spike...

It doesn’t blend in as well...


In my experience, unless they have excellent monitoring and a team looking to ban you they won't notice. It's half courtesy, and the other half is because it's not likely to affect their business operations (by slamming the servers when they're busy) they're less likely to notice or care.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: