Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
pipihu
on June 30, 2014
|
parent
|
context
|
favorite
| on:
Robots.txt Disallow: 20 Years of Mistakes To Avoid
The main use for robots.txt is to prevent crawling of infinite URL spaces:
http://googlewebmastercentral.blogspot.com.br/2008/08/to-inf...
Alongside tagging links to such resources with nofollow.
ashmud
on June 30, 2014
[–]
Back in the day, I would use httrack for offline web browsing, and these were a constant irritation.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Alongside tagging links to such resources with nofollow.