(Disclaimer: I work for a big company which makes a browser, an OS for smartphones and a search engine. This is my own opinion, and does not reflect my employer, and so on, and so on…)
> Some days I wonder what would stop me writing my own web crawler and building my own damn search engine from scratch.
A few things:
* Storing the index. For example if I search for "Sushi", google says there is 1.3 Billion results. That's already 19.3 GiB just to store the relationship between "sushi" and its results. (If you use UUIDs) And you need to do that for most words in the english dictionnary. And I'm assuming you're not even storing what you crawl for debugging/re-indexing.
* Website admins might throttle you. For example, discourse — a website which runs most discussion for Rust, Ocaml, and many other projects — blocked the bingbot at some point[1] due to its crawling aggressivity. Cloudflare, which is the front-facing caching service for almost half of the internet now, is notoriously anti-crawling. (It will quickly display captchas)
* grep over the web will yield you bad results. (most of the time) You want a search engine which at least groups words semantically. If I search for horses, I'm usually also interested in mares and ponies.
> Some days I wonder what would stop me writing my own web crawler and building my own damn search engine from scratch.
A few things:
* Storing the index. For example if I search for "Sushi", google says there is 1.3 Billion results. That's already 19.3 GiB just to store the relationship between "sushi" and its results. (If you use UUIDs) And you need to do that for most words in the english dictionnary. And I'm assuming you're not even storing what you crawl for debugging/re-indexing.
* Website admins might throttle you. For example, discourse — a website which runs most discussion for Rust, Ocaml, and many other projects — blocked the bingbot at some point[1] due to its crawling aggressivity. Cloudflare, which is the front-facing caching service for almost half of the internet now, is notoriously anti-crawling. (It will quickly display captchas)
* grep over the web will yield you bad results. (most of the time) You want a search engine which at least groups words semantically. If I search for horses, I'm usually also interested in mares and ponies.
[1] https://meta.discourse.org/t/bingbot-is-no-longer-default-th...