The Python services probably end up getting rewritten at some point but I want to focus on delivering a product. Right now the Python services can easily be deployed as the don't have that many dependencies on other libs. All further development will be done in Clojure though.
Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.
Of course most things can bee pooled into a multiprocessing pool. It's just a matter of the pain you have to endure while doing so. For some tasks it's a straight forward process while for other tasks it can be pretty painful. The crawler is IO-bound but most of the collaborative filtering algorithms are CPU-bound and having nice concurrency constructions in the language is a bonus.