I worked at an HFT for several years as a developer on the quant lib team. I'd say the algorithm side is the hard and interesting part, while the math is relatively straightforward and solved. For example, it's well known how to price an option - but doing that efficiently and at large scale is the challenging aspect (think kd-trees, etc. for sophisticated caching, messaging protocols, dbs for organizing data). I also built a sophisticated graph algo for relating securities, which built the core of one of our most successful strategies.

So technically, I found it very appealing. However, my experience wasn't so great otherwise. You really get the sense that people are only in it for the money - everything revolves around the year end bonus. And at least where I worked, this kind of huge money drove a lot of really nasty politics.