But does this fragmentation actually matter? For most apps which are at most accessing some sort of online data, like a web shop, chat apps, etc. you just target the lowest API that has the features you want and it's compatible with all newer Android OS releases.

I would think the real fragmentation that developers have to consider is in things like screen size, or custom APIs by hardware vendors. In my experience, it's the difference between Samsung's bluetooth stack and Android's own