A GPU isn't really comparable to the modern CPU sitting beside it, since it lacks many of the CPUs features, eg. virtual addressing, while in the other instances mentioned the processors were pretty much the same, feature-wise.

GPUs do, like most peripherals, contain additional CPUs that control them -- in the case of GPUs these are proprietary architectures specifically designed just for scheduling and sequencing tasks on the GPU and are said to be quite potent, for a device control processor.

Similarly hard drives had multi-core MIPS and ARM processors for quite a while (ditto for SSDs, in both cases they are ASICs, the actual data juggling isn't done by the CPU cores), and they even have quite a bit of RAM -- the cache -- that they'd normally share with the DMA engine, plus some of their own...

The control processors in most other peripherals are usually much weaker, though. They're pretty much just microcontrollers.