Additionally, Transformers are RNNs: https://arxiv.org/abs/2006.16236

Transformers are turing complete RNNs, as are the Neocortex.

There are differences like spiking vs non-spiking neurons, universal vs non-universal activation, the effects of brain waves, bicameralism, etc. But these differences may or may not be important depending on the topic of discussion. Additionally, the differences increasingly disappear as transformers are increasingly augmented with functionally equivalent systems.