A slightly stronger argument not to use 4096-bit certs is that just about every (intermediate) CA certificate out there is 2048-bit, and there's no security advantage to your cert being 4096-bit if it only has a 2048-bit signature. An attacker can just factor your CA's public key and not care how strong your own public key is.

So it's not just that it slows down your site, it slows down your site without any additional security or other benefit.