it's only really affected from a negative perspective (think: so slow that google can't crawl your website or load it).

Anything faster than 2-3 seconds has very diminishing if not 0 returns.