Not much slower than, say, Hyper-V or Virtualbox or other hypervisors that live alongside a general purpose desktop OS. Remember, they're using the same Intel chips and VT-d and such as anything else; as long as you have enough CPU power it's going to run fine.
IIRC the Docker implementation for OS X runs a tiny, carefully-configured Linux VM transparently in the background. This isn't that dissimilar from the Windows Subsystem for Linux from a user standpoint.
I've done plenty of Docker work on OS X; the only real weirdness is that you're running Linux in the containers and OS X outside. Eventually, though, before switching completely to a native Linux desktop, I moved deeper into the containers and would barely ever see the OS X command prompt any more.
Our build time on Linux container is 8 minutes, but it's a few hours on macOS using docker containers. I haven't dug into it but the hypervisor seems to be single threading the normally multicore compile.
Primarily disk access is a lot slower, even with Hyper-V. Eg. Compilation suffers from this. But tasks that don't use the disk too much shouldn't be that bad.
What? We use it at work every day for local development, and my previous employer did as well. It has issues, definitely, but they have more to do with Docker than with xhyve.
https://developer.apple.com/documentation/hypervisor https://github.com/moby/hyperkit