> And since Go is so readable, theoretically getting the core functionality out and rewriting it in a more specialized language would be fairly straightforward.

You'll have to rewrite your whole program (or at least factor out the "core" part you care about into a separate binary and talk to it via IPC or network) because Golang has terrible FFI as pointed out by OP.