You should keep in mind that XFS is primarily intended (it does get used in other places thou) for big systems/large raid arrays, with high IO loads etc.

A kernel build is really a bad workload to benchmark a FS anyway.