The overhead of context switching becomes pretty high. Some say that context switching has become cheap, but you still at the very least need to update the tlb, and schedule the next pthread.
At least the performance of context switching should scale with the number of cores, which seems to be the main direction of increased performance in hardware looking into the future.