There's something wrong with the way you think if you view an accusation directed towards a statement as an accusation towards the person who said it instead.
Branding someone's speech as sexist is implying they are sexist themselves. Who else would, in neutral context, make sexist statements without being a sexist?
Besides which, if you understood my point I don't see how you could view a statement like "Gender X is better at Y" as being sexist in it of itself. You would need to see the supporting evidence or lack thereof of to decide whether the idea conveyed is sexist.
