Speaking from experience, there were plenty of normal families who chose homeschooling decades ago. The problem is that the weird people make a lot of noise, and then homeschooling gets a reputation, which causes the self-aware normal folks to shy away from talking about homeschooling too publicly because they don't want to be associated with the negative stereotypes.

The trouble is, this can become a feedback loop, because the only way to dispel the stereotypes is if more normal folks speak up.

I've seen this same phenomenon affect other subcultures too, and I suspect there's probably a name for it, but I don't know what it is.