> No! If you cannot find a statistically significant benefit, perhaps your experiment design is flawed. Perhaps your sample size is too small. Perhaps you have too many confounders. But you cannot conclude the measures are not effective!
You’re misreading the article. He’s not saying they’re not effective. He’s saying that a high-powered study couldn’t prove them effective, which is the same thing you’re saying.
The measured difference in community transmission between masks required and masks voluntary was 21%. This was not statistically significant because it was underpowered, as GP correctly stated. Saying that no statistical significance means null hypothesis is a kind of type 2 error, which was difficult to make given that the point estimate did not even support his claim.
I'm frankly surprised that Zweig made such an embarrassingly simple public mistake that any statistician could have pointed out to him had he bothered to ask. His readers might not have that kind of access, but he certainly could get this answered in five minutes.
He didn’t say that no statistical significance meant the null and expressly addressed the point you’re making:
> It’s also important to underline that just because the Georgia study did not find a statistically significant benefit doesn’t mean that a larger, more statistically powerful study might not find one, said Hoeg, the epidemiologist. But since this study had 90,000 students, then the question becomes: If you need 500,000 or 1 million people to find a benefit, how marginal is that benefit?
You seem to be misreading his conclusion. He’s not saying “masking children doesn’t work.” His point is that the practice carries significant potential downsides for children, and the best real-world study we have failed to provide affirmative support that those are outweighed by statistically significant reductions in COVID transmission.
It’s the same thing the FDA does—if your drug fails to show statistically significant efficacy in testing that means you don’t get approval.
When Zweig says, "If you need 500,000 or 1 million people to find a benefit, how marginal is that benefit?" that means that he doesn't realize that a 21% point estimate in that study with 0% in the error bar means that 42% is also in the error bar. This is not marginal! The point estimate is not marginal either, so when Zweig says that requiring a larger study size means that the effect is marginal, he is either being intentionally deceiving, or he does not understand statistics. I gave him the benefit of the doubt.
> if your drug fails to show statistically significant efficacy in testing that means you don’t get approval.
He doesn't make that point, but that one is also rather silly. If all of our evidence suggests mask effectiveness among elementary school students (some by mechanism, some by simulation, and some by evaluating mask policies on other groups), including the study he pointed to, the correct decision is to require masks for elementary school students. The harms are negligible, and the benefits for reducing community spread are large. The risk for approving an ineffective drug is far larger.
Elsewhere, you made the point that we don't know if masks should be required for toddlers. That I agree with, but that isn't a point that Zweig has made.
You’re misreading the article. He’s not saying they’re not effective. He’s saying that a high-powered study couldn’t prove them effective, which is the same thing you’re saying.