As a layman who probabbly didn't understand that whole article I ask:
If "statistical significance" is just sort of an empty phrase used to dismiss or prove something somewhat arbitrarily. Then isn't the same person writing the same study likely to be just as arbitrary declaring what is or isn't significant .... anyway?
There are strict rules that define when something is "statistically significant", it's not at all arbitrary. The problem is people thinking that just because something is statistically significant, it is automatically true. Which it isn't. Statistically significance by definition includes the possibility that something was just a statistical oddity. This article essentially just reminds people of that, and urges them to abandon the "statistical significance is the same as ultimate truth" conclusion.
I feel like this misses another important prong of the article, which is: that failing to find a "statistically significant" correlation (according to some given significance test) is often mistakenly interpreted even by scientists themselves as meaning "We have a good basis to conclude that there is no such correlation".
There aren't is the thing. There are widely used standards, but they don't actually have any real basis. Fisher I believe is the one who popularized it but it was for specific circumstances and he acknowledged that it was just a convenient thing.
Sure there are. You can't just call a result "significant" at will. You can pull numbers out of thin air, pre-filter your data or carefully pick a statistical test to be in your favor. But it's still well-defined which outcomes you're allowed to call significant and which ones you can't.
Statistical significance is a simple idea, it's nothing more than the realization that if you flip a thousand coins, getting a thousand heads means they probably have heads on both sides or something, while getting fifty-fifty doesn't mean much. On the other hand, if you flip one coin and get 100% heads, you haven't proven much of anything either. Statistical significance is just one way to put a number on how likely it is that you're reading tea leaves.
It can't stop you from lying, cheating, or stealing. Nobody has ever not been able to commit fraud because of p-values. There is no formula in the world that can notice when you are cheating in what numbers you plug in...
However, if you know what you're doing, and you're honest, it's really important to know whether you are seeing shapes in clouds or actual patterns. That's what statistical significance is about.
the idea of the p-value is a useful and reasonable one. it's a good statistical tool that can help you tell whether your results are due to something real in the world or just dumb luck.
the problem is people leaning on it too heavily and especially obsessing over the 0.05 threshold: so that p=0.049 means your result is statistically significant, you get a paper published and a press release and tenure, while p=0.051 means failure and penury.
the article is arguing specifically against this latter practice, of "bucketing" things according to some cutoff which was literally made up arbitrarily by Ronald Fisher.
If "statistical significance" is just sort of an empty phrase used to dismiss or prove something somewhat arbitrarily. Then isn't the same person writing the same study likely to be just as arbitrary declaring what is or isn't significant .... anyway?