Dr Paul Hewson, School of Mathematics and Statistics, Exeter University
Don’t be put off by the title, I’m determined to stay in my lane and my reflections have implications beyond statistics. As a statistician and educator, I’ve grown increasingly uneasy about the way we teach statistical significance. A huge pedagogic evidence base exists on how we should be doing this; there are even a few people who pay attention to the teaching and learning literature. But that’s not my concern. I’m more concerned from a philosophical perspective; statistical significance has become an appeal to authority, not a tool for understanding. And appeals to authority are one of the very cognitive biases we should be educating students to avoid, not inducting them into using.
For example, we often present significance testing as a definitive answer to the question, “Is there an effect?” But this framing is misleading. It suggests that a p-value below 0.05 confers truth, while one above it implies irrelevance. In reality, the threshold is arbitrary, and the interpretation is fraught with nuance. Of course, part of the problem is academic, in publication statistical significance acts as a gatekeeper. What gets published is what gets deemed “significant”, and so our idea of “true” becomes circular: it’s true because it gets published, and it gets published because it’s statistically significant.
Significantly (yes, I’m playing annoying word games now), this is not what Fisher intended. Fisher’s original conception of the p-value was as a heuristic, a way to measure the strength of evidence against a null hypothesis. It was never meant to be a binary decision rule. Neyman and Pearson introduced the idea of fixed thresholds and error rates, but even they acknowledged the limitations. Over time, these ideas have been simplified, codified, and institutionalized into a ritual: calculate the p-value, check the threshold, declare significance. The problem is that this ritual obscures the real purpose of statistics: to learn from data. It encourages students to think of statistics as a set of rules to follow, rather than a way of reasoning about uncertainty. Worse, it divorces statistical practice from the philosophy of science. We rarely ask students to consider what evidence means, how models relate to reality, or what assumptions underlie their inferences. Instead, we teach them to chase significance.
I think we need a reorientation. Statistics should be taught as part of a broader empirical framework; which would mean exposing students to epistemology, model-building, and scientific reasoning. Students should learn to ask: What is the question? What assumptions are we making? Why should we collect data rather than just believe what we want? What does the data say, and how confident can we be? Statistical tools should support these inquiries, not subvert them.
I’m deliberately avoiding deeper technical arguments about statistical significance and Bayes methods. Nevertheless, I believe in teaching students to think critically about evidence, rather than mechanically applying procedures. And that means acknowledging that statistical practice is shaped by social and institutional forces, not just mathematical ones.
In short, I’m claiming that teaching statistical significance as a definitive answer is a mistake. That’s because it’s an appeal to authority masquerading as objectivity. If we want students to become thoughtful, empirical scientists, we need to teach them to question, not just to calculate. And outside my own lane, how much of “following the science” means listening to scientific experts we approve of rather than being scientific?
For more information: Contact p.hewson2@exeter.ac.uk
Image: Image by Gerd Altmann from Pixabay
