[Estimated Reading Time: 3-4 min]
At SoundRocket, we often hear from our research partners who want to include p values in their summary databooks. These numbers have long been a staple in not just social science research but in many scientific inquiries. However, the role they play is now under much discussion. This article is designed to demystify p values, explain the growing conversation around them, and clarify why we’re choosing a different approach in our data reporting.
What is a P value?
Simply put, a p value helps scientists determine if their findings are significant — that is, unlikely to have occurred by chance. Imagine you’re investigating whether additional sleep can improve test scores. If students who slept more consistently perform better, the p value helps us understand whether this pattern is just a coincidence or if extra sleep and test scores are correlated.
Related, the term ‘statistical significance’ is used to describe how likely it is that the results we see from comparing two things could happen just by chance if we assume there is no real difference or effect (this is known as the ‘null hypothesis’). If a result is statistically significant, it means that what we observed is probably not due to random chance alone.
This is where p values come into play. A p value is a number between 0 and 1 that statisticians calculate to help determine if the results are statistically significant. If the p value is small, typically less than 0.05 [1], it suggests that the observed difference (like the higher test scores for students who slept more) would be unlikely to have been observed under the assumption the null hypothesis is correct. A p value of 0.05 suggests that just by chance it would be less likely than five times in 100 (that’s what p<0.05 means) to have seen this correlation if it were not actually “true.”
But here’s the catch — a low p value doesn’t confirm the cause (like extra sleep leading to better scores) or tell us how strong the effect is. It’s just a way to measure how surprised we should be by the results. And while a p value less than 0.05 has traditionally been viewed as ‘significant’, this threshold is somewhat arbitrary and doesn’t convey the full story.
But remember, even with a small p value, we can’t be certain that the extra sleep directly caused the higher scores. It’s possible there could be other factors involved—maybe the kids got less sleep because they were also sick, which impacted their ability to sleep and focus on the test. Also, a small p value doesn’t tell us how large or important the difference is. Maybe the extra sleep only helped improve the scores by a tiny bit.
The American Statistical Association’s Caution
In 2016, the American Statistical Association (ASA) issued a statement warning against the misuse and misinterpretation of p values. They emphasized these key points:
- P values cannot determine the importance or size of an effect or result.
- They do not provide a good measure of evidence regarding a model or hypothesis.
- Scientific conclusions and business or policy decisions should not be based only on whether a p value passes a specific threshold.
- A p value does not measure the probability that the studied hypothesis is true, nor the probability that the data were produced by random chance alone.
In the ASA’s view, the p value has been widely misused and misunderstood, often leading to conclusions that distract us from making progress on true scientific findings.
Shifting Perspectives in Research
Following suit, many in the research community, including the American Psychological Association, are advocating for a move away from the strict reliance on p value. They’re encouraging the use of a broader set of statistical tools that provide a more comprehensive view of the data. These include effect sizes, confidence intervals, and Bayesian methods, all of which contribute to a richer, more accurate understanding of research findings.
What This Means for SoundRocket Collaborators
We’re committed to robust and insightful research methods and support these shifts in the field. While p values might be familiar, they’re often not the most informative or straightforward way to understand data. While our expertise is in data collection, we are often engaged in data reporting and analysis—a common deliverable is a basic set of data tables. Often we receive requests to include statistical tests with p values in those tables.
When we do have an opportunity to provide some analyses, we will focus on providing exploratory tables, which include descriptive statistics only (and no statistical tests), or we will engage in more robust analyses that are appropriate to address your research questions.
We view statistical tests as tools to uncover and understand the complexities of social phenomena. With SoundRocket, you’ll receive not just data, but insights and interpretations that align with the latest in scientific (including statistical) best practices, helping you make informed, effective decisions.
References:
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133.
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350.
- Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.
- American Statistical Association. (2016). ASA statement on statistical significance and p-values. Retrieved from https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
- McShane, B. B., Gal, D., Gelman, A., Robert, C., & Tackett, J. L. (2019). Abandon statistical significance. The American Statistician, 73(sup1), 235-245.
- American Psychological Association. (2019). Publication Manual of the American Psychological Association (7th ed.).
- Nature Editorial Board. (2019). Scientists rise up against statistical significance. Nature, 567, 305-307.