Earlier this month, the American Statistical Association (ASA) took on the case of the much misunderstood P-Value[i], citing such abuses as ‘p-hacking, ‘data-dredging’ and ‘bright-line rules’ contributing to injuries including ‘publication bias,’ the ‘file-drawer effect’ and the ‘reproducibility crisis’ in scientific studies[ii]. Some scientific journals have gone so far as to deny P-Value inclusion in its articles.
This controversy has relevance in product development when decisions are made based on the results of statistical analysis. Examples include:
- Which product concept is best?
- Which features or benefits should be included in a product?
- Are customers willing to buy the product?
- Does the technology deliver the desired results?
- Which technology is better?
Product developers should read the American Statistician article (starting on page 8) to understand when it is appropriate to use p-values.
P-Value has been disrespected, over-respected and misused since its birth in the 1920s to statistician R. A Fisher. As author Regina Nuzzo describes in a popular Nature article[iii], Fisher intended P-Value to suggest whether evidence was worth a deeper look, not to be a rigorous test of statistical significance. His adversaries Neyman and Pearson derided P-Value as “worse than useless,[iv]” but researchers liked P-Value’s easy-going nature. Before long, P-Value became a celebrity to a loyal following of researchers unfortunately lacking rigorous training in statistical analysis. P-Value became the de facto arbiter of statistical significance, a much more decisive role than it was born to.
In recent years, high-profile scientific studies showing conflicting and irreproducible results[v] have caused many to question P-Value’s competency as judge of statistical significance. P-Value has been investigated and found guilty of abetting some researchers in p-hacking, a felony in the discipline of statisticians, but still legal, though increasingly discouraged, in other disciplines.
Is P-Value a fraudster? Or has P-Value been misused and abused by its followers?
The ASA statement and accompanying article in The American Statistician[vi] clarify the correct role of P-Value and recommend researchers use better judgement in the tools they choose to analyze and communicate the results of their studies. For more information on how P-Value should (and should not) be used and better tests for statistical significance, read these articles:
ASA statement
https://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
American Statistician article
https://www.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.Vulyy-Z4WSQ
The ASA is the world’s largest community of statisticians and the oldest continuously operating professional science society in the United States. Its members serve in industry, government and academia in more than 90 countries, advancing research and promoting sound statistical practice to inform public policy and improve human welfare. For additional information, please visit the ASA website at www.amstat.org.
Disclaimer: Kathy Morrissey has been a member of ASA for more than 30 years and has held officer positions in ASA and the Chicago Chapter of ASA.
[i] “American Statistical Association Releases Statement on Statistical Significance and P-Values” https://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
[ii] Lehrer, J. (2010), “The Truth Wears Off,” The New Yorker. December 13, 2010, available at https://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off
[iii] Nuzzo, R. (2014), “Scientific Method: statistical errors”, Nature, 506, 150-152, available at
https://www.nature.com/news/scientific-method-statistical-errors-1.14700
[iv] Ibid.
[v] Rehman, J. (2013), “Cancer research in crisis: Are the drugs we count on based on bad science?” Salon September 1, 2013, available at https://www.salon.com/2013/09/01/is_cancer_research_facing_a_crisis/
[vi] Wasserstein, R. and Lazar, N. (2016), “The ASA’s statement on p-values: context, process and purpose, ”The American Statistician, available at https://www.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.Vulyy-Z4WSQ
