The ASA’s statement on p-values: context, process, and purpose
Ronald L. Wasserstein & Nicole A. Lazar
P-values are a hotly debated topic recently. Al throughout school I learned that a p value of 0.05 makes or breaks any scientific progress. This paper, in the form of a statement from American Statistical Association, lays out all the flaws and common misuse of p-values. THe article starts with the motivation of the statement, this is a rare move for the ASA. This lends a little gravity to the paper. The statement; however, is where all the interesting parts lie. In the statement, there is a little obtuse definition of P value, quoted below:
“Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups would be equal to or more extreme than its observed value.”
Not exactly the clearest definition. However, the power of this paper lies in the Principles section. ASA outlines 6 principles of proper usage of p-values: replicated in the list below:
1. P-values can indicate how incompatible the data are with a specified statistical model
2.P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decision should not be based only on whether a p-value passes a specific threshold
4. proper inference requires full reporting and transparency
5. a p-value, or statistical significance, does not measure the size of an effect or the importance of a result
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Wow, everyone who works with p-values should have to recite these rules before they begin their analysis. They should have them framed above their desk. I know in the past I have gone against 2,5, and 6. I highly suggest reading this statement. It will take you less than 20 minutes. If you cant spare that, just read the last line:
“No single index should substitute for scientific reasoning”
– Marcello