Scientifically valid findings should replicate, p < .005

Real Clear Science

"Typically, biomedical scientists use a significance level of 0.05 to test new hypotheses or to announce discoveries. In plain English, this means that scientists are willing to accept a 5% probability that the data they collected occurred merely by chance. The scientific community has agreed that the arbitrary "5% chance of being wrong" standard is sufficient for most research.

Not so, says Dr. Valen Johnson of Texas A&M University. He believes the 0.05 significance level needs to be replaced. Using the Bayes factor -- a ratio that measures the probability of competing hypotheses -- Dr. Johnson shows that the 0.05 standard is terribly insufficient. (See figure.)

According to his analysis, a significance level (denoted with the Greek letter "alpha" on the X-axis) of 0.05 corresponds to a Bayes factor of around 3 to 5. That's rather pathetic, as it means the newer, more fascinating hypothesis is only 3 to 5 times more likely to explain the data than the old, boring one.

Significance levels more stringent than 0.05 yield much more convincing Bayes factors. For 0.01, the Bayes factor range is 12 to 20; for 0.005, the range is 25 to 50; and for 0.001, the range is 100 to 200. A Bayes factor of 100 or greater is considered "decisive" evidence in favor of the new hypothesis.

Dr. Johnson advises the scientific community to replace its 0.05 significance level with 0.005. If his statistical analysis is correct, it's difficult to argue with him."

Source: Johnson VE. "Revised standards for statistical evidence." PNAS. Published online before print, 11-Nov-2013.