"... It is Unacceptably Easy to Publish "Statistically Significant" Evidence Consistent with Any Hypothesis"
Want to look and feel younger? Well, there's a properly done study, statistically significant at p < .05, showing that people who listen to The Beatles "When I'm Sixty-Four" actually became a year and a half younger! Far fetched? Sure, but no more so than umpteen conclusions published in the scientific literature every day purporting to establish some causal connection based on nothing more than a statistical analysis of a series of observations. That's the point demonstrated conclusively in False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.
Though a paper may validly claim that the likelihood of the reported causal connection being due to chance alone is 5% or less (i.e. p < .05), "False-Positive Psychology" demonstrates that researchers free to modify as few as four variables (such as the number of observations to be made, sorting those observed by gender or stratifying outcomes) more likely than not have "discovered" a causal association that doesn't exist. The harm done by publishing such false-positives are obvious. As the authors put it:
"First, once they appear in the literature, false positives are particularly persistent. Because null results have many possible causes, failures to replicate previous findings are never conclusive. Furthermore, because it is uncommon for prestigious journals to publish null findings or exact replications, researchers have little incentive to even attempt them. Second, false positives waste resources: They inspire investment in fruitless research programs and can lead to ineffective policy changes. Finally, a field known for publishing false positives risks losing its credibility."
To vaccinate against infecting the scientific literature with false-positives the authors conclude with suggestions similar to those we've seen elsewhere in efforts to promote evidence-based science. At the heart of the suggested approach is transparency from the moment the experiment is conceived all the way through publication. There are six for authors and four for reviewers; be sure to read them all.
