Concentrate And Ask Again
You wouldn't think it reasonable to test a scientific hypothesis by consulting the Magic 8-Ball. You can't imagine any scientist asking: "should I reject the null hypothesis (e.g. that coffee doesn't cause pancreatic cancer)?" and then turning the Magic 8-Ball over to discover the answer. That's because every adolescent who ever wondered hopefully about a pretty girl: "does she really like me?" knows that subtle reformulations of the question and repeat inquiries will eventually cause the Magic 8-Ball to reveal the desired answer. Well, unfortunately, it turns out that some scientists learned the lesson of the Magic 8-Ball only too well.
We've complained repeatedly about the problems resulting from the effort to turn the discovery of knowledge into an academic industrial manufacturing process in which the only quality assurance check is the test of statistical significance as gauged by the p-value. Reliance on a low p-value as some sort of modern Oracle is the reason that most such "science" is wrong. We know why low p-values alone offer little assurance that a conjecture has not been refuted but what accounts for all the low p-values? In psych research, at least, it's now pretty clear that it's the ability to sample until you hit that magic p<0.05 level.
Assume scientists wrote up their experiments, set the sample parameters, ran the tests, gathered the data, analyzed the results and published what they found. The p-values calculated across all hypotheses tested ought to be distributed randomly across the range of potential values, right? What would you think if it turned out that that there were an unusually large clump of p-values just ever so slightly <0.05? You might think "publication bias!". Or you might think "editors don't understand that low p-values don't have much to say about one of the essential elements for establishing reliability: reproducibility." But what you should really be considering is whether or not the researcher, thanks to the power of computers and modern stats packages, wasn't watching the data accumulate and then either rationalized halting the program once statistical significance was achieved or rationalized collecting more and more data until the statistical significance was achieved i.e. until the Magic 8-Ball proclaimed "You may rely on it"; at which point he quit testing and instead set about writing up his latest scientific discovery. The paper reporting the suspicious clump and the unsettling reasons for it is "A Peculiar Prevalence of p Values Just Below .05" It's well worth your time if you're interesting in knowing why so much "science" is so wrong nowadays.
In fact, the problem of irreproducible "science" has become so harmful and so widespread that a plan to enlist independent laboratories to attempt to verify the findings of high-profile research before they're published has been implemented. You can read about it at Nature in "Independent labs to verify high-profile papers".
Remember, only you can prevent experts, bearing "science" resting on nothing more than statistical significance, from depriving your client of her life, liberty or hard earned cash.
