Lies, Damned Lies, and P-Values
In "Odds Are, It's Wrong", Tom Siegfried lays out the argument for the proposition that much of what you read in the scientific literature is wrong because many of the claims being made rely on statistical significance. You see, an impressive sounding statement like "the association between exposure and disease was highly significant (P<0.05)" does NOT mean (a) that there's a 95% chance that the association is causal; (b) that the absence of an association can almost certainly be ruled out; nor does it necessarily mean that (c) the finding is momentous, compelling or even important. It doesn't even say that if the test were to be repeated that its results would likely hold. A P-value, the arbitrary judge of "statistical significance", won't, and can't, have anything to say about the likelihood that a given hypothesis is or is not true.
The fact of the matter is that if you have a bunch of data and can't find at least one statistically significant association it in only proves one thing - that you're not trying hard enough. The magical P-value level of 0.05 is nothing but a trade-off; a balancing act between finding associations that don't exist (false positives) and missing true associations that do (false negatives). As a result, false associations are not only possible, they're guaranteed when you have enough data and slice it enough ways.
Now, lawyers are getting into the act. And while it's bad enough that "[a] lot of scientists don't understand statistics" (Steven Goodman quote from the "Odds Are, It's Wrong" article) it gets awful when lawyers try to deploy statistics to support or rebut claims. Law review articles are littered with claims resting on nothing more than small P-values. Some purport to show that certain appellate courts are biased against accident victims; others that tort reform is good for your health. And hardly a week goes by that I don't see a brief or a pleading asserting that Texas "jurisprudence" requires an epidemiological study with a risk ratio greater than 2 and a P<0.05 before a plaintiff can recover on a toxic tort claim.
Apparently many lawyers, especially on the defense side, either forgot or never learned that it's easy to gin up false associations that meet the greater than 2 and less than 0.05 test. In fact, that's how most categories of toxic tort claims got started. Enshrining such a test in the law would turn out to be The Full Employment Act for toxic tort lawyers.
Causal inference from epidemiological statistical analysis is a crude method that nevertheless worked well for finding big effects like that of smoking on lung cancer risk and amphibole exposure on mesothelioma risk. On more subtle effects though, at the population level or molecular level, reliance on 20th century methods has produced so much bad science of late (bad only because statistics are routinely misused and abused and not because statistics aren't powerfully effective tools when properly used) that new methods of causal analysis are beginning to replace them. And these tools can answer the question of "how likely is it that drug A caused injury B?"
To see what the future of causal proof in toxic torts will look like read: "An Introduction to Causal Inference" by Judea Pearl.
