A Cautionary Tale For Legal Empiricists
Having discovered statistical significance testing and thus p-values, (far too) many legal scholars have set about trying to uncover previously unknown first principles of tort law, otherwise unrecognizable causal relationships between stock markets and telecom regulations, and the secret biases of judges they don't like. Running a statistics app over the top of a bunch of numbers is easy these days. Anyone who can operate a laptop can do it. But finding someone who knows what's going on behind the overlay and why is as rare as finding someone who knows what's going on behind the Windows operating system and why.
Indeed, if you don't understand the bundle of assumptions, limitations and trade-offs that underlies statistical significance testing, especially when it's being misused as a form of semi-automated induction, you can wind up proving that ESP is real. That's the real moral of the NYTimes' story "Journal's Paper on ESP Expected to Prompt Outrage" and it's why the publication of the paper discussed in the story is so deliciously subversive. I'm quite certain that a large part of the outrage isn't that junk science is being published - everybody already knows that most published scientific findings are false anyway. And it's not that people are going to realize that the peer-review process does not stop and was never designed to stop (so long as the methods and conclusions drawn from the data appear sound) obviously untrue conclusions about how nature works from being published - most people know that too. No, the real outrage I'd wager is from those who aren't happy that word has gotten out that the use of statistical significance testing, particularly in the social sciences, can be used to "scientifically prove" whatever you want to prove.
This week's NYTimes' Science and Health pages have surely been an eye-opener for those who think that what gets published in science journals is at least probably true. They've covered the XMRV controversy (about a made-up cause of a made-up disease), the debate over whether the autism-vaccine hoax was junk science or outright fraud and now the scientific peer-reviewed evidence that extra-sensory perception, at least when it comes to "erotic images", is real. If nothing else it may at least cause some people to be a bit more skeptical about science, even if it is peer-reviewed.
For those of you defending mass tort cases or handling any case where significance testing raises its head be sure to read "Evidence For Feeling The Future: An Assessment of The Evidence For Feeling The Future With A Discussionn of Bayes Factor and Significance Testing" linked at the end of the Times' story. Remember always that when using such frequentist techniques say in a chemical plant - styrene study you can never prove that styrene doesn't cause cancer - even if it doesn't - but if you collect enough data and test it enough you're guaranteed to eventually come up with some scientific evidence that will pass peer-review showing styrene does cause some form of cancer even though it doesn't. The best thing about the paper are the graphs. They're at the very end though referenced early on. If you don't want to wade through it here's the conclusion: "Our main methodological concern is that inference by p-values fails to seriously consider the null hypothesis as a viable possibility. Consequently, researchers who use it are apt to reject the null on the basis of insufficient evidence. We recommend that researchers adopt Bayes factor methodology, because this approach provides a rational and consistent assessment of the relative evidence between any two hypotheses (citing Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242)
