Getting at the Truth: Improving Systematic Reviews
Systematic reviews and meta-analyses are considered to be perhaps the best evidence about treatments/exposures and outcomes from which causal inferences can be drawn. The problem is that they're susceptible to a variety of biases. See e.g. "Bias Due to Changes in Specified Outcomes During the Systematic Review Process".
In the litigation context we increasingly see a systematic review launched by an expert-to-be before the first suit is even filed. They will thus have already gone through the studies; used their "judgment" to select which ones ought to be considered (and which ought not); determined how much weight to give each; and established a protocol for conducting the review which, surprise surprise, demonstrates a causal link supporting the lawsuits. The strong suspicion is that selection bias (a form of the Texas Sharpshooter effect) is at work (and that, of course, is being charitable). And the biggest problem is that uncovering selection bias is notoriously difficult.
Now there's a move afoot to bring transparency to systematic reviews by requiring that plans, methods and protocols be registered before the systematic review is begun. See "Best Practice in Systematic Reviews: The Importance of Protocols and Registration"; "New Initiative to Make Systematic Review Protocols More Transparent" and "Open Medicine Endorses PROSPERO". Here's the press release from the Cochrane Collaboration and here's a link to PROSPERO.
Until we get courts to make experts disclose their methods and justifications for selecting, weighing and interpreting data before they develop their opinions articles like the following may come in handy: "How Can We Improve the Interpretation of Systematic Reviews?"
