Sound Causal Inference: Transparency, Transparency, Transparency, Transparency
The recent Milward opinion out of the U.S. First Circuit Court of Appeals notwithstanding, the subjective assessment of an expert, weighing whatever evidence he selects for consideration in the scales of his own "scientific judgment", is not how causal inferences should be reached. At least not according to the National Academy of Sciences (NAS).
Last Friday NAS released its report on EPA's Integrated Risk Information System (IRIS) assessment of formaldehyde. It wasn't impressed with the overlong and detailed discussions of each study the EPA had considered. Rather, it expressed its concern that all EPA had at the end of the day was a pile of studies and its scientific judgment. Specifically the NAS reiterated, as it has in the past, that data + a statement that proper methods of causal inference had been followed ≠ sound science.
Of the criticisms leveled against the EPA's report the most frequent was for a lack of transparency. The EPA hadn't made it at all clear how it was going to go about assembling data, how it was going to assess the data, by what criteria it would include or exclude a given piece of data or how, and by what standards, it was going to weigh the data it found worthy of consideration. How can another scientist assess the soundness of an expert's conclusion if she has nothing other than "I considered the following studies and reached the following conclusion using my best judgment" to go on? She can't.
So on page 115 the NAS reviews best practices for conducting and assessing comprehensive reviews. If you're dealing with an expert who says "I used the A.B. Hill criteria and my scientific judgment to reach my opinions" you'll find the discussion that follows full of great cross examination material. Where, for example, is the evidence table prepared by the expert? What weight does he assign to each datum? How did he come by it? If a particular study has a potential for bias by how much did he lessen the weight assigned, how did he come up with the number and in what other matter has he used the same modifier to account for possible bias?
Finally, the NAS took a particularly dim view of the EPA's conclusion that formaldehyde likely causes all lymphohematopoietic (LHP) cancers. (See pages 80 - 87). There's "[t]he grouping of 'all LHP cancers' includes at least 14 biologically distinct diagnoses in humans and should not be used in determinations of causality". And "there is no clearly articulated framework for establishing causation on the basis of the weight and strength of evidence". Finally "the conclusion of causation appears to be based on a subjective view of the overall data ... [t]he absence of a causation framework is especially problematic for the individual LHP cancers, given the highly variable epidemiologic literature and the high uncertainty of mode of action."
Bottom line: if you're going to use a "weight of the evidence" approach you have to say how you decided what to weigh and by what standards you weighed it. Pretty basic stuff really.
