… we have a universally recognized Supreme Court, to which all disputes are taken eventually, and from whose verdict there is no appeal. I refer, of course, to direct experimental observation of the facts.
E. T. Jaynes, physicist, in Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science (1976)
Recently we’ve been grousing about Daubert-invoking opinions that are actually derived from the strange belief that reliable scientific knowledge does not depend upon the existence of supporting observable facts. Today however we’re applauding City of Pomona v. SQM North America Corporation; an opinion that gets Daubert and sound science right because it gets the scientific method right.
At issue was the trial court’s order excluding an expert witness for the City of Pomona in a groundwater perchlorate contamination case. The expert, Dr. Neil Sturchio, was prepared to testify that the perchlorate detected in the city’s groundwater had most likely originated in the Atacama Desert of Chile. Such evidence, if admissible, would implicate defendant SQM as it had imported for sale to California’s agricultural industry many thousands of metric tons of sodium nitrate, an inorganic nitrogen fertilizer, from Chile’s Atacama Desert – the sodium nitrate from which contains on average about 0.1% perchlorate.
It’s a big world though and there are plenty of potential sources of perchlorate in groundwater. Perchlorate occurs naturally and has been found in relative abundance in some arid regions of the American Southwest. It’s also synthesized industrially for use in solid fuel rocket propulsion systems (manufacturers of which were located in and around Pomona) and fireworks so that defense contractors, the Cal Poly Rocketry Club and 4th of July celebrations all fall under suspicion. How then did Dr. Sturchio determine that the perchlorate in Pomona’s groundwater came from a desert almost 6,000 miles away? Well, he didn’t conduct an experiment in his mind (a la McMunn and Harris) involving unobservable facts (a la Messick) in order to generate an untestable conclusion (a la Milward). Instead he followed the scientific method.
Perchlorate consists of one chlorine atom and four oxygen atoms. Since both chlorine and oxygen atoms come in different varieties (called isotopes) the question arose as to whether perchlorate also comes in different varieties defined by the ratio (or distribution) of the various isotopes making up its component parts. If it does, and if those varieties vary according to where they’re from or how they’re made, then something akin to a fingerprint or signature could be generated by analyzing the relative distribution of isotopes in a perchlorate sample. Finding a match in a database of known perchlorate signatures would thereafter yield a suspect and so satisfy plaintiff’s evidentiary burden of production.
According to Dr. Sturchio there is a method (it’s a subtype of mass spectroscopy known as isotope-ratio, or IRMS) for assessing the ratio of isotopes (or variety) of a particular perchlorate sample and there is a list, albeit a short one, of known perchlorate samples and their IRMS signatures. Most importantly there’s a detailed report available that sets out the theory, its rationale, the prediction it makes, the method by which it was tested and the results obtained. Using that method and comparing the results from prior tests on other samples showed, according to Dr. Sturchio, that the perchlorate in the Pomona, CA groundwater had a fingerprint, a signature if you will, just like that of Chilean perchlorate; and not at all like that of either man-made or indigenous perchlorate. Thus his opinion on its origin.
The district court found Dr. Sturchio’s opinion to be unreliable for three reasons. The first was that no government agency had yet certified the method he employed and that it was still being revised. The second was that the particular procedure he had followed in this instance had not been tested and could not be retested. The third was that the database of known perchlorate varieties and their signatures was too small.
Because the theory that perchlorate comes in different varieties has been tested and corroborated by at least one study, and because those varieties can per the theory be distinguished by use of a widely available method, the Ninth quite rightly held that the criteria at the heart of Daubert, testability and reliability, had been satisfied. Defendant was free to test the theory as well as the method and to refute the former and/or explain why the latter was either inappropriate for the task or so inaccurate as to be unreliable, Without evidence of its own to refute plaintiff’s corroborated theory or the accuracy of the method used to test it Dr. Sturchio’s opinions could not be excluded. In such a case, the court held, arguments aimed at the potential for holes in the theory or error in the methodology go to the weight and not the admissibility of the proffered evidence.
The court did a nice job (with one exception to be discussed below) in setting out its reasoning and so rather than go on about it we’ll just refer you to the opinion. Instead we’ll make one comment and then try to explain why the court’s argument against the need for certainty isn’t the usual straw man argument but is rather an understandably awkward attempt to address the problem known as "the glory of science and the scandal of philosophy".
Our comment goes to the defendant’s argument that the reliability of a scientific method depends upon final government certification of the particular technique. We file it under "Be careful what you wish for" and remind our readers that the power to define what is and what is not a telescope determines what can and cannot be seen through one.
Now for the hard part. Defendant’s argument that the database of perchlorate signatures was too small and the court’s response that the law doesn’t require certainty is just a version of a very old argument that goes like this:
Scientist: If all swans are white then every swan that is seen shall be white. Since all swans ever seen have been white, all swans must be white. It is thus my opinion that if the color of a particular swan is at issue then it will have been white.
Inquisitor: I shall demonstrate that you have fallen into the fallacy known as Affirming the Consequent; which is recognized by all learned men and women to constitute a notorious error of logic. Have you seen, or seen a record of, every swan that lives or has ever lived?
Scientist: I have not.
Inquisitor: So there are swans that have been that have never been seen and swans now living that remain undiscovered and yet you have no knowledge of their color?
Scientist: it is undeniable; and I would add that there are swans yet to be and I am of the opinion that they too will be white.
Inquisitor: Then you further admit that there is an entire category of things, that of swans yet to be, about the color of which you are willing to opine despite having not a single observation of any member of that category?
Scientist: I do. It is after all prediction on which a scientist stakes her reputation.
Inquisitor: A risky gamble, is it not? For despite ten thousand observations of white swans the sighting of a single black one would do what to your theory that all swans are white?
Scientist: It would refute it, utterly.
Inquisitor: How then can your theory be anything more than the most tenuous sort of idle speculation - built out of nothing more than anecdotes, a collection of sightings of some unknown portion of all swans, and at risk of collapse at any moment?
Scientist: Because all those confirmatory sightings with none to the contrary make it at least very probably true.
Inquisitor: Alas, I had hoped you had something more and might save me from my skepticism. Oh well. Do you agree that probability is a measure of uncertainty?
Scientist: Yes, it plays that role in logic.
Inquisitor: And what is meant by uncertainty?
Scientist: To take the case of a coin toss, for example, while I cannot tell you whether when flipped it will come up heads or tails I can tell you the expected frequency of each occurrence; or, given past experience flipping the same coin, I can tell you to what degree I believe the coin will come up say heads. Any probability calculated other than 100% is then a measure of uncertainty.
Inquisitor: But to make such calculations you must know how many sides the coin has or how many cards are in the deck?
Inquisitor: How many swans have there ever been?
Scientist: I do not know.
Inquisitor: And how many swans are yet to be?
Scientist: I do not know that either.
Inquisitor: So you cannot say whether your record of white swan observations constitutes a large, small or merely infinitesimal fraction of all the swans that have ever been or will ever be?
Scientist: I cannot
Inquisitor: And so when you use the word probability you use it in neither the mathematical nor the logical sense, since you have insufficient information to even estimate the probability that your opinion is correct?
Scientist: That is true.
Inquisitor: So now, at last, do you admit that you really do not know?
Scientist (revealed as Sir A. B Hill): Who knows, asked Robert Browning, but the world may end tonight? True, but on available evidence most of us make ready to commute at 8:30 the next day.
In City of Pomona plaintiff’s expert had real evidence – facts that are directly observable and subject to testing – and not just a hunch. Those facts and the method by which they had been gathered had been tested by others, and confirmed. And defendant was free to find a black swan. That, in our view, is good enough.