Odds Are, It’s Wrong

Researchers’ dirty little secret:

“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”

Statistical problems also afflict the “gold standard” for medical research, the randomized, controlled clinical trials that test drugs for their ability to cure or their power to harm. Such trials assign patients at random to receive either the substance being tested or a placebo, typically a sugar pill; random selection supposedly guarantees that patients’ personal characteristics won’t bias the choice of who gets the actual treatment. But in practice, selection biases may still occur, Vance Berger and Sherri Weinstein noted in 2004 in ControlledClinical Trials. “Some of the benefits ascribed to randomization, for example that it eliminates all selection bias, can better be described as fantasy than reality,” they wrote.

2 thoughts on “Odds Are, It’s Wrong

  1. Biostatistics and double-blind trials are difficult to interpret and report on, no doubt about it. And you can make the case that, in effect, ALL science is wrong at the moment you take a snapshot of it–it's just less wrong that it was last year or ten years ago or a hundred years ago. That's the nature of truth-seeking endeavors.

  2. Great point. Thanks. You make the case for the dangers of what Anne Lamott calls excessive certitude.

Comments are closed.