One year after a damning review suggested that many published clinical trials contain statistical errors, The New England Journal of Medicine (NEJM) today is correcting five of the papers fingered and retracting and republishing a sixth, about whether a Mediterranean diet helps prevent heart disease. (Spoiler alert: It still does, according to the new version of the paper.)
The problem is that journal editors and referees are quite lax about reviewing statistics, or even data, and most journals allow studies to be submitted and published without ever even seeing the data and calculations. I would think that the first rule from a proper journal would be for researchers to ‘show their work’, but no.
The ‘damning review’ was really a test to see if so-called ‘randomization’ was really random, and turned up that many researchers, faced with logistical conundrums, would (slightly) compromise on randomization, for instance by randomizing by clinic rather than over the entire subject population. That's enough to introduce consequential error (maybe), and thus should have been noted and accounted for. As it happens, other than the Mediterranean diet study that was ultimately republished with the same big-picture result, no studies have (yet) been retracted, but some of the questionable studies are too old for reanalysis. One hopes that the remaining questionable studies will at least get an editor's note online and an asterisk in the search tools used by other researchers.
The last paragraph of the story is worth a laugh:
Although most of the errors so far are minor ones, [Anaesthesia editor John] Carlisle wonders whether they’re a harbinger of statistical problems in parts of papers he didn’t examine, such as the all-important results section. [NEJM Editor-in-Chief Jeffrey] Drazen was unsettled enough by what his own journal found to give his manuscript editors a statistics course, and implement extra scrutiny of statistics in accepted papers.
Great. But why weren't they doing this already, or requiring better credentials from their editors? Apparently, that's a problem for most journals, medical or otherwise.
Carlisle's discovery screams for a deeper dive into those results sections that he worries about. It will be hard, slogging work, but apparently necessary.