A statistical technique to check whether randomised trials really are randomised

From Science Fictions by Stuart Ritchie.  Page 129.  

Numerical errors are also disturbingly common in scientific fields with much higher stakes. You may recall that the world’s most prolific known scientific fraudster, at the time of writing anyway, is the anaesthesiologist Yoshitaka Fujii. The analysis that ended his long spree of data forging was run by the anaesthetist John Carlisle, who devised a statistical technique to check whether randomised trials really are randomised.  

Randomisation is done by essentially flipping a coin for each participant in order to assign them to a group (say, the active drug group versus the placebo group) at random, rather than in some pre-planned way that might be prone to bias. This process is crucial: the point is to ensure before the trial begins (in the jargon, ‘at baseline’) that there are no substantial differences between the groups. If one group is healthier, better educated, older or markedly different in any other way that might affect the results, the trial won’t be a fair test.  And so, if there are big differences between the groups at the beginning of a randomised controlled trial, there’s an issue: the randomisation process must’ve failed. On the flipside, if the groups are perfectly matched,  inexplicably avoiding the iron rule of the noisiness of all numbers, that’s problematic too: even after randomisation you’d still expect there to be tiny differences between the groups, just by chance. This is what Carlisle’s method relies upon. When he checked Fujii’s papers, he found data of completely implausible consistency: the reported age, height, and weight distributions for Fujii’s patients, for instance, were almost perfectly synchronised. The odds of that happening in reality were less than one in ten to the thirty-third power (that is, one in a billion trillion trillions).  Sure enough, it turned out that Fujii was a fraud.

In 2017, Carlisle applied his error-spotting technique to 5,087 medical trials from eight journals, again checking for randomisation that was either faulty or suspiciously perfect.  Of course, it remains the case that some trials might look suspicious just by bad luck. But even after taking this into account, Carlisle found that 5 per cent of trials had suspicious data: his results thus pointed to hundreds of studies that might have been completely corrupted – their results rendered meaningless – by a failure to randomise their groups properly. Fujii-like fraud was responsible only for a small proportion of these broken trials; in all likelihood, Carlisle had mainly uncovered innocent mistakes. Considering what’s at stake in a medical trial, though, with doctors using the results to choose treatments for their patients, those innocent mistakes could end up being extraordinarily serious.