In the context of publication bias, what we’re interested in is how the effect size and the sample size relate to one another. If you plot one versus the other, with one dot per study, you’d expect your graph to look something like Figure 2A below. (Note that this is an idealised version of a meta-analysis; real datasets almost never look this clear-cut.) Looking at this ‘funnel plot’ (so named for what are hopefully obvious reasons), you can see how all the smaller studies, towards the bottom of the y-axis, fluctuate widely; as you progress up the y-axis, to bigger studies, the dots begin to cluster around the average effect, illustrating what we’ve just discussed about larger studies being more precise. The variation on the x-axis is why it’s a bad idea to take for granted the effect from any individual study: even though there is a real effect in this example, individual studies have under- and over-shot its ‘true’ size by varying degrees (though the biggest studies do an admirable job). In any case, nothing seems to be missing here: the upside-down funnel shape is just what we’d expect if all the studies had converged upon a real effect.
<s>Click to enlarge.
Figure 2. Funnel plots from an imaginary meta-analysis, in two different scenarios. In scenario A, the distribution of the thirty studies is about what you’d expect if every study ever done on the topic had been published. In scenario B, the six studies from the bottom-left section (studies with small samples and small effects) are missing – a pattern that might signal publication bias. The vertical line in the middle of each graph is the overall effect size calculated by each meta-analysis. In the case of scenario B, it’s been shifted to the right, meaning that the meta-analysis is coming up with a bigger effect than it should.</s>
Just as in an archaeological dig, where the absence of particular objects tells you interesting things about the historical people you’re investigating – for instance, a lack of weapons might mean they were civilians rather than soldiers – we can learn a lot from what we don’t see in a meta-analysis. What if our plot looks more like Figure 2B? Here, we’ve lost a chunk from the expected shape. The studies we’d expect to see in the lower left of our, which had small sample sizes as well as small effects, are missing. Thinking like an archaeologist, a meta-analyst might infer that those studies were done, yet instead of being published, were file-drawered. Why? A likely explanation is that these small-sample, small-effect studies had p-values higher than 0.05 and were dismissed as unimportant nulls.
Perhaps the scientists who ran these studies thought something like: ‘well, it was only a small study, and the small effect I found is probably just due to noisy data. Come to think of it, I was silly even to expect to find an effect here! There’s no point in trying to publish this.’ Crucially, though, this post hoc rationalisation wouldn’t have occurred to them if the same small-sample study, with its potentially noisy data, happened to show a large effect: they’d have eagerly sent off their positive results to a journal. This double standard, based on the entrenched human tendency towards confirmation bias (interpreting evidence in the way that fits our pre-existing beliefs and desires), is what’s at the root of publication bias.
If you consider the overall conclusions of a meta-analysis based on Figure 2B rather than 2A, you can see how publication bias deranges the scientific literature. If the studies with small effects have been removed from the funnel shape, the overall effect that shows up in the meta-analysis will by definition be larger than is justified. We get an exaggerated view of the importance of the effects and can be misled into believing something exists when it doesn’t. By failing to publish null or ambiguous studies, researchers force blinkers onto anyone who reads the scientific literature.
No comments:
Post a Comment