To assess the replicability of social priming findings we reviewed the extant close replication attempts in the field.
In total, we found 65 close replications, that replicated 46 unique findings.
94% of the replications had effect sizes smaller than the effect they replicated, only 18% of the replications reported a statistically-significant p-value in the original direction, and the 95% confidence interval of the replication effects included the original effects only 26% of the time. The strongest predictor of replication success was whether or not the replication team included at least one of the authors of the original paper. 12 of the 16 replications with at least one original author produced a statistically-significant effect in the original direction and the meta-analytic average of these studies suggest a statistically-significant priming effect (d = 0.33, 95% CI [0.26; 0.65]). In stark contrast, 0 of the 49 replications by independent research teams produced a statistically-significant effect in the original direction and the meta-analytic average was virtually zero (d = 0.001, 95% CI [−0.03; 0.03]).
Figure 2: Beeswarm boxplots illustrating the distribution of p-values (A) and effect sizes (B). The horizontal dotted line in (A) equals 0.05. The horizontal dotted line in (B) equals zero. Diamonds equal unweighted means and the horizontal line in the boxplots equal unweighted medians.
We argue that these results have shifted the burden of proof back onto advocates of social priming. Successful replications from independent research teams will likely be required to convince sceptics that social priming exists at all.
[Keywords: meta-analysis, preregistration, priming, replication]