“Comparing Meta-Analyses and Preregistered Multiple-Laboratory Replication Projects”, 2019-12-23 (; backlinks; similar):
Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes.
We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications.
We find that meta-analytic effect sizes are statistically-significantly different from replication effect sizes for 12⁄15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost 3× as large as replication effect sizes.
We also implement 3 methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results.