“Footprint of Publication Selection Bias on Meta-Analyses in Medicine, Environmental Sciences, Psychology, and Economics”, 2024-02-07 ():
What is already known: Publication selection bias, where studies with statistically-significant or positive results are more likely to be reported and published, distorts the available scientific record.
What is new:
This study surveyed over 68,000 meta-analyses from medicine, environmental sciences, psychology, and economics to assess the extent of publication selection bias. As a result, it underscores the importance of addressing publication bias in evidence synthesis.
Results suggest that meta-analyses in economics are the most affected by publication selection bias, followed by environmental sciences and psychology. In contrast, meta-analyses in medicine are suggested to be the least affected. Yet, notable biases are found across all of these scientific disciplines.
Potential impact for readers: This study documents the potential extent of publication bias in different fields, which could help researchers and the public better understand the limitations of research and the potential biases of research synthesis.
Publication selection bias undermines the systematic accumulation of evidence.
To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421).
After adjusting for publication selection bias [using Bayesian model-averaging by RoBMA-PSMA], the median probability of the presence of an effect decreased 99.9% → 29.7% in economics, 98.9% → 55.7% in psychology, 99.8% → 70.7% in environmental sciences, and 38.0% → 29.7% in medicine.
The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.
Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least.
See Also: