“Statistically-Significant Meta-Analyses of Clinical Trials Have Modest Credibility and Inflated Effects”, 2011-10 ():
Most statistically-significant results from meta-analyses of clinical trials are more likely to reflect truly non-null effects than false-positive results.
It is more probable that the credibility of the updated meta-analyses increases rather than decreases.
Data added to the existing meta-analysis in a 5-year window (2005–5201014ya) indicated less prominent effects than did the summary estimates in 2005.
The median fold change in these summary estimates was 0.85, but the reduction was greater for meta-analyses with less cumulative data (median reduction of 0.67×).
Objective: To assess whether nominally statistically-significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.
Study Design & Setting: Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and 4 or more trials in 2005 with p < 0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.
Results: 461 meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005–5201014ya) than in the respective meta-analyses in 2005 (median 0.85×, interquartile range [IQR]: 0.66–1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67×, IQR: 0.54–0.96). Mean credibility of the 461 meta-analyses in 2005 was 63–84% depending on the assumptions made. Credibility estimates changed >20% in 19–31 (24–39%) of the 80 updated meta-analyses.
Conclusion: Most meta-analyses with nominally statistically-significant results pertain to truly non-null effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.
[Keywords: meta-analysis, bias, treatment effect, Bayes factor, winner’s curse, outcomes]