“Do Pre-Registration and Pre-Analysis Plans Reduce p-Hacking and Publication Bias?”, Abel Brodeur, Nikolai Cook, Jonathan Hartley, Anthony Heyes2022-08-03 (, )⁠:

randomized controlled trials (RCTs) are increasingly prominent in economics, with pre-registration and pre-analysis plans (PAPs) promoted as important in ensuring the credibility of findings.

We investigate whether these tools reduce the extent of p-hacking and publication bias by collecting and studying the universe of test statistics, 15,992 in total, from RCTs published in 15 leading economics journals from 2018 through 2021.

In our primary analysis, we find no meaningful difference in the distribution of test statistics from pre-registered studies, compared to their non-pre-registered counterparts. However, pre-registered studies that have a complete PAP are statistically-significantly less p-hacked.

These results point to the importance of PAPs, rather than pre-registration in itself, in ensuring credibility.

[Keywords: pre-analysis plan, pre-registration, p-hacking, publication bias, research credibility]

Figure 3: Test Statistics Distribution by Pre-Registration Status. Notes: This figure displays the distribution of test statistics for z ∈ [0, 10] from randomized control trials from 2018–32021 by pre-registration status. We define a pre-registered RCT as a study that was registered before its trial end date listed in a registry. Studies that were registered after the trial end date are counted as non-preregistered. The tests are from studies published in 15 leading economics journals. Bins are 0.1 wide. We have also superimposed an Epanechnikov kernel. We do not weight articles.

…Figure 6 illustrates the distribution of test statistics for pre-registered RCTs for those containing a PAP (right panel) and those without a PAP (left panel), respectively.18 Both curves are monotonically falling with a bump around the 5% statistical-significance threshold. But, of note, visually the bump is more pronounced for the curve without a PAP and the curve for PAP contains more tests with large p-values.

Figure 6: Test Statistics Distribution for Pre-Registered RCTs by a Presence of Pre-Analysis Plan. Notes: This figure displays the distribution of test statistics for z ∈ [0, 10] from randomized control trials that were pre-registered from 2018–32021 by presence of a pre-analysis plan. We define a pre-registered RCT as a study that was registered before its trial end date listed in a registry. Studies that were registered after the trial end date are counted as non-pre-registered. The tests are from studies published in 15 leading economics journals. Bins are 0.1 wide. We have also superimposed an Epanechnikov kernel. We do not weight articles.

See Also:

View External Link:

Do Pre-Registration and Pre-Analysis Plans Reduce p-Hacking and Publication Bias?