“Of Forking Paths and Tied Hands: Selective Publication of Findings, and What Economists Should Do about It”, 2021-06-01 (; similar):
A key challenge for interpreting published empirical research is the fact that published findings might be selected by researchers or by journals. Selection might be based on criteria such as statistical-significance, consistency with theory, or the surprisingness of findings or their plausibility. Selection leads to biased estimates, reduced coverage of confidence intervals, and distorted posterior beliefs.
I review methods for detecting and quantifying selection based on the distribution of p-values, systematic replication studies, and meta-studies.
I then discuss the conflicting recommendations regarding selection resulting from alternative objectives, in particular, the validity of inference versus the relevance of findings for decision-makers.
Based on this discussion, I consider various reform proposals, such as de-emphasizing statistical-significance, pre-analysis plans, journals for null results and replication studies, and a functionally differentiated publication system.
In conclusion, I argue that we need alternative foundations of statistics that go beyond the single-agent model of decision theory.