“Is Economics Self-Correcting? Replications in the American Economic Review”, 2024-04-13 (; similar):
This paper reviews the impact of replications published as comments in the American Economic Review 2010–102020. We examine their citations and influence on the original papers’ (OPs) subsequent citations.
Our results show that comments are barely cited, and they do not affect the OP’s citations—even if the comment diagnoses substantive problems. Furthermore, we conduct an opinion survey among replicators and authors and find that there often is no consensus on whether the OP’s contribution sustains.
We conclude that the economics literature does not self-correct, and that robustness and replicability are hard to define in economics.
…For the self-correction claim to hold, we hypothesize that a comment should lead to a strong reaction of the literature, especially for a comment raising substantive concerns about an OP. If it does not respond strongly, we argue, the prior in the literature sustains. We look at two facets of a strong response: (1) Citations of the comment relative to citations to the OP after comment publication (henceforth: citation ratio), and (2) Whether the comment affects the OP’s annual citations. In our main analysis, we do not conduct formal statistical testing and rather subject these two indicators to a descriptive analysis. Effective self-correction should either lead to a very high citation ratio because the comment is cited most of the time when the OP is cited ( et al 2017; et al 2021) or to a clear and discernible effect on the OP’s annual citations.
For this, we visually inspect the OP’s annual citations before and after the publication of the comment. If there is no visibly discernible decline in citations, especially for substantive comments, we reject the hypothesis of self-correction. Note that whenever we use causal expressions such as “effect”, “impact”, or “influence”, we refer to this visual inspection, not a quantitative analysis. We believe this approach does justice to our very generic research question about scientific self-correction, as well as to the nature of our sample, which is small but contains very influential OPs and highly published comments. We argue that if there is no discernible effect on the literature for comments that made it into the AER, it is unlikely to materialize for other replications.
…We find that AER comments do not affect the OP’s citations and hence their influence on the literature. We observe an average citation ratio of 14%. Comments are cited on average 7× per year since their publication—compared to an average of 74 citations per year for the OP since publication of the comment. Comments are, hence, not cited much in absolute terms, and a lot less than the OP. The latter implies that most OP citations ignore the comment. This issue has been discussed by et al 2017 who call for a normative change toward citing the replication next to its OP, which would also ensure that “well-executed replications receive credit.” We furthermore find that the publication of a comment does not affect the OP’s citation trend. These findings confirm 2022, who, likewise, do not find what they call a “penalty” of replications on post-replication citations.