Is Economics Self-Correcting?

The subtitle of that article is Replications in the American Economic Review, and the authors are Jörg Ankel-Peters, Nathan Fiala, and Florian Neubauer.  Here is the abstract:

Replication and constructive controversy are essential for scientific progress. This paper reviews the impact of all replications published as comments in the American Economic Review between 2010 and 2020. We investigate the citation rates of comments and whether a comment affects its original paper’s citation rates. We find that most comments are barely cited, and they have no impact on the original papers’ subsequent citations. This finding holds for original papers for which the comment diagnoses a substantive problem. We conclude from these citation patterns that replications do not update the economics literature. In an online opinion survey, we elicited viewpoints of both comment authors and original authors and find that in most cases, there is no consensus regarding the replication’s success and to what extent the original paper’s contribution sustains. This resonates with the conventional wisdom that robustness and replicability are hard to define in economics.

If you see a critical comment in the AER, the odds that it is correct, and significantly so, are really pretty high, given the barriers to getting in.  Yet no one seems to care.  (Note that the lack of caring is connected to the Bayesian inference that the published critical comments likely are correct.)  This is to me one of the more significant indictments of the economics professions as we know it today.  And it is not obvious how we might change this state of affairs.

Here is the argument in tweet storm form.