We attempted to replicate 67 macroeconomic papers published in 13 well-regarded economics journals using author-provided replication files that included both data and code by following a preanalysis plan. Aside from six papers that used confidential data, we obtained data and code replication files for 29 of 35 papers (83%) that were required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that were not required to provide data and code replication files. Defining replication success as our ability to use the author-provided data and code files to produce the key qualitative conclusions of the original paper, we successfully replicated 22 of 67 papers (33%) without contacting the authors. Excluding the six papers that used confidential data and the two papers that used software we did not possess, we replicated 29 of 59 papers (49%) with assistance from the authors. Because we were able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is often not replicable. We conclude with recommendations on improving replication of economics research.
Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Not”
The views and opinions expressed here are ours and are not necessarily those of the Board of Governors of the Federal Reserve System, the Department of the Treasury, the Office of Comptroller of the Currency, or the Office of Financial Research. The earlier version of this paper was titled “Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not.”’ This paper was started while Phillip Li was at the Office of the Comptroller of the Currency and before he joined the Office of Financial Research. We thank Sarena F. Goodman, Jan H. Höeffler, Lawrence Katz, Evan F. Koenig, Robert J. MacCoun, B.D. McCullough, Michael Nahas, Min Qi, W. Robert Reed, Bo Sun, Richard H. Thaler, J. Luke Van Cleve, Kurt von Tish, Ivo Welch, Christian Zimmermann, and seminar participants at the 2017 ASSA annual meeting, BEA, the Board, the 2016 CEA annual meeting, CEU, Duke, FRB Cleveland, FRB St. Louis, GWU, OCC, OFR, Penn State, UC - Irvine, and the 2016 WEAI annual meeting for helpful comments. We thank Tyler J. Hanson, Erik Larsson, Dustin Lartigue, Kim T. Mai, Anthony Marcozzi, Shawn M. Martin, Tyler Radler, Adam Scherling, and John Stromme for research assistance. We also thank Sophia Allison, Felix Galbis-Reig, Spencer C. Li, Misty Mirpuri, David Ratner, and Jeffrey Westphal for technical assistance. We are responsible for any errors.
C. Chang A, Li P (2022), "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Not”". Critical Finance Review, Vol. 11 No. 1 pp. 185–206, doi: https://doi.org/10.1561/104.00000053
Download citation file:
Sign in
ICE Member Sign In
Log inPurchased this content as a guest? Enter your email address to restore access.
