Encuentre a un experto

Open Science & Development Engineering: Evidence to Inform Improved Replication Models

Serie

  • MetaArXiv

Resumen

  • Replication is a critical component of scientific credibility. Replication increases our confidence in the reliability of knowledge generated by original research. Yet, replication is the exception rather than the rule in economics. Few replications are completed and even fewer are published. Indeed, in the last 6 years, only 11 replication studies were published in top 11 (empirical) Economics Journals. In this paper, we examine why so little replication is done and propose changes to the incentives to replicate in order to solve these problems. Our study focuses on code replication, which seeks to replicate the results in the original paper uses the same data as the original study. Specifically, these studies seek to replicate exactly the same analyses performed by the authors. The objective is to verify that the analysis code is correct and confirm that there are no coding errors. This is usually done in a two-step process. The first step is to reconstruct the sample and variables used in the analysis from the raw data. The second step is to confirm that the analysis code (i.e., the code that fits a statistical model to the data) reproduces the reported results. By definition, the results reported in the original paper can then be replicated if this two-step procedure is successful. The threat of code replication provides an incentive for authors to put more effort into writing the code to avoid errors and incentive not to purposely misreport results. We analyze the effectiveness of code replication in the context a model that has three characteristics: 1. Unbiasedness: there is no “overturn bias” i.e., the model does not create incentives to find or claim mistakes in the original analysis. 2. Fair: all papers have some probability of being replicated, the sample of papers replicated is representative, and the likelihood that a paper is replicated is independent of author identity, topic, and results. 3. Cost: the model should provide the right incentives at low.

fecha de publicación

  • 2017

Issue

  • ryvud