The reproducibility of earlier findings is fundamental to ensure the accountability and trustworthiness of the empirical sciences. Several reviews in different research fields showed that many earlier findings are not reproducible. To this end, the objective of this survey is to help setting a common ground and understanding of what reproducibility in Information Retrieval (IR) is and what can be done for reproducibility, also creating connections to similar issues in neighboring fields, with the ultimate goal of fostering the adoption of good reproducibility practices and easing a more widespread and systematic production of reproducible papers and research in the field.
The survey focuses on reproducibility in IR, discusses the related terminology, challenges and barriers to reproducibility, ways of measuring and quantifying the achieved degree of reproducibility, and various reproducibility initiatives in the field. The authors also discuss and review best practices for achieving and improving reproducibility, concerning how to share data and code, how to document experiments and how to better adhere to open science principles; the authors also present the corresponding tools that can ease the day-by-day operation of such practices. The authors broaden the perspective on reproducibility to neighboring fields, namely, Recommender Systems (RS), Natural Language Processing (NLP) and others. Readers will be provided with a summary of current state of the art, the theoretical background, and actionable recommendations about how to validate the quality of their reimplementations, and how to prepare their experiments for reuse by others.
