by Arnaud Legrand (Univ. Grenoble Alpes/CNRS/Inria)
To accelerate the adoption of reproducible research methods, researchers from CNRS and Inria have designed a MOOC targeting PhD students, research scientists and engineers working in any scientific domain.
A Reproducibility Crisis?
As the influence of computer science on society keeps growing, and in particular the recent widespread and enthusiastic use of AI/learning techniques, computer scientists are increasingly concerned with ethical issues. Fairness in recommendation systems, accountability in healthcare decision making process and guarantees in autonomous vehicular systems are only the tip of the iceberg.
The tuning of algorithms with (often heavily biased) real-life data involves lengthy and computationally intensive adjustments of hyper parameters and the classical training/testing procedures are often too costly to be rigorously followed and statistically rigorous. Furthermore, publishing results never requires a full disclosure of all the work, especially when trade secrets or confidential data is at stake. As a consequence many research results that may have worked in a specific restricted context turn out to be very difficult to reproduce by other independent researchers.
Although the root causes may be different, similar difficulties have been under the spotlight in every other scientific domain (in particular biology) and publicised under the terms of “reproducibility crisis”. If data science and artificial intelligence are particularly exposed because of the hopes they inspire, it is computer science (operating systems, architecture, image processing) as a whole which suffers from similar reproducibility issues. It is past time that computer scientists adopt a robust and transparent research/experimental methodology.
Reproducible Research
Reproducible research aims at facilitating the exploitation and reuse of research results by advocating for the use of computerised procedures and laboratory notebooks, for the full disclosure of code, data and provenance as well as for standardised and well-controlled statistical procedures and experimental testbeds. It is often seen as a solution to scientific integrity issues but it is foremost an essential step of modern science that has become increasingly complex and error-prone.
In recent years, many researchers, scientific institutions, funding agencies, and publishers, have started to initiate a change in our research and publication practices. Many conferences have set up artifact evaluation procedures and the ACM has proposed some reproducibility badges. The European Research Council, the National Science Foundation and many national research agencies now require data management plans and open-access publications to promote open science. But no standard practice clearly stands out as yet and researchers are often left unarmed to efficiently answer such requirements.
Reproducible Research Challenges
Although computer programs are often thought of as deterministic systems, our algorithms, our software stack and our digital infrastructures have become so complex and evolve so quickly that even skilled computer scientists fail to fully control them. Several interesting projects stand out to address some of the key challenges of reproducible research:
Explainability and tracability: Computational notebooks have become particularly popular as they are easy to share and allow easy implementation of some form of literate programming which emphasises the narration. Scientific workflows have also become essential to orchestrate complex computations, track provenance and exploit parallel architectures.
Software environment control and reconstruction: Notebooks and workflows only track limited information regarding the software environment they build on. Without rigorously preserving this environment, the notebook will hardly be re-executable on another machine and will very likely either fail at runtime in a few months or, even worse, silently produce a different result. Virtual machines or containerization are often considered as a solution but only projects like Nix or Guix propose a clean solution to the traceability and the reconstructability of the environments.
Software and data preservation: Link rot in academic literature is a well-known issue but solutions emerge. The Software Heritage project addresses this by preserving software commons. The Zenodo data warehouse allows any researcher, regardless of their domain, to upload scientific data and share them in a perennial way.
Accelerating Reproducible Research with a Massive Open Online Course
Carrying out reproducible research is obviously not a one-size-fits-all effort. Mastering all these technologies and integrating them in a daily methodology requires time. To get a majority of researchers started, we have designed with the support of Inria a MOOC entitled “Reproducible Research: Methodological Principles for a Transparent Science” [L1] which targets graduate and PhD students, research scientists and engineers working in any scientific domain. The first edition of this MOOC started in October 2018 and has been followed by more than 1,000 individuals (out of about 3,200 registered individuals) working mostly in computer science and biology. This MOOC consists of four modules that combine videos and quizzes with exercises for acquiring hands-on experience with open source tools and methods (Markdown for taking structured and exploitable notes, GitLab for version control and collaborative working, Computational notebooks for efficiently combining the computation, presentation, and analysis of data). We propose three paths, each of which uses a different notebook technology: (1) Jupyter notebooks and the Python (or R) language, which requires no software installation on students’ computers, (2) RStudio and the R language, and (3) the Org-Mode package of the Emacs editor and the languages Python and R. We also introduce the main challenges of reproducible research (data management, software environment, numerical issues) and present a few alternatives. At the end of this MOOC, students and researchers will have acquired good habits for preparing replicable documents and for sharing the results of their work in a transparent fashion.
Please contact:
Arnaud Legrand
Univ. Grenoble Alpes/CNRS/Inria, France
