P-RECS'21

Speaker: Tanu Malik

Title: Artifact Description/Artifact Evaluation: A Reproducibility Bane or a Boon

Abstract: Several systems research conferences now incorporate an artifact description and artifact evaluation (AD/AE) process as part of the paper submission. Authors of accepted papers optionally submit a plethora of artifacts: documentation, links, tools, code, data, and scripts for independent validation of the claims in their paper. An artifact evaluation committee (AEC) evaluates the artifacts and stamps papers with accepted artifacts, which then receive publisher badges. Does this AD/AE process serve authors and reviewers? Is it scalable for large conferences such as SCxy? Using the last three SCxy Reproducibility Initiatives as the basis, this talk will analyze the benefits and the miseries of the AD/AE process.

We will present a data-driven approach, using survey results to analyze technical and human challenges in conducting the AD/AE process. Our method will distinguish studies that benefit from AD, i.e., increased transparency versus areas that benefit from AE. The AD/AE research objects [1] present an interesting set of data management and systems challenges [2,3]. We will look under the hood of the research objects, describe prominent characteristics, and how cloud infrastructures, documented workflows, and reproducible containers [4] ease some of the AD/AE process hand-shakes. Finally, we will present a vision for the resulting curated, reusable research objects—how such research objects are a treasure in themselves for advancing computational reproducibility and making reproducible evaluation practical in the coming years.

Bio: Tanu Malik is an Associate Professor in the School of Computing at DePaul University, Chicago. Her research work concerns developing formal methods, tools, systems and cyber- infrastructure for data provenance and computational reproducibility. She leads and develops the Sciunit project [4], which is improving conduct of reproducible science in computational and data science disciplines. Tanu received the 2019 NSF CAREER award for her work on computational reproducibility. She was also the 2019 Better Scientific Software (BSSw) fellow. Tanu received her PhD in Computer Science from the Johns Hopkins University and was previously a Fellow at The University of Chicago. More about her at https://facsrv.cs.depaul.edu/~tmalik1/.

SPONSOR