Does Reviewer Recommendation Help Developers?Industry ProgramJournal-First
Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience.
This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users’ choices, along with other performance metrics.
Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders.
Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools.
Preprint: https://doi.org/10.5281/zenodo.1404814
Fri 31 MayDisplayed time zone: Eastern Time (US & Canada) change
16:00 - 17:20 | Code ReviewsPapers / Journal-First Papers / Technical Track at Laurier Chair(s): Alexander Serebrenik Eindhoven University of Technology | ||
16:00 20mTalk | Test-Driven Code Review: An Empirical StudyTechnical TrackIndustry Program Technical Track Davide Spadini Delft University of Technology, Netherlands, Fabio Palomba University of Zurich, Tobias Baum Leibniz Universität Hannover, Stefan Hanenberg University of Duisburg-Essen, Germany, Magiel Bruntink Software Improvement Group, Alberto Bacchelli University of Zurich DOI Pre-print Media Attached | ||
16:20 20mTalk | Why Does Code Review Work for Open Source Software Communities?Technical Track Technical Track Adam Alami IT University of Copenhagen, Marisa Leavitt Cohn IT University of Copenhagen, Andrzej Wąsowski IT University of Copenhagen, Denmark Pre-print | ||
16:40 10mTalk | Does Reviewer Recommendation Help Developers?Industry ProgramJournal-First Journal-First Papers Vladimir Kovalenko TU Delft, Nava Tintarev Delft University of Technology, Evgeny Pasynkov JetBrains GmbH, Christian Bird Microsoft Research, Alberto Bacchelli University of Zurich DOI Pre-print | ||
16:50 10mTalk | The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code ReviewJournal-First Journal-First Papers Shade Ruangwan Nara Institute of Science and Technology, Patanamon Thongtanunam The University of Melbourne, Akinori Ihara Wakayama University, Kenichi Matsumoto Nara Institute of Science and Technology DOI Pre-print | ||
17:00 20mTalk | Discussion Period Papers |