Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada
Fri 31 May 2019 16:40 - 16:50 at Laurier - Code Reviews Chair(s): Alexander Serebrenik

Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience.

This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users’ choices, along with other performance metrics.

Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders.

Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools.

Preprint: https://doi.org/10.5281/zenodo.1404814

Fri 31 May

Displayed time zone: Eastern Time (US & Canada) change

16:00 - 17:20
Code ReviewsPapers / Journal-First Papers / Technical Track at Laurier
Chair(s): Alexander Serebrenik Eindhoven University of Technology
16:00
20m
Talk
Test-Driven Code Review: An Empirical StudyTechnical TrackIndustry Program
Technical Track
Davide Spadini Delft University of Technology, Netherlands, Fabio Palomba University of Zurich, Tobias Baum Leibniz Universität Hannover, Stefan Hanenberg University of Duisburg-Essen, Germany, Magiel Bruntink Software Improvement Group, Alberto Bacchelli University of Zurich
DOI Pre-print Media Attached
16:20
20m
Talk
Why Does Code Review Work for Open Source Software Communities?Technical Track
Technical Track
Adam Alami IT University of Copenhagen, Marisa Leavitt Cohn IT University of Copenhagen, Andrzej Wąsowski IT University of Copenhagen, Denmark
Pre-print
16:40
10m
Talk
Does Reviewer Recommendation Help Developers?Industry ProgramJournal-First
Journal-First Papers
Vladimir Kovalenko TU Delft, Nava Tintarev Delft University of Technology, Evgeny Pasynkov JetBrains GmbH, Christian Bird Microsoft Research, Alberto Bacchelli University of Zurich
DOI Pre-print
16:50
10m
Talk
The Impact of Human Factors on the Participation Decision of Reviewers in Modern Code ReviewJournal-First
Journal-First Papers
Shade Ruangwan Nara Institute of Science and Technology, Patanamon Thongtanunam The University of Melbourne, Akinori Ihara Wakayama University, Kenichi Matsumoto Nara Institute of Science and Technology
DOI Pre-print
17:00
20m
Talk
Discussion Period
Papers