Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada
Fri 31 May 2019 16:40 - 16:50 at Laurier - Code Reviews Chair(s): Alexander Serebrenik

Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience.

This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users’ choices, along with other performance metrics.

Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders.

Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools.


Fri 31 May
Times are displayed in time zone: (GMT-04:00) Eastern Time (US & Canada) change

16:00 - 17:20: Papers - Code Reviews at Laurier
Chair(s): Alexander SerebrenikEindhoven University of Technology
icse-2019-Technical-Papers16:00 - 16:20
Davide SpadiniDelft University of Technology, Netherlands, Fabio PalombaUniversity of Zurich, Tobias BaumLeibniz Universität Hannover, Stefan HanenbergUniversity of Duisburg-Essen, Germany, Magiel BruntinkSoftware Improvement Group, Alberto BacchelliUniversity of Zurich
DOI Pre-print Media Attached
icse-2019-Technical-Papers16:20 - 16:40
Adam AlamiIT University of Copenhagen, Marisa Leavitt CohnIT University of Copenhagen, Andrzej WąsowskiIT University of Copenhagen, Denmark
icse-2019-Journal-First-Paper16:40 - 16:50
Vladimir KovalenkoTU Delft, Nava TintarevDelft University of Technology, Evgeny PasynkovJetBrains GmbH, Christian BirdMicrosoft Research, Alberto BacchelliUniversity of Zurich
DOI Pre-print
icse-2019-Journal-First-Paper16:50 - 17:00
Shade RuangwanNara Institute of Science and Technology, Patanamon ThongtanunamThe University of Melbourne, Akinori IharaWakayama University, Kenichi MatsumotoNara Institute of Science and Technology
DOI Pre-print
icse-2019-Paper-Presentations17:00 - 17:20