Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada
Fri 31 May 2019 14:00 - 14:20 at Mansfield / Sherbrooke - IDEs Chair(s): Sarah Nadi

Code completion is commonly used by software developers and is integrated into all major IDE’s. Good completion tools can not only save time and effort, but may even help to avoid incorrect API usage. Many proposed completion tools have shown promising results on synthetic benchmarks, but these benchmarks make no claims about the realism of the completions they test. This lack of grounding in real-world data could hinder our scientific understanding of developer needs and of the efficacy of completion models. This paper presents a case study on 15,000 code completions that were applied by 66 real developers, in which we analyze how well the recorded completions are represented in synthetic benchmarks to inform future research in this area. Our results shows substantial differences in the distributions of the completed tokens, showing non-trivial differences with synthetic data such as remarkable prevalence of intra-project API completions. When applied to the recorded completions, our models were also far less accurate than on synthetic data. Furthermore, while most of the recorded completions were applied very fast, the cases that consumed most of the developers’ time were also the ones for which the prediction performance was far lower – an effect that is invisible in synthetic benchmarks. We highlight nine issues through our investigation and offer concrete recommendations to improve the design and evaluation of future completion tools.

Fri 31 May
Times are displayed in time zone: (GMT-04:00) Eastern Time (US & Canada) change

14:00 - 14:20
Technical Track
Vincent HellendoornUniversity of California at Davis, USA, Sebastian ProkschUniversity of Zurich, Harald GallUniversity of Zurich, Alberto BacchelliUniversity of Zurich
DOI Pre-print Media Attached
14:20 - 14:40
Technical Track
Jürgen CitoMIT, Philipp LeitnerChalmers University of Technology & University of Gothenburg, Martin RinardMassachusetts Institute of Technology, Harald GallUniversity of Zurich
14:40 - 15:00
Naoyasu UbayashiKyushu University, Watanabe TakuyaEdirium K.K., Yasutaka KameiKyushu University, Ryosuke SatoKyushu University, Japan
Pre-print Media Attached
15:00 - 15:10
New Ideas and Emerging Results
Gail MurphyUniversity of British Columbia
15:10 - 15:30