Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada
Wed 29 May 2019 11:20 - 11:40 at Duluth - Testing Effectiveness Chair(s): Diomidis Spinellis

Tens of thousands engineers are contributing to Google’s codebase that spans billions of lines of code. To ensure high code quality, tremendous amount of effort has been made with new testing techniques and frameworks. However, with increasingly complex data structures and software pipelines, traditional test cases based testing strategies cannot scale well to achieve the desired level of test adequacy. Differential (Diff) testing is one of the new testing techniques adapted to fill this gap. It uses the same input to run two versions of a software, and then compares the output of two runs to find abnormalities that may lead to bugs.

Differential testing has been adopted quickly by hundreds of teams across at Google. Meanwhile, many new diff testing frameworks were developed to simplify the creation, maintenance, and analysis of diff tests. Curious by this emerging popularity, we conducted the first empirical study on differential testing in practice at large scale. In this study, we investigated common practices and benefits of diff testing. We further explore the features of diff tests that users value the most and the pain points of using diff testing. Through a user study, we discovered that differential testing does not replace fine-grained testing techniques, Instead it supplements existing testing suites. It helps users verify the impact on unmodified and unfamiliar components in the absence of a test oracle. In terms of limitations, diff often appears to generate noisy and flaky outcomes. Finally, we highlight problems to guide future research in testing.

Wed 29 May

11:00 - 12:30: Papers - Testing Effectiveness at Duluth
Chair(s): Diomidis SpinellisAthens University of Economics and Business
icse-2019-Software-Engineering-in-Practice11:00 - 11:20
Pavneet Singh KochharMicrosoft, Xin XiaMonash University, David LoSingapore Management University
icse-2019-Software-Engineering-in-Practice11:20 - 11:40
Muhammad Ali GulzarUniversity of California, Los Angeles, Yongkang ZhuGoogle, Xiaofeng HanGoogle
icse-2019-Journal-First-Paper11:40 - 11:50
Xintao Niu, Changhai Nie, Hareton Leung, Yu Lei, Xiaoyin WangUniversity of Texas at San Antonio, USA, Jiaxi XuSchool of Information Engineering, Nanjing Xiaozhuang University, Yan Wang
icse-2019-Journal-First-Paper11:50 - 12:00
Huayao WuNanjing University, Changhai Nie, Justyna PetkeUniversity College London, Yue JiaUniversity College London, Mark HarmanFacebook and University College London
icse-2019-New-Ideas-and-Emerging-Reults12:00 - 12:10
Marcel BöhmeMonash University
icse-2019-Journal-First-Paper12:10 - 12:20
Benjamin DanglotUniversity Lille 1 and INRIA, Oscar Luis Vera PérezINRIA, Benoit BaudryKTH Royal Institute of Technology, Sweden, Martin MonperrusKTH Royal Institute of Technology
icse-2019-Paper-Presentations12:20 - 12:30