Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada
Thu 30 May 2019 11:30 - 11:50 at Laurier - Unit Testing Chair(s): Martin Kropp

Reliable code coverage tools are critically important as it is heavily used to facilitate many quality assurance activities, such as software testing, fuzzing, and debugging. However, little attention has been devoted to assessing the reliability of code coverage tools. In this study, we propose a randomized differential testing approach to hunting for bugs in the most widely used C code coverage tools. Specifically, by generating random input programs, our approach seeks for inconsistencies in code coverage reports produced by different code coverage tools, and then identifies inconsistencies as potential code coverage bugs. To effectively report code coverage bugs, we addressed three specific challenges: (1) How to filter out duplicate test programs as many of them triggering the same bugs in code coverage tools; (2) how to automatically reduce large test programs to much smaller ones that have the same properties; and (3) how to determine which code coverage tools have bugs? The extensive evaluations validate the effectiveness of our approach, resulting in 42 and 28 confirmed/fixed bugs for gcov and llvm-cov, respectively. This case study indicates that code coverage tools are not as reliable as it might have been envisaged. It not only demonstrates the effectiveness of our approach, but also highlights the need to continue improving the reliability of code coverage tools. This work opens up a new direction in code coverage validation which calls for more attention in this area.

Conference Day
Thu 30 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Unit TestingTechnical Track / Software Engineering in Practice / Papers / Journal-First Papers at Laurier
Chair(s): Martin KroppUniversity of Applied Sciences Northwestern Switzerland
(SEIP Talk) Mythical Unit Test CoverageSEIPIndustry Program
Software Engineering in Practice
Vard AntinyanVolvo Car Group, Miroslaw StaronUniversity of Gothenburg
Research paper
Hunting for Bugs in Code Coverage Tools via Randomized Differential TestingTechnical Track
Technical Track
Yibiao YangNanjing University, China, Yuming Zhou, Hao SunUnaffiliated, Zhendong SuETH Zurich, Zhiqiang ZuoNanjing University, China, Lei XuNanjing University, Baowen Xu
Rotten Green TestsTechnical Track
Technical Track
Julien DelplanqueUniversity of Lille, Stéphane Ducasse INRIA Lille, Guillermo PolitoUniv. Lille, CNRS, Centrale Lille, Inria, UMR 9189 - CRIStAL - Centre de Recherche en Informatique Signal et Automatique de Lille, Andrew P. BlackPortland State University and INRIA, Anne EtienEcole Polytechnique Universitaire de Lille
A comprehensive study of pseudo-tested methodsJournal-First
Journal-First Papers
Oscar Luis Vera PérezINRIA, Benjamin DanglotUniversity Lille 1 and INRIA, Martin MonperrusKTH Royal Institute of Technology, Benoit BaudryKTH Royal Institute of Technology, Sweden
Link to publication DOI Pre-print
Discussion Period