Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies. To benefit from OCEs and accurately learn what delivers value to customers, stringent analysis of every experiment needs to be performed. Experiments are sensitive to missing data, skipped checks, wrong designs, and other ‘hiccups’ in the analysis process. As a result, the analysis of experiments has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Relying on experts, however, is neither scalable nor reliable. To democratize experimentation, experiment analysis should be streamlined and performed by engineers, managers, or others responsible for the development of the product. In this paper, based on synthesized experience of companies that run hundreds or thousands of OCEs per year, we examine how experts analyze experiments. We reveal that most of the experiment analysis happens before experiments are even started, and we summarize our learnings in three checklists that can be used to streamline the experiment analysis process. The value of the checklists is threefold. First, they can increase the accuracy of experiment set-up and decision-making process. Second, checklists can enable non-experienced data-scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy tooling.

Wed 29 May
Times are displayed in time zone: (GMT-04:00) Eastern Time (US & Canada) change

11:00 - 12:30: Papers - Controlled Experiments of Production Software at St-Denis / Notre-Dame
Chair(s): Yvonne DittrichIT University of Copenhagen, Denmark
icse-2019-Software-Engineering-in-Practice11:00 - 11:20
Aleksander FabijanMicrosoft, Pavel, Helena Holmström OlssonMalmö University, Jan BoschChalmers University of Technology, Sweden, Lukas, Dylan LewisIntuit
icse-2019-Software-Engineering-in-Practice11:20 - 11:40
Tong XiaMicrosoft, Sumit BhardwajMicrosoft, Pavel, Aleksander FabijanMicrosoft
icse-2019-Software-Engineering-in-Practice11:40 - 12:00
Paul Luo LiMicrosoft, Pavel, Huibin Mary HuMicrosoft, Xiaoyu ChaiMicrosoft, Zoran DimovMicrosoft, Brandon PaddockMicrosoft, Ying LiMicrosoft, Alex KirshenbaumMicrosoft, Irina NiculescuMicrosoft, Taj ThoresenMicrosoft
icse-2019-Software-Engineering-in-Practice12:00 - 12:20
Ali BasiriNetflix, Lorin HochsteinNetflix, Nora JonesNetflix, Haley TuckerNetflix
icse-2019-Paper-Presentations12:20 - 12:30