Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies. To benefit from OCEs and accurately learn what delivers value to customers, stringent analysis of every experiment needs to be performed. Experiments are sensitive to missing data, skipped checks, wrong designs, and other ‘hiccups’ in the analysis process. As a result, the analysis of experiments has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Relying on experts, however, is neither scalable nor reliable. To democratize experimentation, experiment analysis should be streamlined and performed by engineers, managers, or others responsible for the development of the product. In this paper, based on synthesized experience of companies that run hundreds or thousands of OCEs per year, we examine how experts analyze experiments. We reveal that most of the experiment analysis happens before experiments are even started, and we summarize our learnings in three checklists that can be used to streamline the experiment analysis process. The value of the checklists is threefold. First, they can increase the accuracy of experiment set-up and decision-making process. Second, checklists can enable non-experienced data-scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy tooling.

Wed 29 May
Times are displayed in time zone: (GMT-04:00) Eastern Time (US & Canada) change

11:00 - 12:30: Controlled Experiments of Production SoftwarePapers / Software Engineering in Practice at St-Denis / Notre-Dame
Chair(s): Yvonne DittrichIT University of Copenhagen, Denmark
11:00 - 11:20
Talk
Software Engineering in Practice
Aleksander FabijanMicrosoft, Pavel DmitrievOutreach.io, Helena Holmström OlssonMalmö University, Jan BoschChalmers University of Technology, Sweden, Lukas VermeerBooking.com, Dylan LewisIntuit
11:20 - 11:40
Talk
Software Engineering in Practice
Tong XiaMicrosoft, Sumit BhardwajMicrosoft, Pavel DmitrievOutreach.io, Aleksander FabijanMicrosoft
11:40 - 12:00
Talk
Software Engineering in Practice
Paul Luo LiMicrosoft, Pavel DmitrievOutreach.io, Huibin Mary HuMicrosoft, Xiaoyu ChaiMicrosoft, Zoran DimovMicrosoft, Brandon PaddockMicrosoft, Ying LiMicrosoft, Alex KirshenbaumMicrosoft, Irina NiculescuMicrosoft, Taj ThoresenMicrosoft
12:00 - 12:20
Talk
Software Engineering in Practice
Ali BasiriNetflix, Lorin HochsteinNetflix, Nora JonesNetflix, Haley TuckerNetflix
Pre-print
12:20 - 12:30
Talk
Papers