Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies. To benefit from OCEs and accurately learn what delivers value to customers, stringent analysis of every experiment needs to be performed. Experiments are sensitive to missing data, skipped checks, wrong designs, and other ‘hiccups’ in the analysis process. As a result, the analysis of experiments has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Relying on experts, however, is neither scalable nor reliable. To democratize experimentation, experiment analysis should be streamlined and performed by engineers, managers, or others responsible for the development of the product. In this paper, based on synthesized experience of companies that run hundreds or thousands of OCEs per year, we examine how experts analyze experiments. We reveal that most of the experiment analysis happens before experiments are even started, and we summarize our learnings in three checklists that can be used to streamline the experiment analysis process. The value of the checklists is threefold. First, they can increase the accuracy of experiment set-up and decision-making process. Second, checklists can enable non-experienced data-scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy tooling.

Wed 29 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Controlled Experiments of Production SoftwareSoftware Engineering in Practice / Papers at St-Denis / Notre-Dame
Chair(s): Yvonne Dittrich IT University of Copenhagen, Denmark
11:00
20m
Talk
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at ScaleSEIPIndustry Program
Software Engineering in Practice
Aleksander Fabijan Microsoft, Pavel Dmitriev Outreach.io, Helena Holmström Olsson Malmö University, Jan Bosch Chalmers University of Technology, Sweden, Lukas Vermeer Booking.com, Dylan Lewis Intuit
11:20
20m
Talk
Safe Velocity: A Practical Guide to Software Deployment at Scale using Controlled RolloutSEIPIndustry Program
Software Engineering in Practice
Tong Xia Microsoft, Sumit Bhardwaj Microsoft, Pavel Dmitriev Outreach.io, Aleksander Fabijan Microsoft
11:40
20m
Talk
Experimentation in the Operating System: The Windows Experimentation PlatformSEIPIndustry Program
Software Engineering in Practice
Paul Luo Li Microsoft, Pavel Dmitriev Outreach.io, Huibin Mary Hu Microsoft, Xiaoyu Chai Microsoft, Zoran Dimov Microsoft, Brandon Paddock Microsoft, Ying Li Microsoft, Alex Kirshenbaum Microsoft, Irina Niculescu Microsoft, Taj Thoresen Microsoft
12:00
20m
Talk
Automating chaos experiments in productionSEIPIndustry Program
Software Engineering in Practice
Ali Basiri Netflix, Lorin Hochstein Netflix, Nora Jones Netflix, Haley Tucker Netflix
Pre-print
12:20
10m
Talk
Discussion Period
Papers