Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

Software companies are increasingly adopting novel approaches to ensure their products perform correctly, succeed in improving user experience, and increasing revenue. Two approaches that have significantly impacted product development are A/B tests – concurrent experiments with different variations of the same product, and phased rollouts - deployments to smaller audiences (rings) before deploying broadly. Although powerful in isolation, product teams experience most benefits when the two approaches are integrated. Intuitively, combining them may seem trivial. However, in practice and at large scale, this is hard. For example, it requires careful data analysis to correctly handle exposed populations, determining the duration of exposure, and identifying the differences between the populations. All of this is needed to optimize the likelihood of successful deployments, maximize learnings, and minimize potential harm to users of the products. In this paper, based on case study research at Microsoft, we introduce controlled rollout (CRL), which applies controlled experimentation to each ring of a traditional phased rollout. We describe its implementation on several products used by hundreds of millions of users along with the complexities encountered and overcome. In particular, we explain strategies for selecting the length of the rollout period, metrics of focus, and define the pass criterion for each of the rings. Finally, we evaluate the effectiveness of CLR by examining hundreds of controlled rollouts at Microsoft Office. With our work, we hope to help other companies in optimizing their software deployment practices.

Wed 29 May

Displayed time zone: Eastern Time (US & Canada) change

11:00 - 12:30
Controlled Experiments of Production SoftwareSoftware Engineering in Practice / Papers at St-Denis / Notre-Dame
Chair(s): Yvonne Dittrich IT University of Copenhagen, Denmark
11:00
20m
Talk
Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at ScaleSEIPIndustry Program
Software Engineering in Practice
Aleksander Fabijan Microsoft, Pavel Dmitriev Outreach.io, Helena Holmström Olsson Malmö University, Jan Bosch Chalmers University of Technology, Sweden, Lukas Vermeer Booking.com, Dylan Lewis Intuit
11:20
20m
Talk
Safe Velocity: A Practical Guide to Software Deployment at Scale using Controlled RolloutSEIPIndustry Program
Software Engineering in Practice
Tong Xia Microsoft, Sumit Bhardwaj Microsoft, Pavel Dmitriev Outreach.io, Aleksander Fabijan Microsoft
11:40
20m
Talk
Experimentation in the Operating System: The Windows Experimentation PlatformSEIPIndustry Program
Software Engineering in Practice
Paul Luo Li Microsoft, Pavel Dmitriev Outreach.io, Huibin Mary Hu Microsoft, Xiaoyu Chai Microsoft, Zoran Dimov Microsoft, Brandon Paddock Microsoft, Ying Li Microsoft, Alex Kirshenbaum Microsoft, Irina Niculescu Microsoft, Taj Thoresen Microsoft
12:00
20m
Talk
Automating chaos experiments in productionSEIPIndustry Program
Software Engineering in Practice
Ali Basiri Netflix, Lorin Hochstein Netflix, Nora Jones Netflix, Haley Tucker Netflix
Pre-print
12:20
10m
Talk
Discussion Period
Papers