Blogs (1) >>
ICSE 2019
Sat 25 - Fri 31 May 2019 Montreal, QC, Canada

Software companies are increasingly adopting novel approaches to ensure their products perform correctly, succeed in improving user experience, and increasing revenue. Two approaches that have significantly impacted product development are A/B tests – concurrent experiments with different variations of the same product, and phased rollouts - deployments to smaller audiences (rings) before deploying broadly. Although powerful in isolation, product teams experience most benefits when the two approaches are integrated. Intuitively, combining them may seem trivial. However, in practice and at large scale, this is hard. For example, it requires careful data analysis to correctly handle exposed populations, determining the duration of exposure, and identifying the differences between the populations. All of this is needed to optimize the likelihood of successful deployments, maximize learnings, and minimize potential harm to users of the products. In this paper, based on case study research at Microsoft, we introduce controlled rollout (CRL), which applies controlled experimentation to each ring of a traditional phased rollout. We describe its implementation on several products used by hundreds of millions of users along with the complexities encountered and overcome. In particular, we explain strategies for selecting the length of the rollout period, metrics of focus, and define the pass criterion for each of the rings. Finally, we evaluate the effectiveness of CLR by examining hundreds of controlled rollouts at Microsoft Office. With our work, we hope to help other companies in optimizing their software deployment practices.

Wed 29 May

icse-2019-Paper-Presentations
11:00 - 12:30: Papers - Controlled Experiments of Production Software at St-Denis / Notre-Dame
Chair(s): Yvonne DittrichIT University of Copenhagen, Denmark
icse-2019-Software-Engineering-in-Practice11:00 - 11:20
Talk
Aleksander FabijanMicrosoft, Pavel DmitrievOutreach.io, Helena Holmström OlssonMalmö University, Jan BoschChalmers University of Technology, Sweden, Lukas VermeerBooking.com, Dylan LewisIntuit
icse-2019-Software-Engineering-in-Practice11:20 - 11:40
Talk
Tong XiaMicrosoft, Sumit BhardwajMicrosoft, Pavel DmitrievOutreach.io, Aleksander FabijanMicrosoft
icse-2019-Software-Engineering-in-Practice11:40 - 12:00
Talk
Paul Luo LiMicrosoft, Pavel DmitrievOutreach.io, Huibin Mary HuMicrosoft, Xiaoyu ChaiMicrosoft, Zoran DimovMicrosoft, Brandon PaddockMicrosoft, Ying LiMicrosoft, Alex KirshenbaumMicrosoft, Irina NiculescuMicrosoft, Taj ThoresenMicrosoft
icse-2019-Software-Engineering-in-Practice12:00 - 12:20
Talk
Ali BasiriNetflix, Lorin HochsteinNetflix, Nora JonesNetflix, Haley TuckerNetflix
Pre-print
icse-2019-Paper-Presentations12:20 - 12:30
Talk