Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at ScaleSEIPIndustry Program
Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies. To benefit from OCEs and accurately learn what delivers value to customers, stringent analysis of every experiment needs to be performed. Experiments are sensitive to missing data, skipped checks, wrong designs, and other ‘hiccups’ in the analysis process. As a result, the analysis of experiments has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Relying on experts, however, is neither scalable nor reliable. To democratize experimentation, experiment analysis should be streamlined and performed by engineers, managers, or others responsible for the development of the product. In this paper, based on synthesized experience of companies that run hundreds or thousands of OCEs per year, we examine how experts analyze experiments. We reveal that most of the experiment analysis happens before experiments are even started, and we summarize our learnings in three checklists that can be used to streamline the experiment analysis process. The value of the checklists is threefold. First, they can increase the accuracy of experiment set-up and decision-making process. Second, checklists can enable non-experienced data-scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy tooling.