Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale
Paper in proceedings, 2019

Online Controlled Experiments (OCEs) are transforming the decision-making process of data-driven companies into an experimental laboratory. Despite their great power in identifying what customers actually value, experimentation is very sensitive to data loss, skipped checks, wrong designs, and many other 'hiccups' in the analysis process. For this purpose, experiment analysis has traditionally been done by experienced data analysts and scientists that closely monitored experiments throughout their lifecycle. Depending solely on scarce experts, however, is neither scalable nor bulletproof. To democratize experimentation, analysis should be streamlined and meticulously performed by engineers, managers, or others responsible for the development of a product. In this paper, based on synthesized experience of companies that run thousands of OCEs per year, we examined how experts inspect online experiments. We reveal that most of the experiment analysis happens before OCEs are even started, and we summarize the key analysis steps in three checklists. The value of the checklists is threefold. First, they can increase the accuracy of experiment set-up and decision-making process. Second, checklists can enable novice data scientists and software engineers to become more autonomous in setting-up and analyzing experiments. Finally, they can serve as a base to develop trustworthy platforms and tools for OCE set-up and analysis.

Online Controlled Experiments

A/B testing

Experiment Checklists


A. Fabijan

Malmö university

Pavel Dmitriev


Helena Holmström Olsson

Malmö university

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Cyber Physical Systems

Lukas Vermeer

Dylan Lewis

Intuit Inc.

Proceedings - 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2019

Vol. May 2019 1-10 8804459

41st IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP 2019
Montreal, Canada,

Subject Categories

Other Engineering and Technologies not elsewhere specified

Software Engineering

Information Science



More information

Latest update