The benefits of controlled experimentation at scale
Paper in proceedings, 2017

Online controlled experiments (for example A/B tests) are increasingly being performed to guide product development and accelerate innovation in online software product companies. The benefits of controlled experiments have been shown in many cases with incremental product improvement as the objective. In this paper, we demonstrate that the value of controlled experimentation at scale extends beyond this recognized scenario. Based on an exhaustive and collaborative case study in a large software-intensive company with highly developed experimentation culture, we inductively derive the benefits of controlled experimentation. The contribution of our paper is twofold. First, we present a comprehensive list of benefits and illustrate our findings with five case examples of controlled experiments conducted at Microsoft. Second, we provide guidance on how to achieve each of the benefits. With our work, we aim to provide practitioners in the online domain with knowledge on how to use controlled experimentation to maximize the benefits on the portfolio, product and team level.

Lagging metrics

Controlled experimentation

Leading metrics

Data-driven development

Author

A. Fabijan

Malmö university

P. Dmitriev

Microsoft

Helena Holmström Olsson

Malmö university

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

43rd Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2017; Vienna; Austria; 30 August 2017 through 1 September 2017

18-26 8051322

Subject Categories

Computer and Information Science

DOI

10.1109/SEAA.2017.47

ISBN

978-153862140-0

More information

Latest update

9/6/2018 1