The benefits of controlled experimentation at scale
Paper i proceeding, 2017

Online controlled experiments (for example A/B tests) are increasingly being performed to guide product development and accelerate innovation in online software product companies. The benefits of controlled experiments have been shown in many cases with incremental product improvement as the objective. In this paper, we demonstrate that the value of controlled experimentation at scale extends beyond this recognized scenario. Based on an exhaustive and collaborative case study in a large software-intensive company with highly developed experimentation culture, we inductively derive the benefits of controlled experimentation. The contribution of our paper is twofold. First, we present a comprehensive list of benefits and illustrate our findings with five case examples of controlled experiments conducted at Microsoft. Second, we provide guidance on how to achieve each of the benefits. With our work, we aim to provide practitioners in the online domain with knowledge on how to use controlled experimentation to maximize the benefits on the portfolio, product and team level.

Controlled experimentation

Lagging metrics

Leading metrics

Data-driven development

Författare

A. Fabijan

Malmö universitet

P. Dmitriev

Microsoft Corporation

Helena Holmström Olsson

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering

43rd Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2017; Vienna; Austria; 30 August 2017 through 1 September 2017

18-26 8051322

Ämneskategorier

Data- och informationsvetenskap

DOI

10.1109/SEAA.2017.47

ISBN

978-153862140-0