An activity and metric model for online controlled experiments
Paper in proceedings, 2018

Accurate prioritization of efforts in product and services development is critical to the success of every company. Online controlled experiments, also known as A/B tests, enable software companies to establish causal relationships between changes in their systems and the movements in the metrics. By experimenting, product development can be directed towards identifying and delivering value. Previous research stresses the need for data-driven development and experimentation. However, the level of granularity in which existing models explain the experimentation process is neither sufficient, in terms of details, nor scalable, in terms of how to increase number and run different types of experiments, in an online setting. Based on a case study of multiple products running online controlled experiments at Microsoft, we provide an experimentation framework composed of two detailed experimentation models focused on two main aspects; the experimentation activities and the experimentation metrics. This work intends to provide guidelines to companies and practitioners on how to set and organize experimentation activities for running trustworthy online controlled experiments.

Online controlled experiments

Data-driven development

A/B tests


David Issa Mattos

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for People, Architecture, Requirements and Traceability

Pavel Dmitriev


A. Fabijan

Malmö university

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Testing, Requirements, Innovation and Psychology

Helena Holmström Olsson

Malmö university

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

03029743 (ISSN) 16113349 (eISSN)

Vol. 11271 LNCS 182-198

19th International Conference on Product-Focused Software Process Improvement, PROFES 2018
Wolfsburg, Germany,

Subject Categories

Other Engineering and Technologies not elsewhere specified

Business Administration

Software Engineering



More information

Latest update