An activity and metric model for online controlled experiments
Paper i proceeding, 2018

Accurate prioritization of efforts in product and services development is critical to the success of every company. Online controlled experiments, also known as A/B tests, enable software companies to establish causal relationships between changes in their systems and the movements in the metrics. By experimenting, product development can be directed towards identifying and delivering value. Previous research stresses the need for data-driven development and experimentation. However, the level of granularity in which existing models explain the experimentation process is neither sufficient, in terms of details, nor scalable, in terms of how to increase number and run different types of experiments, in an online setting. Based on a case study of multiple products running online controlled experiments at Microsoft, we provide an experimentation framework composed of two detailed experimentation models focused on two main aspects; the experimentation activities and the experimentation metrics. This work intends to provide guidelines to companies and practitioners on how to set and organize experimentation activities for running trustworthy online controlled experiments.

Online controlled experiments

Data-driven development

A/B tests

Författare

David Issa Mattos

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for People, Architecture, Requirements and Traceability

Pavel Dmitriev

Outreach

A. Fabijan

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Helena Holmström Olsson

Malmö universitet

Lecture Notes in Computer Science

0302-9743 (ISSN)

Vol. 11271 LNCS 182-198

19th International Conference on Product-Focused Software Process Improvement, PROFES 2018
Wolfsburg, Germany,

Ämneskategorier

Övrig annan teknik

Företagsekonomi

Programvaruteknik

DOI

10.1007/978-3-030-03673-7_14

Mer information

Senast uppdaterat

2019-02-13