Effective online controlled experiment analysis at large scale
Paper i proceeding, 2018

Online Controlled Experiments (OCEs) are the norm in data-driven software companies because of the benefits they provide for building and deploying software. Product teams experiment to accurately learn whether the changes that they do to their products (e.g. adding new features) cause any impact (e.g. customers use them more frequently). Experiments also help reduce the risk from deploying software by minimizing the magnitude and duration of harm caused by software bugs, allowing software to be shipped more frequently. To make informed decisions in product development, experiment analysis needs to be granular with a large number of metrics over heterogeneous devices and audiences. Discovering experiment insights by hand, however, can be cumbersome. In this paper, and based on case study research at a large-scale software development company with a long tradition of experimentation, we (1) describe the standard process of experiment analysis, and (2) introduce an artifact to improve the effectiveness and comprehensiveness of this process.

Guided experiment analysis

A/B testing

Online controlled experiments


A. Fabijan

Malmö universitet

Pavel Dmitriev


Helena Holmström Olsson

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018

64-67 8498187

44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018
Prague, Czech Republic,


Övrig annan teknik





Mer information

Senast uppdaterat