Online controlled experimentation at scale: An empirical survey on the current state of A/B testing
Paper in proceedings, 2018

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.

Empirical survey

A/B testing

Experimentation growth model

Controlled experimentation


A. Fabijan

Malmö university

Pavel Dmitriev


Helena Holmström Olsson

Malmö university

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Testing, Requirements, Innovation and Psychology

Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018

68-72 8498188

44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018
Prague, Czech Republic,

Subject Categories

Production Engineering, Human Work Science and Ergonomics

Other Engineering and Technologies not elsewhere specified

Software Engineering



More information

Latest update