Online controlled experimentation at scale: An empirical survey on the current state of A/B testing
Paper i proceeding, 2018

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.

Empirical survey

A/B testing

Experimentation growth model

Controlled experimentation


A. Fabijan

Malmö universitet

Pavel Dmitriev


Helena Holmström Olsson

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering

Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018

68-72 8498188
978-1-5386-7383-6 (ISBN)

44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018
Prague, Czech Republic,


Produktionsteknik, arbetsvetenskap och ergonomi

Övrig annan teknik




Mer information

Senast uppdaterat