Online controlled experimentation at scale: An empirical survey on the current state of A/B testing
Paper i proceeding, 2018

Online Controlled Experiments (OCEs, aka A/B tests) are one of the most powerful methods for measuring how much value new features and changes deployed to software products bring to users. Companies like Microsoft, Amazon, and Booking.com report the ability to conduct thousands of OCEs every year. However, the competences of the remainder of the online software industry remain unknown. The main objective of this paper is to reveal the current state of A/B testing maturity in the software industry based on a maturity model from our previous research. We base our findings on 44 responses from an online empirical survey. Our main contribution of this paper is the current state of experimentation maturity as operationalized by the ExG model for a convenience sample of companies doing online controlled experiments. Our findings show that, among others, companies typically develop in-house experimentation platforms, that these platforms are of various levels of maturity, and that designing key metrics - Overall Evaluation Criteria - remains the key challenge for successful experimentation.

Empirical survey

A/B testing

Experimentation growth model

Controlled experimentation

Författare

A. Fabijan

Malmö universitet

Pavel Dmitriev

Microsoft

Helena Holmström Olsson

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for Testing, Requirements, Innovation and Psychology

Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018

68-72 8498188

44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018
Prague, Czech Republic,

Ämneskategorier

Produktionsteknik, arbetsvetenskap och ergonomi

Övrig annan teknik

Programvaruteknik

DOI

10.1109/SEAA.2018.00021

Mer information

Senast uppdaterat

2019-01-22