Experimentation that matters: A multi-case study on the challenges with A/B testing
Paper i proceeding, 2017

From having been exclusive for companies in the online domain, feature experiments are becoming increasingly important for software-intensive companies also in other domains. Today, companies run experiments, such as e.g. A/B tests, to optimize product performance and to learn about user behaviors, as well as to guide product development and innovation. However, although experimentation with customers has become an effective mechanism to improve products and increase revenue, companies struggle with how to leverage the results of the experiments they run. In this paper, we study the reasons for this and we identify three key challenges that make feature experimentation a difficult task. Our research reveals the following challenges: (1) the impact of experiments doesn’t scale, (2) business KPIs and team level metrics are not aligned and (3) it is unclear if the available solutions are applicable across domains.

Data-driven development

A/B testing

Feature experimentation


Helena Holmström Olsson

Malmö universitet

Jan Bosch

Chalmers, Data- och informationsteknik, Software Engineering

A. Fabijan

Malmö universitet

Lecture Notes in Business Information Processing

1865-1348 (ISSN)

Vol. 304 179-185
978-331969190-9 (ISBN)


Data- och informationsvetenskap





Mer information

Senast uppdaterat