Experimentation that matters: A multi-case study on the challenges with A/B testing
Paper in proceeding, 2017

From having been exclusive for companies in the online domain, feature experiments are becoming increasingly important for software-intensive companies also in other domains. Today, companies run experiments, such as e.g. A/B tests, to optimize product performance and to learn about user behaviors, as well as to guide product development and innovation. However, although experimentation with customers has become an effective mechanism to improve products and increase revenue, companies struggle with how to leverage the results of the experiments they run. In this paper, we study the reasons for this and we identify three key challenges that make feature experimentation a difficult task. Our research reveals the following challenges: (1) the impact of experiments doesn’t scale, (2) business KPIs and team level metrics are not aligned and (3) it is unclear if the available solutions are applicable across domains.

Data-driven development

A/B testing

Feature experimentation

Author

Helena Holmström Olsson

Malmö university

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

A. Fabijan

Malmö university

Lecture Notes in Business Information Processing

1865-1348 (ISSN) 18651356 (eISSN)

Vol. 304 179-185
978-331969190-9 (ISBN)

Subject Categories

Computer and Information Science

DOI

10.1007/978-3-319-69191-6_12

ISBN

978-331969190-9

More information

Latest update

2/22/2018