We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation
Journal article, 2018

Context: Continuous experimentation guides development activities based on data collected on a subset of online users on a new experimental version of the software. It includes practices such as canary releases, gradual rollouts, dark launches, or A/B testing. Objective: Unfortunately, our knowledge of continuous experimentation is currently primarily based on well-known and outspoken industrial leaders. To assess the actual state of practice in continuous experimentation, we conducted a mixed-method empirical study. Method: In our empirical study consisting of four steps, we interviewed 31 developers or release engineers, and performed a survey that attracted 187 complete responses. We analyzed the resulting data using statistical analysis and open coding. Results: Our results lead to several conclusions: (1) from a software architecture perspective, continuous experimentation is especially enabled by architectures that foster independently deployable services, such as microservices-based architectures; (2) from a developer perspective, experiments require extensive monitoring and analytics to discover runtime problems, consequently leading to developer on call policies and influencing the role and skill sets required by developers; and (3) from a process perspective, many organizations conduct experiments based on intuition rather than clear guidelines and robust statistics. Conclusion: Our findings show that more principled and structured approaches for release decision making are needed, striving for highly automated, systematic, and data- and hypothesis-driven deployment and experimentation.

Empirical study

Continuous experimentation

Release engineering

Continuous deployment


Gerald Schermann

University of Zürich

Jürgen Cito

University of Zürich

Philipp Leitner

University of Zürich

Uwe Zdun

University of Vienna

Harald C. Gall

University of Zürich

Information and Software Technology

0950-5849 (ISSN)

Vol. 99 41-57

Areas of Advance

Information and Communication Technology

Subject Categories

Software Engineering

Information Science

Computer Science



More information

Latest update