We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation
Artikel i vetenskaplig tidskrift, 2018

Context: Continuous experimentation guides development activities based on data collected on a subset of online users on a new experimental version of the software. It includes practices such as canary releases, gradual rollouts, dark launches, or A/B testing. Objective: Unfortunately, our knowledge of continuous experimentation is currently primarily based on well-known and outspoken industrial leaders. To assess the actual state of practice in continuous experimentation, we conducted a mixed-method empirical study. Method: In our empirical study consisting of four steps, we interviewed 31 developers or release engineers, and performed a survey that attracted 187 complete responses. We analyzed the resulting data using statistical analysis and open coding. Results: Our results lead to several conclusions: (1) from a software architecture perspective, continuous experimentation is especially enabled by architectures that foster independently deployable services, such as microservices-based architectures; (2) from a developer perspective, experiments require extensive monitoring and analytics to discover runtime problems, consequently leading to developer on call policies and influencing the role and skill sets required by developers; and (3) from a process perspective, many organizations conduct experiments based on intuition rather than clear guidelines and robust statistics. Conclusion: Our findings show that more principled and structured approaches for release decision making are needed, striving for highly automated, systematic, and data- and hypothesis-driven deployment and experimentation.

Empirical study

Continuous experimentation

Release engineering

Continuous deployment


Gerald Schermann

Universität Zürich

Jürgen Cito

Universität Zürich

Philipp Leitner

Universität Zürich

Uwe Zdun

Universität Wien

Harald C. Gall

Universität Zürich

Information and Software Technology

0950-5849 (ISSN)

Vol. 99 41-57


Informations- och kommunikationsteknik




Datavetenskap (datalogi)



Mer information

Senast uppdaterat