On Experimentation in Software-Intensive Systems
Doctoral thesis, 2021

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain.

Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies.

Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations.

Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business.

Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint.

Experimentation process

Experimentation

Automated Experimentation

Optimization

Multi-armed bandits

Embedded Systems

Lindholmen Jupiter 473
Opponent: Prof. Tomas Bures - Charles University, Czech Republic

Author

David Issa Mattos

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers), Software Engineering for Cyber Physical Systems

Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions

IEEE Transactions on Evolutionary Computation,; Vol. In Press(2021)

Journal article

Multi-armed bandits in the wild: Pitfalls and strategies in online experiments

Information and Software Technology,; Vol. 113(2019)p. 68-81

Journal article

An activity and metric model for online controlled experiments

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),; Vol. 11271 LNCS(2018)p. 182-198

Paper in proceeding

Automated optimization of software parameters in a long term evolution radio base station

SysCon 2019 - 13th Annual IEEE International Systems Conference, Proceedings,; (2019)

Paper in proceeding

Your system gets better every day you use it: Towards automated continuous experimentation

43rd Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2017; Vienna; Austria; 30 August 2017 through 1 September 2017,; (2017)p. 256-265

Paper in proceeding

Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives

Lecture Notes in Business Information Processing,; Vol. 314(2018)p. 277-292

Paper in proceeding

Mattos, D. I., Dakkak, A., Bosch, J. and Olsson, H. H. “The HURRIER Process for Experimentation in Business-to-Business Mission-Critical Systems”

Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain.

This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies.

The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint.

Subject Categories

Software Engineering

ISBN

978-91-7905-465-6

Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 4932

Publisher

Chalmers University of Technology

Lindholmen Jupiter 473

Online

Opponent: Prof. Tomas Bures - Charles University, Czech Republic

More information

Latest update

8/9/2021 6