On Experimentation in Software-Intensive Systems
Doktorsavhandling, 2021

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain.

Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies.

Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations.

Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business.

Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint.

Optimization

Automated Experimentation

Experimentation process

Multi-armed bandits

Embedded Systems

Experimentation

Lindholmen Jupiter 473
Opponent: Prof. Tomas Bures - Charles University, Czech Republic

Författare

David Issa Mattos

Chalmers, Data- och informationsteknik, Software Engineering

Statistical Models for the Analysis of Optimization Algorithms with Benchmark Functions

IEEE Transactions on Evolutionary Computation,; Vol. 25(2021)p. 1163-1177

Artikel i vetenskaplig tidskrift

Multi-armed bandits in the wild: Pitfalls and strategies in online experiments

Information and Software Technology,; Vol. 113(2019)p. 68-81

Artikel i vetenskaplig tidskrift

An activity and metric model for online controlled experiments

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),; Vol. 11271 LNCS(2018)p. 182-198

Paper i proceeding

Automated optimization of software parameters in a long term evolution radio base station

SysCon 2019 - 13th Annual IEEE International Systems Conference, Proceedings,; (2019)

Paper i proceeding

Your system gets better every day you use it: Towards automated continuous experimentation

Proceedings - 43rd Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2017,; (2017)p. 256-265

Paper i proceeding

Challenges and strategies for undertaking continuous experimentation to embedded systems: Industry and research perspectives

Lecture Notes in Business Information Processing,; Vol. 314(2018)p. 277-292

Paper i proceeding

The HURRIER process for experimentation in business-to-business mission-critical systems

Journal of Software: Evolution and Process,; Vol. 35(2023)

Artikel i vetenskaplig tidskrift

Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain.

This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies.

The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint.

Ämneskategorier

Programvaruteknik

Datorsystem

ISBN

978-91-7905-465-6

Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 4932

Utgivare

Chalmers

Lindholmen Jupiter 473

Online

Opponent: Prof. Tomas Bures - Charles University, Czech Republic

Mer information

Senast uppdaterat

2023-11-12