Online experimentation in automotive software engineering
Licentiate thesis, 2022

Context: Online experimentation has long been the gold standard for evaluating software towards the actual needs and preferences of customers. In the Software-as-a-Service domain, various online experimentation techniques are applied and proven successful. As software is becoming the main differentiator for automotive products, the automotive sector has started to express an interest in adopting online experimentation to strengthen their software development process.

Objective: In this research, we aim to systematically address the challenges in adopting online experimentation in the automotive domain.

Method: We apply a multidisciplinary approach to this research. To understand the state-of-practise in online experimentation in the industry, we conduct case studies with three manufacturers. We introduce our experimental design and evaluation methods to real vehicles driven by customers at scale. Moreover, we run experiments to quantitatively evaluate experiment design and causal inference models.

Results: Four main research outcomes are presented in this thesis. First, we propose an architecture for continuous online experimentation given the limitations experienced in the automotive domain. Second, after identifying an inherent limitation of sample sizes in the automotive domain, we apply and evaluate an experimentation design method. The method allows us to utilise pre-experimental data for generating balanced groups even when sample sizes are limited. Third, we present an alternative approach to randomised experiments and demonstrate the application of Bayesian causal inference in online software evaluation. With the models, we enable software online evaluation without the need for a fully randomised experiment. Finally, we relate the formal assumption in the Bayesian causal models to the implications in practise, and we demonstrate the inference models with cases from the automotive domain.

Outlook: In our future work, we plan to explore causal structural and graphical models applied in software engineering, and demonstrate the application of causal discovery in machine learning-based autonomous drive software.

Automotive software

Bayesian statistics

Online experimentation

Embedded software

Causal inference

Chalmers University of Technology, Campus Lindholmen, Building Kuggen, Room 342
Opponent: Prof. Carlo A. Furia, USI Università della Svizzera italiana, Switzerland


Yuchu Liu

Chalmers, Computer Science and Engineering (Chalmers), Interaction Design and Software Engineering

Volvo Cars

An architecture for enabling A/B experiments in automotive embedded software

Proceedings - 2021 IEEE 45th Annual Computers, Software, and Applications Conference, COMPSAC 2021,; (2021)p. 992-997

Paper in proceeding

Size matters? or not: A/B testing with limited sample in automotive embedded software

Proceedings - 2021 47th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2021,; (2021)p. 300-307

Paper in proceeding

Bayesian propensity score matching in automotive embedded software engineering

Proceedings - Asia-Pacific Software Engineering Conference, APSEC,; Vol. 2021-December(2021)p. 233-242

Paper in proceeding

Areas of Advance


Subject Categories

Software Engineering

Computer Science



Chalmers University of Technology, Campus Lindholmen, Building Kuggen, Room 342


Opponent: Prof. Carlo A. Furia, USI Università della Svizzera italiana, Switzerland

More information

Latest update

6/2/2022 1