Bayesian causal inference in automotive software engineering and online evaluation
Preprint, 2022

Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating software changes. In the automotive domain, running randomised field experiments is not always desired, possible, or even ethical. In the face of such restrictions, we show how to utilise observational studies in combination with Bayesian causal inference to understand real-world impacts from complex automotive software updates and help development organisations arrive at causal conclusions. In this study, we present three causal inference models in the Bayesian framework and their corresponding cases to address three commonly experienced challenges of software evaluation in the automotive domain. We apply Bayesian propensity score matching for producing balanced control and treatment groups, Bayesian regression discontinuity for identifying covariate dependent treatment assignments, and Bayesian difference-in-differences for causal inference on treatment effect overtime. We demonstrate the potential of causal inference with our industry collaborators with studies conducted on a fleet of vehicles. The cases are presented in details as well as the related the theory of causal assumption to the practice of running observational studies. Finally, we discuss the potential and pitfalls of the Bayesian causal models.

Automotive Software

Causal Inference

Online Experimentation

Software Engineering

Bayesian Statistics

Author

Yuchu Liu

Chalmers, Computer Science and Engineering (Chalmers), Interaction Design and Software Engineering

Volvo Cars

David Issa Mattos

Volvo Cars

Jan Bosch

Chalmers, Computer Science and Engineering (Chalmers), Interaction Design and Software Engineering

Helena Holmström Olsson

Malmö university

Jonn Lantz

Volvo Cars

Subject Categories

Software Engineering

Computer Science

More information

Latest update

12/16/2022