Bayesian causal inference in automotive software engineering and online evaluation

Preprint, 2022

Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating software changes. In the automotive domain, running randomised field experiments is not always desired, possible, or even ethical. In the face of such restrictions, we show how to utilise observational studies in combination with Bayesian causal inference to understand real-world impacts from complex automotive software updates and help development organisations arrive at causal conclusions. In this study, we present three causal inference models in the Bayesian framework and their corresponding cases to address three commonly experienced challenges of software evaluation in the automotive domain. We apply Bayesian propensity score matching for producing balanced control and treatment groups, Bayesian regression discontinuity for identifying covariate dependent treatment assignments, and Bayesian difference-in-differences for causal inference on treatment effect overtime. We demonstrate the potential of causal inference with our industry collaborators with studies conducted on a fleet of vehicles. The cases are presented in details as well as the related the theory of causal assumption to the practice of running observational studies. Finally, we discuss the potential and pitfalls of the Bayesian causal models.