The Automotive Take on Continuous Experimentation: A Multiple Case Study

Paper in proceeding, 2019

Recently, an increasingly growing number of companies is focusing on achieving self-driving systems towards SAE level 3 and higher. Such systems will have much more complex capabilities than today's advanced driver assistance systems (ADAS) like adaptive cruise control and lane-keeping assistance. While the typical development process especially at established automotive OEMs includes systematic testing of such functionality in simulations, at proving grounds, and in pilot cars, the complexity of the systems themselves next to the sheer infinite amount of situations such cars have to handle safely demands an adjustment of such processes. For growingly complex software systems especially in the Web-application domain, the logical successor for Continuous Integration and Deployment (CI/CD) is known as Continuous Experimentation (CE), where product owners jointly with engineers systematically run A/B experiments on possible new features to get quantifiable data about a feature's adoption from the users. While this methodology is increasingly adopted in software-intensive companies, our study is set out to explore advantages and challenges when applying CE during the development and roll-out of functionalities required for self-driving vehicles. This paper reports about the design and results from a multiple case study that was conducted at four companies including two automotive OEMs with a long history of developing vehicles, a Tier-1 supplier, and a start-up company within the area of automated driving systems. Unanimously, all expect higher quality and fast roll-out cycles to the fleet; as major challenges, however, safety concerns next to organizational structures are mentioned. As such, while CE is an increasingly important instrument for companies that focus on non-safety-critical pure software systems, its implementation for safety-critical domains is perceived challenging; nevertheless, clear benefits are foreseen and hence a systematic transition for both today's organizational structures and system architecture is necessary to preserve a company's competitiveness.