On Statistical Methods for Safety Validation of Automated Vehicles
Doctoral thesis, 2022

Automated vehicles (AVs) are expected to bring safer and more convenient transport in the future. Consequently, before introducing AVs at scale to the general public, the required levels of safety should be shown with evidence. However, statistical evidence generated by brute force testing using safety drivers in real traffic does not scale well. Therefore, more efficient methods are needed to evaluate if an AV exhibits acceptable levels of risk.

This thesis studies the use of two methods to evaluate the AV's safety performance efficiently. Both methods are based on assessing near-collision using threat metrics to estimate the frequency of actual collisions. The first method, called subset simulation, is here used to search the scenario parameter space in a simulation environment to estimate the probability of collision for an AV under development. More specifically, this thesis explores how the choice of threat metric, used to guide the search, affects the precision of the failure rate estimation. The result shows significant differences between the metrics and that some provide precise and accurate estimates.

The second method is based on Extreme Value Theory (EVT), which is used to model the behavior of rare events. In this thesis, near-collision scenarios are identified using threat metrics and then extrapolated to estimate the frequency of actual collisions. The collision frequency estimates from different types of threat metrics are assessed when used with EVT for AV safety validation. Results show that a metric relating to the point where a collision is unavoidable works best and provides credible estimates.

In addition, this thesis proposes how EVT and threat metrics can be used as a proactive safety monitor for AVs deployed in real traffic. The concept is evaluated in a fictive development case and compared to a reactive approach of counting the actual events. It is found that the risk exposure of releasing a non-safe function can be significantly reduced by applying the proposed EVT monitor.

automated vehicles

performance evaluation

extreme value theory

simulation

Automotive

automated driving systems

verification

validation

SB-H5
Opponent: Prof. Dr. Simon Burton, Fraunhofer IKS, Germany

Author

Daniel Åsljung

Chalmers, Electrical Engineering, Systems and control

Using Extreme Value Theory for Vehicle Level Safety Validation and Implications for Autonomous Vehicles

IEEE Transactions on Intelligent Vehicles,; Vol. 2(2017)p. 288-297

Journal article

Validation of Collision Frequency Estimation Using Extreme Value Theory

Proceedings of the IEEE Intelligent Transportation Systems Conference, 2017,; (2018)p. 1857-1862

Paper in proceeding

A probabilistic framework for collision probability estimation and an analysis of the discretization precision

IEEE Intelligent Vehicles Symposium, Proceedings,; Vol. 2019-June(2019)p. 52-57

Paper in proceeding

On Automated Vehicle Collision Risk Estimation using Threat Metrics in Subset Simulation

IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC,; (2021)

Paper in proceeding

D. Åsljung, C. Zandén, J. Fredriksson. A Risk Reducing Fleet Monitor for Automated Vehicles Based on Extreme Value Theory

Every year more than a million lives are cut short due to traffic accidents. However, most traffic accidents are caused by human error and if these causes can be removed, many lives could be saved. Autonomous vehicles (AVs) will never be as tired or distracted as humans are and are expected to lead to a significantly safer traffic environment.

Before AVs can be used by the public and enable a safer future, it needs to be shown that they are as safe as they should be. As a result, we need evidence that AVs have fewer accidents than human drivers in real traffic. This evidence is not simple to obtain since humans are, on average good drivers, and fatalities may occur less often than once every 100 million kilometers. Driving this distance to show the absence of accidents before every release does not scale well.

This thesis presents multiple approaches to creating this evidence more efficiently. The first method uses computer simulations of the actual vehicle to provide safety evidence of the software before it is used in an actual vehicle. Simulation efforts are also focused on the areas where it is believed to be closest to failure, which makes it more efficient. The result is a precise estimate of how often the AV software will fail and the specific scenarios where it will happen.

A second method is evaluating the safety of AVs in real traffic. It evaluates situations that were close to being accidents and uses them to estimate the frequency of actual accidents. The method makes it possible to show that the AVs are safe without experiencing any real accidents. In addition, the second method is also used to form a predictive safety monitor for a fleet of AVs. The results show that the predictive monitor significantly reduces the risk of deploying unsafe AVs.

Tidseffektiv RobUSt verifiering av auTonoma fordon - teori och MEtoder (TRUST-ME)

VINNOVA (2014-01373), 2014-05-01 -- 2017-12-31.

Areas of Advance

Transport

Subject Categories

Electrical Engineering, Electronic Engineering, Information Engineering

Probability Theory and Statistics

ISBN

978-91-7905-757-2

Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5223

Publisher

Chalmers

SB-H5

Opponent: Prof. Dr. Simon Burton, Fraunhofer IKS, Germany

More information

Latest update

11/14/2022