Challenges and Recommendations Using Surrogates for Safety Impact Assessment: Correcting Misestimations

Paper in proceeding, 2025

To assess the safety of automated driving systems (ADS) when severe injuries and fatalities (SI&F) are too rare to observe, surrogate measures like near-crashes are often used. However, when assessing the aggregated safety impact across scenario types (e.g., rear-end, side-swipe, lane-departure, intersection negotiation), this practice may result in substantial misestimation of the system's safety impact on higher-severity crashes. This work first demonstrates how these erroneous safety impact estimates can occur when ADS are assessed through surrogate- and scenario-based testing across multiple scenario types. Second, we show that a 'multiplier' for each scenario type - representing the number of surrogate events (near-crashes) that equate to one target event ((SI&F) - can and should be used to correct the estimates. Specifically, we estimate multipliers for different scenario types and for events of different severity levels in the Second Strategic Highway Research Program (SHRP2) naturalistic driving data. A specific multiplier for each scenario type is called for since each type is typically represented in the data by a different surrogate:target-crash ratio. For example, in the SHRP2 data there are approximately 16 near-crashes for each severe SHRP2 crash in the turn across path scenario, while there are 115 near-crashes for each severe SHRP2 crash in the rear-end striking scenario. Using SHRP2 data along with hypothetical target/surrogate event reduction rates for an ADS, we demonstrate how to use corrective multipliers. We assume the same system reduction rates for both surrogate and target events. However, this assumption may often not be justified, which presents an additional challenge. The avoidance rate may differ for surrogate and target events. There may even be a negative correlation: in the treatment condition, the system might avoid fewer surrogate events (near-crashes) while actually creating more target events (severe crashes), or vice versa. We also demonstrate and discuss this challenge. Finally, we provide guidelines for the appropriate use of surrogate scenarios and other types of surrogate measures, in the context of multiscenario aggregation for safety impact assessment. This research provides insights and practical recommendations that can improve the accuracy and reliability of ADS safety assessments, ultimately contributing to more correct and trustworthy traffic safety assessments.