Evaluating Train-Test Data Leakage in Automotive Image Datasets

Paper i proceeding, 2026

Reliable evaluation of machine learning (ML)-enabled perception systems for intelligent vehicles critically depends on the integrity of training and test datasets. A major risk arises when near-duplicate or visually similar images appear across subsets, leading to inflated performance estimates. This study systematically quantifies train-test similarity in six widely used automotive datasets - KITTI, ZOD, BDD100k, ONCE, Cirrus, and SODA10M - in their default splits. We employ perceptual hashing (pHash) and deep feature embeddings to measure image-level redundancy. Results show 4,513 pairs of KITTI and 562 of ZOD images were almost identical, corresponding to 25% and 5% of their test images, respectively. The other examined datasets contain only at most 3 pairs (almost 0%) of almost identical images in their existing train-test splits. These findings underscore the need for similarity analysis during dataset preparation, particularly for video-based collections with strong spatio-temporal dependencies. By exposing dataset-specific risks of data leakage in popular datasets, this study contributes practical insights for both dataset curators and ML practitioners. These insights are valuable when using public benchmark datasets in safety-critical domains such as autonomous driving (AD).