Qually: A Quality Validation Toolbox for Automotive Perception Data Towards Trustworthy AI

Report, 2022

Data-driven techniques such as artificial intelligence (AI) and deep learning are frequently deployed as part of automotive perception systems. Due to their heavy dependency on data, data quality is at the essence. In particular, in an automotive perception system, data is captured by sensors and transformed into different formats depending on where it is in the AI data processing pipeline. Although data at different stages share similar attributes, the impact of their properties at each individual stage differ significantly from one another. Therefore, data quality requirements need to be defined specifically at each stage.

In this project, the objective is to develop an end-to-end quality control toolbox to detect errors and anomalies throughout the entire pipeline. To achieve this objective, we divide the project into three work packages, where the first step is to design a set of data properties and their corresponding requirements as quality specifications for data at each stage. Given these specifications, as a second step, we have developed a toolbox, Qually, to evaluate data quality metrics and detect errors and anomalies throughout the AI pipeline. In the last work package, as a demonstrator, Qually is applied to improve automated annotations. This is implemented in three steps: 1) errors are identified using the quality metrics evaluated by Qually; 2) Qually suggests an automatic correction using ensemble techniques; 3) the corrected annotations are evaluated by Qually to confirm the improvement in quality. The error detection and suggested corrections are manually inspected to statistically validate the outcome of Qually.

As the next step, besides further developing Qually as a software to improve its robustness, capacity, scalability and completeness, we plan to focus on enriching the set of data properties and quality specifications, especially by including technical and business requirements from various automotive stakeholders. We also plan to investigate the possibility and scalability of integrating formal verification techniques for quality control.