V-Mon: Scalable and Fault-Tolerant Stream Processing Pipeline for Monitoring Vehicular Data Validity

Paper in proceeding, 2025

With constant increases in edge devices in industry settings, increases in data rates naturally follow. However, with high, unbounded data rates, traditional (store-then-process) database procedures and batch-based processing are struggling to remain performant. To this end, processing streams of data continuously is an increasingly appealing approach, targeting low latency, high scalability and real-time data processing. This work examines design considerations as well as performance trade-offs for a stream processing pipeline targeting stateful analysis. The pipeline implementation employs Apache Kafka, Apache Flink and Apache Druid, and is studied through an example use case at Volvo Trucks, focusing on signal data set validity analysis. Performance evaluation of the pipeline reveals that the throughput requirements of the use case are satisfied, while also achieving sub-second latencies and offering a degree of fault tolerance. The pipeline also shows promise of adapting well to different levels of scale, providing enough headroom for a tenfold increase in data volumes over current demands. Further, the extensible nature of the pipeline enables the support of various feature extraction methods, e.g., data synopsis and sketching, and alternative data representations, e.g., knowledge graphs.