Towards Real-World Federated Learning: Empirical Studies in the Domain of Embedded Systems

Licentiate thesis, 2022

Context: Artificial intelligence (AI) has led a new phase of technical revolution and industrial development around the world since the twenty-first century, revolutionizing the way of production. Artificial intelligence (AI), an emerging information technology, is thriving, and AI application technologies are gaining traction, particularly in professional services such as healthcare, education, finance, security, etc. More machine learning technologies have begun to be thoroughly applied to the production stage as big data and cloud computing capabilities have improved. With the increased focus on Machine Learning applications and the rapid growth of distributed edge devices in the industry, we believe that utilizing a large number of edge devices will become increasingly important.
The introduction of Federated Learning changes the situation in which data must be centrally uploaded to the cloud for processing and maximizes the use of edge devices' computing and storage capabilities. With local data processing, the learning approach eliminates the need to upload large amounts of local data and reduces data transfer latency. Because Federated Learning does not require centralized data for model training, it is better suited to edge learning scenarios with limited data and privacy concerns.
Objective: The purpose of this research is to identify the characteristics and problems of the Federated Learning methods, our new algorithms and frameworks that can assist companies in making the transition to Federated Learning, and empirically validate the proposed approaches.
Method: To achieve these objectives, we adopted an empirical research approach with design science being our primary research method. We conducted a literature review, case studies, including semi-structured interviews and simulation experiments in close collaboration with software-intensive companies in the embedded systems domain.
Results: We present four major findings in this paper. First, we present a state-of-the-art review of the empirical results reported in the existing Federated Learning literature. We then categorize those Federated Learning implementations into different application domains, identify their challenges, and propose six open research questions based on the problems identified in the literature. Second, we conduct a case study to explain why companies anticipate Federated Learning as a potential solution to the challenges they encountered when implementing machine learning components. We summarize the services that a comprehensive Federated Learning system must enable in industrial settings. Furthermore, we identify the primary barriers that companies must overcome in order to embrace and transition to Federated Learning. Based on our empirical findings, we propose five requirements for companies implementing reliable Federated Learning systems. Third, we develop and evaluate four architecture alternatives for a Federated Learning system, including centralized, hierarchical, regional, and decentralized architectures. We investigate the trade-o between communication latency, model evolution time, and model classification performance, which is critical for applying our findings to real-world industrial systems. Fourth, we introduce techniques and asynchronous frameworks for end-to-end on-device Federated Learning. The method is validated using a steering wheel angle prediction case. The local models of each edge vehicle can be continuously trained and shared with other vehicles to improve their local model prediction accuracy. Furthermore, we combine the asynchronous Federated Learning approach with Deep Neural Decision Forests and validate our method using important industry use cases in the automotive domain. Our findings show that Federated Learning can improve model training speed while lowering communication overhead without sacrificing accuracy, demonstrating that this technique has significant benefits to a wide range of real-world embedded systems.
Future Work: In the future, we plan to test our approach in other use cases and look into more sophisticated neural networks integrated with our approach. In order to improve model training performance on resource-constrained edge devices in real-world embedded systems, we intend to design more appropriate aggregation methods and protocols. Furthermore, we intend to use the Federated Learning and Reinforcement Learning methods to assist the edge in evolving themselves autonomously and fully utilizing the computation capabilities of the edge devices.