On Reinforcement Learning and Digital Twins for Intelligent Automation
Doctoral thesis, 2023
Current trends, such as the fourth industrial revolution and sustainable manufacturing, enable and necessitate manufacturing automation to become more intelligent to meet ever new design requirements in terms of flexibility, speed, quality, and cost.
Two distinct research streams towards intelligent manufacturing exist in the scientific literature: the model-based digital twin approach and the data-driven learning approach. Research that incorporates advantages of the one into the other approach is frequently called for.
Accordingly, this thesis investigates how machine learning can be used to mitigate the model-system mismatch in digital twins and how prior model-based knowledge can be introduced in reinforcement learning in the context of intelligent automation.
In terms of mitigating mismatches in digital twins, research presented in this thesis suggests that learning is of limited usefulness when employed naively in static and systemic mismatch scenarios. In such settings, blackbox optimization algorithms, that leverage properties of the problem, are more useful in terms of sample-efficiency, performance within a given budget, and regret (i.e. when compared to an optimal controller). Learning seems to be of some merit, however, in individualized production control and when used for adapting parameters within a digital twin.
An additional research outcome presented in this thesis is a principled method for incorporating prior knowledge in form of automata specifications into reinforcement learning. Furthermore, the benefits of introducing rich prior model-based knowledge in form of economic non-linear model predictive controllers as model class for function approximation in reinforcement learning is demonstrated in the context of energy optimization.
Lastly, this thesis highlights that adaptive economic non-linear model predictive control may be understood as a unifying framework for both research streams towards intelligent automation.
Two distinct research streams towards intelligent manufacturing exist in the scientific literature: the model-based digital twin approach and the data-driven learning approach. Research that incorporates advantages of the one into the other approach is frequently called for.
Accordingly, this thesis investigates how machine learning can be used to mitigate the model-system mismatch in digital twins and how prior model-based knowledge can be introduced in reinforcement learning in the context of intelligent automation.
In terms of mitigating mismatches in digital twins, research presented in this thesis suggests that learning is of limited usefulness when employed naively in static and systemic mismatch scenarios. In such settings, blackbox optimization algorithms, that leverage properties of the problem, are more useful in terms of sample-efficiency, performance within a given budget, and regret (i.e. when compared to an optimal controller). Learning seems to be of some merit, however, in individualized production control and when used for adapting parameters within a digital twin.
An additional research outcome presented in this thesis is a principled method for incorporating prior knowledge in form of automata specifications into reinforcement learning. Furthermore, the benefits of introducing rich prior model-based knowledge in form of economic non-linear model predictive controllers as model class for function approximation in reinforcement learning is demonstrated in the context of energy optimization.
Lastly, this thesis highlights that adaptive economic non-linear model predictive control may be understood as a unifying framework for both research streams towards intelligent automation.
Digital Twins
Intelligent Automation
Reinforcement Learning
manufacturing
Campus Johanneberg: HC1 (chalmers.se)
Opponent: Professor Dimos Dimarogonas, Division of Decision and Control Systems, KTH, Sweden
Author
Constantin Cronrath
Chalmers, Electrical Engineering, Systems and control
How Useful is Learning in Mitigating Mismatch Between Digital Twins and Physical Systems?
IEEE Transactions on Automation Science and Engineering,;Vol. 21(2024)p. 758-770
Journal article
Enhancing digital twins through reinforcement learning
IEEE International Conference on Automation Science and Engineering,;Vol. 2019-August(2019)p. 293-298
Paper in proceeding
Online geometry assurance in individualized production by feedback control and model calibration of digital twins
Journal of Manufacturing Systems,;Vol. 66(2023)p. 71-81
Journal article
Relevant Safety Falsification by Automata Constrained Reinforcement Learning
IEEE International Conference on Automation Science and Engineering,;Vol. 2022-August(2022)p. 2273-2280
Paper in proceeding
Hovgard, M., Cronrath, C., Bengtsson, K., Lennartson, B. Adaptive Energy Optimization of Flexible Robot Stations
Manufacturing systems must become more flexible, collaborative, and--most importantly--more intelligent to meet the requirements of today's markets and societies. To that end, we observe two distinct trends in the broader research community: the model-based digital twin approach, and the data-driven learning approach to the intelligent automation of manufacturing systems. While both approaches have their merit, it is argued in this thesis, that neither approach will succeed on their own. A combination of both approaches, however, may prove successful.
In that light, this thesis reviews the related work in both streams of research. It is shown that the digital twin approach, which aims to mirror a physical system in the virtual space at all times, translates well to model-based optimal control. The high-fidelity models of the digital twin are, however, mostly computationally expensive and may suffer from mismatches with the physical system. The model-free approach in form of reinforcement learning, requires--as described in this thesis--often large quantities of data, that are expensive to sample from the physical system. Methods, that leverage the advantages of both approaches, are thus investigated in various settings.
In this thesis, it is shown how reinforcement learning, and other blackbox optimizers, can be used to directly adapt control inputs derived by the digital twin to mitigate for the model-system mismatch in the context of assembling sheet metal parts. In the context of energy optimization and safety falsification of collaborative robotics, it is furthermore demonstrated how prior knowledge can be introduced into the learning procedure. Our investigations also suggest that a recent adaptive model-predictive control method may represent a promising combination of both model- and learning-based approaches towards intelligent automation.
In that light, this thesis reviews the related work in both streams of research. It is shown that the digital twin approach, which aims to mirror a physical system in the virtual space at all times, translates well to model-based optimal control. The high-fidelity models of the digital twin are, however, mostly computationally expensive and may suffer from mismatches with the physical system. The model-free approach in form of reinforcement learning, requires--as described in this thesis--often large quantities of data, that are expensive to sample from the physical system. Methods, that leverage the advantages of both approaches, are thus investigated in various settings.
In this thesis, it is shown how reinforcement learning, and other blackbox optimizers, can be used to directly adapt control inputs derived by the digital twin to mitigate for the model-system mismatch in the context of assembling sheet metal parts. In the context of energy optimization and safety falsification of collaborative robotics, it is furthermore demonstrated how prior knowledge can be introduced into the learning procedure. Our investigations also suggest that a recent adaptive model-predictive control method may represent a promising combination of both model- and learning-based approaches towards intelligent automation.
EUREKA ITEA3 AIToC
VINNOVA (2020-01947), 2020-10-01 -- 2023-09-30.
Smart Assembly 4.0
Swedish Foundation for Strategic Research (SSF) (RIT15-0025), 2016-05-01 -- 2021-06-30.
Subject Categories
Production Engineering, Human Work Science and Ergonomics
Control Engineering
Areas of Advance
Production
ISBN
978-91-7905-838-8
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5304
Publisher
Chalmers
Campus Johanneberg: HC1 (chalmers.se)
Opponent: Professor Dimos Dimarogonas, Division of Decision and Control Systems, KTH, Sweden