On Reinforcement Learning and Digital Twins for Intelligent Automation
Doktorsavhandling, 2023
Current trends, such as the fourth industrial revolution and sustainable manufacturing, enable and necessitate manufacturing automation to become more intelligent to meet ever new design requirements in terms of flexibility, speed, quality, and cost.
Two distinct research streams towards intelligent manufacturing exist in the scientific literature: the model-based digital twin approach and the data-driven learning approach. Research that incorporates advantages of the one into the other approach is frequently called for.
Accordingly, this thesis investigates how machine learning can be used to mitigate the model-system mismatch in digital twins and how prior model-based knowledge can be introduced in reinforcement learning in the context of intelligent automation.
In terms of mitigating mismatches in digital twins, research presented in this thesis suggests that learning is of limited usefulness when employed naively in static and systemic mismatch scenarios. In such settings, blackbox optimization algorithms, that leverage properties of the problem, are more useful in terms of sample-efficiency, performance within a given budget, and regret (i.e. when compared to an optimal controller). Learning seems to be of some merit, however, in individualized production control and when used for adapting parameters within a digital twin.
An additional research outcome presented in this thesis is a principled method for incorporating prior knowledge in form of automata specifications into reinforcement learning. Furthermore, the benefits of introducing rich prior model-based knowledge in form of economic non-linear model predictive controllers as model class for function approximation in reinforcement learning is demonstrated in the context of energy optimization.
Lastly, this thesis highlights that adaptive economic non-linear model predictive control may be understood as a unifying framework for both research streams towards intelligent automation.
Two distinct research streams towards intelligent manufacturing exist in the scientific literature: the model-based digital twin approach and the data-driven learning approach. Research that incorporates advantages of the one into the other approach is frequently called for.
Accordingly, this thesis investigates how machine learning can be used to mitigate the model-system mismatch in digital twins and how prior model-based knowledge can be introduced in reinforcement learning in the context of intelligent automation.
In terms of mitigating mismatches in digital twins, research presented in this thesis suggests that learning is of limited usefulness when employed naively in static and systemic mismatch scenarios. In such settings, blackbox optimization algorithms, that leverage properties of the problem, are more useful in terms of sample-efficiency, performance within a given budget, and regret (i.e. when compared to an optimal controller). Learning seems to be of some merit, however, in individualized production control and when used for adapting parameters within a digital twin.
An additional research outcome presented in this thesis is a principled method for incorporating prior knowledge in form of automata specifications into reinforcement learning. Furthermore, the benefits of introducing rich prior model-based knowledge in form of economic non-linear model predictive controllers as model class for function approximation in reinforcement learning is demonstrated in the context of energy optimization.
Lastly, this thesis highlights that adaptive economic non-linear model predictive control may be understood as a unifying framework for both research streams towards intelligent automation.
Digital Twins
Intelligent Automation
Reinforcement Learning
manufacturing
Campus Johanneberg: HC1 (chalmers.se)
Opponent: Professor Dimos Dimarogonas, Division of Decision and Control Systems, KTH, Sweden
Författare
Constantin Cronrath
Chalmers, Elektroteknik, System- och reglerteknik
How Useful is Learning in Mitigating Mismatch Between Digital Twins and Physical Systems?
IEEE Transactions on Automation Science and Engineering,;Vol. 21(2024)p. 758-770
Artikel i vetenskaplig tidskrift
Enhancing digital twins through reinforcement learning
IEEE International Conference on Automation Science and Engineering,;Vol. 2019-August(2019)p. 293-298
Paper i proceeding
Online geometry assurance in individualized production by feedback control and model calibration of digital twins
Journal of Manufacturing Systems,;Vol. 66(2023)p. 71-81
Artikel i vetenskaplig tidskrift
Relevant Safety Falsification by Automata Constrained Reinforcement Learning
IEEE International Conference on Automation Science and Engineering,;Vol. 2022-August(2022)p. 2273-2280
Paper i proceeding
Hovgard, M., Cronrath, C., Bengtsson, K., Lennartson, B. Adaptive Energy Optimization of Flexible Robot Stations
Manufacturing systems must become more flexible, collaborative, and--most importantly--more intelligent to meet the requirements of today's markets and societies. To that end, we observe two distinct trends in the broader research community: the model-based digital twin approach, and the data-driven learning approach to the intelligent automation of manufacturing systems. While both approaches have their merit, it is argued in this thesis, that neither approach will succeed on their own. A combination of both approaches, however, may prove successful.
In that light, this thesis reviews the related work in both streams of research. It is shown that the digital twin approach, which aims to mirror a physical system in the virtual space at all times, translates well to model-based optimal control. The high-fidelity models of the digital twin are, however, mostly computationally expensive and may suffer from mismatches with the physical system. The model-free approach in form of reinforcement learning, requires--as described in this thesis--often large quantities of data, that are expensive to sample from the physical system. Methods, that leverage the advantages of both approaches, are thus investigated in various settings.
In this thesis, it is shown how reinforcement learning, and other blackbox optimizers, can be used to directly adapt control inputs derived by the digital twin to mitigate for the model-system mismatch in the context of assembling sheet metal parts. In the context of energy optimization and safety falsification of collaborative robotics, it is furthermore demonstrated how prior knowledge can be introduced into the learning procedure. Our investigations also suggest that a recent adaptive model-predictive control method may represent a promising combination of both model- and learning-based approaches towards intelligent automation.
In that light, this thesis reviews the related work in both streams of research. It is shown that the digital twin approach, which aims to mirror a physical system in the virtual space at all times, translates well to model-based optimal control. The high-fidelity models of the digital twin are, however, mostly computationally expensive and may suffer from mismatches with the physical system. The model-free approach in form of reinforcement learning, requires--as described in this thesis--often large quantities of data, that are expensive to sample from the physical system. Methods, that leverage the advantages of both approaches, are thus investigated in various settings.
In this thesis, it is shown how reinforcement learning, and other blackbox optimizers, can be used to directly adapt control inputs derived by the digital twin to mitigate for the model-system mismatch in the context of assembling sheet metal parts. In the context of energy optimization and safety falsification of collaborative robotics, it is furthermore demonstrated how prior knowledge can be introduced into the learning procedure. Our investigations also suggest that a recent adaptive model-predictive control method may represent a promising combination of both model- and learning-based approaches towards intelligent automation.
EUREKA ITEA3 AIToC
VINNOVA (2020-01947), 2020-10-01 -- 2023-09-30.
Smart Assembly 4.0
Stiftelsen för Strategisk forskning (SSF) (RIT15-0025), 2016-05-01 -- 2021-06-30.
Ämneskategorier
Produktionsteknik, arbetsvetenskap och ergonomi
Reglerteknik
Styrkeområden
Produktion
ISBN
978-91-7905-838-8
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5304
Utgivare
Chalmers
Campus Johanneberg: HC1 (chalmers.se)
Opponent: Professor Dimos Dimarogonas, Division of Decision and Control Systems, KTH, Sweden