Learning When To Drive in Uncertain Scenarios: A deep Q-learning approach
Doctoral thesis, 2024
The main focus of this thesis is tactical decision-making for autonomous driving (AD) through intersections with other road users. Human drivers can navigate diverse environments and situations, even those they have never encountered before. Autonomous vehicles are expected to have similar capabilities. This thesis specifically addresses the challenge of navigating intersections where the intentions of other drivers are unknown, as these intentions can be influenced by factors such as driver mood, attention, right-of-way, and traffic signals.
To tackle the complexity of manually specifying reactions for every possible situation, this thesis adopts a learning-based strategy using reinforcement learning (RL). The problem is formulated as a partially observable Markov decision process (POMDP) to account for the uncertainty of unknown driver intentions. A general decision-making agent, based on the deep Q-learning algorithm, is proposed. The contributions of this thesis include the development and application of this method to various simulated intersection scenarios, demonstrating its adaptability and effectiveness in different environments with minimal modifications. By accounting for the inherent uncertainty in driver behavior, this approach enhances the robustness and reliability of the autonomous driving system.
To tackle the complexity of manually specifying reactions for every possible situation, this thesis adopts a learning-based strategy using reinforcement learning (RL). The problem is formulated as a partially observable Markov decision process (POMDP) to account for the uncertainty of unknown driver intentions. A general decision-making agent, based on the deep Q-learning algorithm, is proposed. The contributions of this thesis include the development and application of this method to various simulated intersection scenarios, demonstrating its adaptability and effectiveness in different environments with minimal modifications. By accounting for the inherent uncertainty in driver behavior, this approach enhances the robustness and reliability of the autonomous driving system.
transfer learning
Autonomous driving
Partially observable Markov decision process
uncertain environments
neural networks
model predictive control
deep Q-learning
reinforcement learning
decision making
Author
Tommy Tram
Chalmers, Electrical Engineering, Systems and control
Included papers
Learning Negotiating Behavior Between Cars in Intersections using Deep Q-Learning
IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC,;Vol. 2018-November(2018)p. 3169-3174
Paper in proceeding
Learning When to Drive in Intersections by Combining Reinforcement Learning and Model Predictive Control
2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019,;(2019)p. 3263-3268
Paper in proceeding
Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersections
IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC,;(2020)
Paper in proceeding
Reinforcement Learning in the Wild with Maximum Likelihood-based Model Transfer
Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS,;Vol. 2024(2024)p. 516-524
Paper in proceeding
T. Tram, M. Bouton, J. Fredriksson, J. Sjöberg, M. J. Kochenderfer Belief State Representations in Reinforcement Learning for Autonomous Driving in Intersections
Manuscript
Popular science description
English
Our world of transportation is rapidly evolving, with autonomous driving (AD) technology set to revolutionize urban mobility by redefining how we navigate cities, enhancing safety, and paving the way for a more efficient, inclusive, and sustainable future. However, navigating intersections safely remains a critical challenge on the path to this future. This thesis explores using Deep Q-learning to train decision-making agents for intersection navigation, focusing on understanding and managing uncertainty in other drivers' intentions.
Deep Q-learning is a Reinforcement Learning (RL) algorithm which learns through trial and error. In this thesis, agents were developed and tested to master intersection strategies within simulated environments. The goal is to equip these agents with the ability to make split-second decisions based on real-time data about vehicle positions and velocities to estimate what other drivers might do next.
Key highlights of the research include the successful integration of RL algorithms with advanced control methods like Model Predictive Control (MPC), addressing uncertainties through sophisticated AI techniques, and leveraging previous models to train new ones. This combined approach significantly enhanced the agents' ability to anticipate and respond to unpredictable driver behaviors, contributing to the advancement of safety and reliability in autonomous driving systems within real-world environments. The findings of this thesis hold promise for future innovations in AI-driven transportation systems, aiming towards safer roads and more efficient traffic management solutions in our increasingly interconnected world.
Deep Q-learning is a Reinforcement Learning (RL) algorithm which learns through trial and error. In this thesis, agents were developed and tested to master intersection strategies within simulated environments. The goal is to equip these agents with the ability to make split-second decisions based on real-time data about vehicle positions and velocities to estimate what other drivers might do next.
Key highlights of the research include the successful integration of RL algorithms with advanced control methods like Model Predictive Control (MPC), addressing uncertainties through sophisticated AI techniques, and leveraging previous models to train new ones. This combined approach significantly enhanced the agents' ability to anticipate and respond to unpredictable driver behaviors, contributing to the advancement of safety and reliability in autonomous driving systems within real-world environments. The findings of this thesis hold promise for future innovations in AI-driven transportation systems, aiming towards safer roads and more efficient traffic management solutions in our increasingly interconnected world.
Research Project(s)
WASP SAS: Structuring data for continuous processing and ML systems
Wallenberg AI, Autonomous Systems and Software Program, 2018-01-01 -- 2023-01-01.
Categorizing
Areas of Advance
Transport
Subject Categories (SSIF 2011)
Control Engineering
Identifiers
ISBN
978-91-8103-089-1
Other
Series
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5547
Publisher
Chalmers
Public defence
2024-09-10 10:00 -- 13:00
HC1, Hörsalsvägen 14
Opponent: Professor Miguel Ángel Sotelo, University of Alcala, Spanien