Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
Preprint, 2021

This paper presents a constrained policy gradient algorithm. We introduce constraints for safe learning with the following steps. First, learning is slowed down (lazy learning) so that the episodic policy change can be computed with the help of the policy gradient theorem and the neural tangent kernel. Then, this enables us the evaluation of the policy at arbitrary states too. In the same spirit, learning can be guided, ensuring safety via augmenting episode batches with states where the desired action probabilities are prescribed. Finally, exogenous discounted sum of future rewards (returns) can be computed at these specific state-action pairs such that the policy network satisfies constraints. Computing the returns is based on solving a system of linear equations (equality constraints) or a constrained quadratic program (inequality constraints). Simulation results suggest that adding constraints (external information) to the learning can improve learning in terms of speed and safety reasonably if constraints are appropriately selected. The efficiency of the constrained learning was demonstrated with a shallow and wide ReLU network in the Cartpole and Lunar Lander OpenAI gym environments. The main novelty of the paper is giving a practical use of the neural tangent kernel in reinforcement learning.

Neural Tangent Kernel

RL

ML

Författare

Balázs Varga

Chalmers, Elektroteknik, System- och reglerteknik, Reglerteknik

Balázs Adam Kulcsár

Chalmers, Elektroteknik, System- och reglerteknik, Reglerteknik

Morteza Haghir Chehreghani

Chalmers, Data- och informationsteknik, Data Science

Real-Time Robust and AdaptIve Learning in ElecTric VEhicles (RITE)

Chalmers AI-forskningscentrum (CHAIR), 2020-01-01 -- 2021-12-31.

Chalmers, 2020-01-01 -- 2021-12-31.

Styrkeområden

Informations- och kommunikationsteknik

Transport

Ämneskategorier

Lärande

Systemvetenskap

Datavetenskap (datalogi)

Relaterade dataset

URI: https://arxiv.org/abs/2107.09139

Mer information

Skapat

2021-08-13