Safe Exploration Algorithms for Reinforcement Learning Controllers

Mannucci, T.; van Kampen, E.; de Visser, C.C.; Chu, Q. P.

doi:10.1109/TNNLS.2017.2654539

Safe Exploration Algorithms for Reinforcement Learning Controllers

Title

Safe Exploration Algorithms for Reinforcement Learning Controllers

Author

Mannucci, T. (TU Delft Control & Simulation)
van Kampen, E. (TU Delft Control & Simulation)
de Visser, C.C. (TU Delft Control & Simulation)
Chu, Q. P. (TU Delft Control & Simulation)

Date

2018-04-01

Abstract

Self-learning approaches, such as reinforcement learning, offer new possibilities for autonomous control of uncertain or time-varying systems. However, exploring an unknown environment under limited prediction capabilities is a challenge for a learning agent. If the environment is dangerous, free exploration can result in physical damage or in an otherwise unacceptable behavior. With respect to existing methods, the main contribution of this paper is the definition of a new approach that does not require global safety functions, nor specific formulations of the dynamics or of the environment, but relies on interval estimation of the dynamics of the agent during the exploration phase, assuming a limited capability of the agent to perceive the presence of incoming fatal states. Two algorithms are presented with this approach. The first is the Safety Handling Exploration with Risk Perception Algorithm (SHERPA), which provides safety by individuating temporary safety functions, called backups. SHERPA is shown in a simulated, simplified quadrotor task, for which dangerous states are avoided. The second algorithm, denominated OptiSHERPA, can safely handle more dynamically complex systems for which SHERPA is not sufficient through the use of safety metrics. An application of OptiSHERPA is simulated on an aircraft altitude control task.

Subject

Adaptive controllers
model-free control
reinforcement learning (RL)
safe exploration

To reference this document use:

http://resolver.tudelft.nl/uuid:a38766ef-e74c-4c91-81c7-a2a13b6ec1a8

DOI

https://doi.org/10.1109/TNNLS.2017.2654539

ISSN

2162-237X

Source

IEEE Transactions on Neural Networks and Learning Systems, 29 (4), 1069-1081

Part of collection

Institutional Repository