Know what it does not know

Know what it does not know: Improving Offline Deep Reinforcement Learning with Uncertainty Estimation

Smit, Jordi (TU Delft Electrical Engineering, Mathematics and Computer Science)

Oliehoek, F.A. (mentor)
Ponnambalam, C.T. (mentor)
van Gemert, J.C. (graduation committee)

Delft University of Technology

Computer Science | Data Science and Technology

2021-08-31

Offline reinforcement learning, or learning from a fixed data set, is an attractive alternative to online reinforcement learning. Offline reinforcement learning promises to address the cost and safety implications of taking numerous random or bad actions online, which is a crucial aspect of traditional reinforcement learning that makes it difficult to apply in real-world problems. However, when offline reinforcement learning is naïvely applied to a fixed data set, the resulting policy may exhibit poor performance in the real environment. This happens due to over-estimations of the expected return for state-action pairs not sufficiently covered in the data set. Therefore, offline reinforcement learning agents must know what they do not know, allowing them to avoid these over-estimated state-action pairs and their potentially erroneous outcomes. A promising way to instill offline reinforcement learning agents with this ability is the pessimism principle, which states that agents should select actions that maximize an uncertainty-based lower bound of the expected return. This pessimism principle has drastically improved the performance of offline reinforcement learning methods in the tabular and linear function approximation domain. However, in deep reinforcement learning, uncertainty estimation is highly non-trivial, and the development of effective uncertainty-based pessimistic algorithms remains an open question. That is why in this thesis, we explore various existing deep learning-based uncertainty estimation techniques with the aim to combine them with existing deep reinforcement learning methods to create an uncertainty-aware offline deep reinforcement learning algorithm. This research has resulted in two novel offline deep reinforcement learning methods built on Double Deep Q-Learning and Soft Actor-Critic. We applied these methods to various benchmarks and experiments to demonstrate their interesting and unique properties. In some situations, they even beat the current state-of-the-art results of these benchmarks.

Deep Reinforcement Learning
Offline Reinforcement Learning
Uncertainty Estimation
Reinforcement Learning
Deep Uncertainty Estimation
batch reinforcement learning
ensemble
pessimism
uncertainty

http://resolver.tudelft.nl/uuid:b1fcafbe-172a-40f7-8c2a-39b75986d992

https://github.com/j0rd1smit/PEBL An open source version of the code used in the thesis.

Student theses

master thesis

© 2021 Jordi Smit