Comparing Deep Reinforcement Learning Approaches for Sparse Reward Settings with Discrete State-Action Spaces

Comparing Deep Reinforcement Learning Approaches for Sparse Reward Settings with Discrete State-Action Spaces

Çapanoğlu, Alp (TU Delft Electrical Engineering, Mathematics and Computer Science; TU Delft Software Technology)

Neustroev, G. (mentor)
de Weerdt, M.M. (graduation committee)
Zuñiga Zamalloa, M.A. (graduation committee)

Delft University of Technology

2021-06-30

One of the most challenging types of environments for a Deep Reinforcement Learning agent to learn in are those with sparse reward functions. There exist algorithms that are designed to perform well in settings with sparse rewards, but they are often applied to continuous state-action spaces, since economically relevant problems like robotic control and stock trading fall under this category. This means the continuous version overshadows the discrete state-action version of the sparse reward problem. Furthermore, research that focuses on sparse rewards is lacking in comparisons of algorithms dedicated to performing in this type of setting with other state-of-the-art Deep Reinforcement Learning algorithms. We devise an experimental setup to test a selection of algorithms from three state-of-the-art Deep Reinforcement Learning approaches; Hindsight Experience Replay, Maximum Entropy Reinforcement Learning and Distributional Reinforcement Learning. We show that as the cardinality of the state spaces in sparse reward settings increase, Hindsight Experience Replay approaches are superior in sample efficiency compared to the other two approaches studied.

Deep Reinforcement Learning
Sparse rewards
proximal policy optimization
hindsight experience replay
Q-Learning
Discrete state-space
Discrete action-space

http://resolver.tudelft.nl/uuid:aa87a2ad-ed2b-4e7d-91e8-73fe17879d6d

Student theses

bachelor thesis

© 2021 Alp Çapanoğlu