REAL Reinforcement Learning

REAL Reinforcement Learning: Planning with adversarial models

Foffano, Daniele (TU Delft Electrical Engineering, Mathematics and Computer Science)

Oliehoek, F.A. (mentor)
He, J. (mentor)
van Gemert, J.C. (graduation committee)

Delft University of Technology

Computer Science | Data Science and Technology

2022-01-14

Model-Based Reinforcement Learning (MBRL) algorithms solve sequential decision-making problems, usually formalised as Markov Decision Processes, using a model of the environment dynamics to compute the optimal policy. When dealing with complex environments, the environment dynamics are frequently approximated with function approximators (such as Neural Netoworks) that are not guaranteed to converge to an optimal solution. As a consequence, the planning process using samples generated by an imperfect model is also not guaranteed to converge to the optimal policy. In fact, the mismatch between source and target dynamics distribution can result in compounding errors, leading to poor algorithm performance during testing. To mitigate this, we combine the Robust Markov Decision Processes (RMDPs) framework and an ensemble of models to take into account the uncertainty in the approximation of the dynamics. With RMDPs, we can study the uncertainty problem as a two-player stochastic game where Player 1 aims to maximize the expected return and Player 2 wants to minimize it. Using an ensemble of models, Player 2 can choose the worst model to carry out the transitions when performing rollout for the policy improvement. We experimentally show that finding a maximin strategy for this game results in a policy robust to model errors leading to better performance when compared to assuming the learned dynamics to be correct.

Reinforcement Learning
Model-Based
Adversarial robustness
pessimism
Robust Decision Making
Markov Decision Processes

http://resolver.tudelft.nl/uuid:e357848d-0ae1-47f1-a5ac-46900fcd1225

Student theses

master thesis

© 2022 Daniele Foffano