Print Email Facebook Twitter Reinforcement Learning Across Timescales Title Reinforcement Learning Across Timescales Author Ravi, Siddharth (TU Delft Mechanical, Maritime and Materials Engineering) Contributor Kober, Jens (mentor) de Bruin, Tim (mentor) van Wingerden, Jan-Willem (graduation committee) Boersma, Sjoerd (graduation committee) Degree granting institution Delft University of Technology Date 2017-08-18 Abstract This project addresses a fundamental problem faced by many reinforcement learning agents. Commonly used reinforcement learning agents can be seen to have deteriorating performances at increasing frequencies, as they are unable to correctly learn the ordering of expected returns for actions that are applied. We call this the disappearing reinforcements problem. Moreover, truly multi-task reinforcement learning is only possible when agents are able to operate across frequencies, as different platforms operate at different frequencies. Most algorithms from control theory working on similar tasks, on the other hand, show improved performances when their operating frequencies are increased. This suggests that addressing disappearing reinforcements should enable reinforcement learning agents to have improved performance and generalization ability across timescales and tasks.In this project, we show that disappearing reinforcements is an effect seen independent of the function approximator used in reinforcement learning, and is instead of a more fundamental nature. We explore both theoretically and empirically the relationship between agents and their performances at increasing frequencies. We show that two specific types of agents from literature address the problem, and we benchmark the agents' performances with novel benchmarking measures inspired from control theory. Finally, we create a novel agent we call the dueling advantage learner, by combining both approaches from the state-of-the-art. We then benchmark the different agents across frequencies and tasks, and our agent is seen to outperform each of the individual approaches on the majority of the tasks. Subject Reinforcement LearningControlSamplingcontrol systemsQ-LearningDeep neural networksMachine Learningartificial intelligenceAutonomouslearning by doing To reference this document use: http://resolver.tudelft.nl/uuid:cea64928-6d5e-4342-8517-546c8fc2e551 Embargo date 2017-08-31 Part of collection Student theses Document type master thesis Rights © 2017 Siddharth Ravi Files PDF MasterThesis_Siddharth_Ravi.pdf 1.38 MB Close viewer /islandora/object/uuid:cea64928-6d5e-4342-8517-546c8fc2e551/datastream/OBJ/view