Reinforcement Learning for Tracking Control in Robotics

Reinforcement Learning for Tracking Control in Robotics

Pane, Y.P.

Babuska, R. (mentor)

Mechanical, Maritime and Materials Engineering

Delft Center for Systems and Control

2015-11-02

For a robot manipulator, an accurate reference tracking capability is one of the most important performance factor. This is especially the case for a robot which is deployed for an industrial use such as welding and laser cutting. Recently, an emerging application of 3D printing also attracts researchers to use robotic manipulators in hope of gaining a higher dimension in the printing process as well as a larger workspace. The conventional method for controlling robot manipulator involves a model based feed-forward controller to eliminate the nonlinearities such as gravity and the coriolis term, combined with a feedback controller to compensate the residual error. This method, however, has a drawback in obtaining a good model of the robot. Furthermore, a robot's physical properties are subject to change over time due to, for instance, degraded gearboxes, etc. Such a problem will require the robot to re-perform a system identification which is often unacceptable due to the time it takes and the possibly dangerous movement the robot must execute. Therefore, this class of controller may not be the best option for such condition. This thesis addresses the aforementioned problem from the point of view of model-free, learning controller which improves the tracking performance safely in an online fashion. The thesis exploits the reinforcement learning (RL) technique which uses the nature of trial and errors to correct the tracking error. Improvement to the control performance is done by feeding an additive compensation signal given by a learned policy function to the robot's nominal controller. Two different compensation methods are proposed in the thesis. Both methods are realized by the so called actor-critic (AC) algorithm in order to cope with the continuous state of a robot arm. The first method compensates the control input of the nominal controller while the second method modifies the reference trajectory. The methods are implemented on a real 6 DoFs 3D printing robotic setup. For the test, three different reference tracking tasks are designed and executed. Furthermore, to provide a comparative study, two conventional control methods namely model predictive control (MPC) and iterative learning control (ILC) are implemented as well. Their results are then analysed and compared to that of the proposed RL-based controllers. The experimental result shows that the RL-based controllers improve the performance of the nominal control significantly. It is also shown that the performance of the learning controller is close to that of the model-based control method. For a circular, smooth trajectory, the experiment even shows that the RL-based controller is able to outmatch both MPC and ILC performance.

reinforcement learning
actor-critic
reference tracking
robot manipulator

http://resolver.tudelft.nl/uuid:ce85d671-8c62-460f-9277-6e0b1d9eb0ba

Student theses

master thesis

(c) 2015 Pane, Y.P.