Evaluation of physical damage associated with action selection strategies in reinforcement learning

Evaluation of physical damage associated with action selection strategies in reinforcement learning

Koryakovskiy, I. (TU Delft Biomechatronics & Human-Machine Control)
Vallery, H. (TU Delft Biomechatronics & Human-Machine Control)
Babuska, R. (TU Delft Learning & Autonomous Control)
Caarls, W. (Pontifical Catholic University of Rio de Janeiro)

2017

Reinforcement learning techniques enable robots to deal with their own dynamics and with unknown environments without using explicit models or preprogrammed behaviors. However, reinforcement learning relies on intrinsically risky exploration, which is often damaging for physical systems. In the case of the bipedal walking robot Leo, which is studied in this paper, two sources of damage can be identified: fatigue of gearboxes due to backlash re-engagements, and the overall system damage due to falls of the robot. We investigate several exploration techniques and compare them in terms of gearbox fatigue, cumulative number of falls and undiscounted return. The results show that exploration with the Ornstein-Uhlenbeck (OU) process noise leads to the highest return, but at the same time it causes the largest number of falls. The Previous Action-Dependent Action (PADA) method results in drastically reduced fatigue, but also a large number of falls. The results reveal a previously unknown trade-off between the two sources of damage. Inspired by the OU and PADA methods, we propose four new action-selection methods in a systematic way. One of the proposed methods with a time-correlated noise outperforms the well-known e-greedy method in all three benchmarks. We provide guidance towards the choice of exploration strategy for reinforcement learning applications on real physical systems.

Adaptation
Analysis of reliability
Autonomous robotic systems
diagnosis
Fault detection
learning in physical agents
Reinforcement learning control
safety

http://resolver.tudelft.nl/uuid:8df049df-64ce-430c-b4c5-e345c961d58d

https://doi.org/10.1016/j.ifacol.2017.08.1218

IFAC-PapersOnLine, 50 (1), 6928-6933

20th World Congress of the International Federation of Automatic Control (IFAC), 2017, 2017-07-09 → 2017-07-14, Toulouse, France

Institutional Repository

journal article

© 2017 I. Koryakovskiy, H. Vallery, R. Babuska, W. Caarls