Optimizing deep reinforcement learning policies for deteriorating systems considering ordered action structuring and value of information

Optimizing deep reinforcement learning policies for deteriorating systems considering ordered action structuring and value of information

Andriotis, C. (TU Delft Structural Design & Mechanics)
Papakonstantinou, K.G. (Pennsylvania State University)

Li, J. (editor)
Spanos, Pol D. (editor)
Chen, J.B. (editor)
Peng, Y.B. (editor)

2022

Inspection and maintenance (I&M) optimization entails many sources of computational complexity, among others, due to high-dimensional decision and state variables in multi-component systems, long planning horizons, stochasticity of objectives and constraints, and inherent uncertainties in measurements and models. This paper studies how the above can be addressed within the context of constrained Partially Observable Markov Decision Processes (POMDPs) and Deep Reinforcement Learning (DRL) in a unified fashion. Special emphasis is paid on how ordered action structuring of I&M actions can be exploited to decompose the respective policy parametrizations in actor-critic DRL schemes, resulting into fully decoupled maintenance and inspection actors. It is shown that the Value of Information (VoI) is naturally utilized in such POMDP control frameworks, as directly associated with the DRL advantage functions that emerge in the gradient computations of the inspection policy parameters. Overall, the presented approach, following the natural flow of engineering decisions, results in new architectural configurations for policy networks, facilitating more efficient training, while alleviating further the dimensionality burdens related to combinatorial definitions of I&M actions. The efficiency of the methodology is demonstrated in numerical experiments of a structural system subject to corrosion, where the optimization problem is formulated to concurrently account for state and model uncertainties as well as long-term probability of failure exceedance constraints. Results showcase that the obtained DRL policies considerably outperform standard decision rules.

inspection & maintenance
deep reinforcement learning
partially observable Markov decision processes
value of information
stochastic constraints
decision theory

http://resolver.tudelft.nl/uuid:b64f6296-e747-47e9-81e3-622db0ebc781

2023-07-01

Proceedings of the 13th International Conference on Structural Safety and Reliability (ICOSSAR)

International Conference on Structural Safety and Reliability, 2022-09-13 → 2022-09-17, Tongji University, Shanghai, China

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Institutional Repository

conference paper

© 2022 C. Andriotis, K.G. Papakonstantinou