Using Reinforcement Learning in a Layered Airspace to Improve Layer Change Decision

Ribeiro, M.J.; Ellerbroek, Joost; Hoekstra, J.M.

doi:10.3390/aerospace9080413

Using Reinforcement Learning in a Layered Airspace to Improve Layer Change Decision

Title

Using Reinforcement Learning in a Layered Airspace to Improve Layer Change Decision

Author

Ribeiro, M.J. (TU Delft Control & Simulation)
Ellerbroek, Joost (TU Delft Control & Simulation)
Hoekstra, J.M. (TU Delft Control & Simulation)

Date

2022

Abstract

Current predictions for future operations with drones estimate traffic densities orders of magnitude higher than any observed in manned aviation. Such densities call for further research and innovation, in particular, into conflict detection and resolution without the need for human intervention. The layered airspace concept, where aircraft are separated per vertical layer according to their heading, has been widely researched and proven to increase traffic capacity. However, aircraft traversing between layers do not benefit from this separation and alignment effect. As a result, interactions between climbing/descending and cruising aircraft can lead to a large increase in conflicts and intrusions. This paper looks into ways of reducing the impact of vertical transitions within the environment. We test two reinforcement learning methods: a decision-making module and a control execution module. The former issues a lane change command based on the planned route. The latter performs operational control to coordinate the longitude and vertical movement of the aircraft for a safe merging manoeuvre. The results show that reinforcement learning is capable of optimising an efficient driving policy for layer change manoeuvres, decreasing the number of conflicts and losses of minimum separation compared to manually defined navigation rules.

Subject

air traffic control
BlueSky ATC Simulator
conflict detection and resolution
deep deterministic policy gradient
layer change decision
layered airspace
modified voltage potential
reinforcement learning
self-separation
U-space