Explainability of Transformer-Based Reinforcement Learning

Sammanfattning

The transformer architecture has revolutionized the field of sequential processing within AI. It has been widely adopted in fields such as Natural Language Process- ing (NLP) and Computer Vision (CV) due to their ability to capture long-range dependencies and contextual information through attention mechanism. This capability opens up new possibilities for their application in Reinforcement Learn- ing (RL), particularly for tasks that can be modeled as sequence learning problems. In RL, transformers can enhance decision-making by understanding and leveraging temporal dependencies. As models like transformers grow complex, the need to meet the AI Act’s transparency requirement and validate that the models are performing correctly is also critical. This thesis explores the applicability and explainability of the transformer architecture in RL for autonomous coordination in mobile networks. For the applicability, the Decision Transformer (DT) is applied and evaluated in a mo- bile network setup for a specific use case, that is autonomous coordination. By mod- eling RL as a sequence learning problem, the DT architecture is used to generate a policy. The findings reveal that even sub-optimal trajectories can generate a policy comparable to those trained through conventional online methods. The study shows the advantages of using DT and the relationship between the generated policies with the quality of the trajectories used for training. It also identifies limitations in stochas- tic environments where DT does not fully leverage temporal dependencies. For the explainability of DT, two different approaches were used. Firstly, the intrinsic explainability methods, which are based on the internal attention mechanism of the transformer are used. They revealed some useful information about the dynamics of the environment and the temporal dependencies for our use case. Secondly, post-hoc methods were used to perform token-level attribution and then feature-level attribution by extracting features from the state tokens. This work presents a detailed analysis and comparison of both methods, including their working details, limitations, types of explanations drawn from them, and their evaluation.