Policy Gradient Algorithms in Atari Asteroids

Sammanfattning

This research tests the abilities of the Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO) methods of Reinforcement Learning (RL), as well as the long short-term memory (LSTM) optimized PPO, in the benchmark Arcade Learning Environment Atari Asteroids. Since deep q-networks (DQNs) have trouble solving the game out-of-the-box, this research compares the previously mentioned algorithms with the benchmark set in the paper Human-level control through deep reinforcement learning as well as state-of-the-art models drawn from the literature. Through a series of hyperparameter optimization steps as well as full training runs, this paper tests the models and analyzes the learning behaviors of the algorithms. The resulting models showed that the algorithms were unable to reach the performance of modern solutions, and also did not significantly outperform the DQN or human expert. The key conclusions found were that these algorithms, despite their quality in other tasks, were unable to meaningfully attribute delayed rewards to past actions, and faced difficulties learning long-term survival strategies; instead, they fall into optimizing local regions and converge to a suboptimal policy. Additionally, LSTM layers added to PPO were beneficial when using raw visual input, though not significantly compared to expert scores. However, this research will serve to improve availability of detailed data on the planning, hyperparameter optimization, and training stages for tasks with similar dynamics to Atari Asteroids.