Machine Learning Algorithms for Energy Trading of Battery Energy Storage Systems : Reinforcement learning for trading energy on dual electricity markets

Sammanfattning

The battery energy storage system (BESS) holds the promise of becoming an essential element in our energy landscape. With the increasing need for renewable energy and electrification, a BESS serves as backup power, grid frequency balancing, and playing a crucial role in achieving 100% renewable electricity production by 2040 in Sweden. This thesis aims to study intelligent energy trading algorithms for BESS with two markets. The algorithms would allow BESS to trade simultaneously with two markets, day-ahead (energy) market and FCR-N.The problem of trading is solved using reinforcement learning (RL) particularly, multi-agent reinforcement learning (MARL), are proposed as potential solutions for learning energy trading strategies across multiple markets, addressing a gap in current research. In this study two main algorithms are used: Deep Q-networks (DQN), and Advantage Actor-Critic (A2C). These two algorithms are adapted to the MARL’s paradigms. This thesis answers three main questions. First, if any of the MARL variations of the two mentioned algorithms have any advantage over the others. The results suggest that the CTDE variation of A2C performs the best, followed by centralized variation of A2C. Second, the discrete action spaces and continuous action spaces are compared. The algorithms with continuous action spaces achieved higher revenues. The continuous action spaces let the agents decide the exact volumes of the energy to trade. This is while in the discrete action spaces the agents can only choose the volumes from a defined set of values. Third and last, the results from the experiments suggest that trading with two markets results in higher revenue than trading with one market. All the MARL algorithms have higher revenue compared to the simple hard-rule strategy designed for trading with two markets. This thesis shows that the RL and MARL algorithms can be used for creating profitable trading agents and for identifying successful trading strategies.