Learning to Steer Strategic Agents with Fast Convergence

Sammanfattning

Incentive Design is the study of how a central-decision maker, or principal, steers self-interested decision makers, or agents, towards a socially optimal outcome. In the current paper, we investigate a dynamical incentive problem, where the agents adapt their actions to the incentives of the principal on a fixed timescale, and in turn the principal alters the incentives on a slower timescale in response to the observed behaviour. In order to ensure convergence to the social optimum, the adaption rate of the principal, or step-size schedule, must satisfy the Robinson-Moore timescale separation assumptions. As a sufficient condition, it is not clear how to design an optimal schedule, one with fast convergence, within this class. Our proposed solution is to use a Reinforcement Learning (RL) model to design a schedule. To test this approach, we implement an environment, a Network Aggregate Game, and train the model using Proximal Policy Optimization (PPO). Incentive interactions within the environment are modelled as either a Markov Decision Process (MDP) or a a partially observable Markov Decision Process (poMDP). Our learned adaptation rate results in a significantly faster convergence towards the social optimum compared to a baseline exponential function for both environment formulations. These positive results come at the expense of guaranteed convergence for arbitrary timescales compared to the baseline. Through this report, we conclude that RL methods may be a promising direction for deriving optimal adaptation rates in incentive problems with dynamics, and may serve to complement the theoretical guarantees offered by previous analysis on two-timescale approximation.