Training end-to-end planners in sensor-level simulation

Sammanfattning

End-to-end autonomous driving planners are commonly trained with imitation learningon offline expert demonstrations, an approach that can achieve strong openloopperformance. However, planners following this paradigm usually suffer duringclosed-loop deployment, since they have not been exposed to the consequencesof their own actions during training. Addressing this requires closed-loop training,which has typically involved high-level representations, forgoing the benefits ofsensor-level end-to-end planning.This work investigates whether recent advances in efficient 3D Gaussian Splattingcan make closed-loop training of sensor-level end-to-end planners feasible. A closedloopsimulator is developed by reconstructing driving sequences and integratingthem with a trajectory tracker and a kinematic vehicle model capable of executingthe planner’s predicted trajectories. This makes it possible to render new viewsthat deviate from the original logged trajectory. The resulting framework is usedto fine-tune a pretrained Latent TransFuser planner through reinforcement learningand closed-loop imitation learning strategies.The simulator shows that closed-loop execution reveals failure modes that are notvisible from open-loop evaluation, highlighting the importance of closed-loop trainingand evaluation. The explored training strategies produced mixed but some modestimprovements over the baseline, supporting the feasibility of Gaussian Splattingbasedsensor-level simulation as a training platform and laying the groundwork forfuture work on scalable closed-loop learning for end-to-end autonomous driving.