Real-time generation of Gaussian splats for vehicle monitoring and remote control

Sammanfattning

3D Gaussian Splatting has shown strong potential for high-quality real-time rendering of reconstructed 3D scenes. However, most existing pipelines rely on offline optimization, where the full image sequence is available before training begins. This makes them difficult to apply in latency-sensitive applications such as remote vehicle operation, where the scene representation must be generated and updated continuously. This thesis investigates whether Gaussian Splatting can be adapted for real-time scene reconstruction from streaming camera data, with the goal of supporting either first-, or third-person visualization for teleoperation. A custom incremental Gaussian Splatting pipeline was implemented by combining ORB-SLAM3 for camera tracking and sparse point extraction with online Gaussian initialization, photometric optimization, adaptive density control, and CUDA-based rendering. The system was primarily evaluated using a synchronized stereo fisheye camera rig, with additional comparisons using a Serve robot dataset with LiDAR data and an RGB-D dataset captured with an Intel RealSense D435i. The results were also compared against an offline 3D Gaussian Splatting reconstruction to highlight the trade-off between reconstruction quality and real-time operation. The results show that real-time Gaussian Splatting is feasible on consumer-level hardware in controlled indoor environments, but that reconstruction quality remains lower than offline 3DGS. The real-time system produces softer geometry, more floating artifacts, and lower consistency, mainly due to sparse input geometry, limited optimization time per frame, tracking errors, and hardware constraints. The RGB-D results showed more stable geometry than the stereo setup, as well as cheaper to compute, indicating that direct depth measurements and having fewer pixels to process are highly valuable for this type of pipeline. The Serve dataset proved more challenging because of fisheye imagery, sparse LiDAR supervision, forward vehicle motion, and fewer repeated observations. Overall, the thesis shows that real-time Gaussian Splatting is a promising approach for interactive 3D visualization in teleoperation, but further work is needed before it can replace conventional camera-based remote operation systems. Future improvements should focus on more reliable depth integration, better handling of dynamic objects, fisheye-aware processing, broader evaluation, and more scalable optimization.