Leveraging Diffusion Models and Gaussian Splatting for Contextually Consistent 3D Scene Expansion

Sammanfattning

There have been significant advances in high resolution 3D reconstruction with the use of Gaussian Splatting. With this technique multi-view high resolution images can be reconstructed into high resolution 3D scenes by first building up a point cloud and optimizing 3D Gaussians projected on the point cloud, minimizing the difference between images and the rendered scene. Recent research has delved into using generative models to modify reconstructions created with Gaussian Splatting. Previous work is either limited in the scale of which scenes can be generated or focused on editing parts of existing reconstructions. To address the limitations, we propose a framework of iterative scene generation with the purpose of extending 3D reconstructions created using Gaussian Splatting. The framework extends a baseline reconstruction by adding new information created through inpainting and estimating point clouds in the missing areas of the reconstruction. In efforts to automate this process, functions for identifying poor reconstruction and a dynamic weighting loss are presented. The results show that it is possible to extend existing reconstructions without a significant loss in 3D representation.