Automatic point cloud colourisation at scene-level using various neural network approaches

Sammanfattning

Recent breakthroughs in 3D digitisation have opened up new possibilities for capturing and preserving the physical world. These advances offer practical solutions for the industry to deploy digital twins. Modelling real-world scenes involves collecting and processing high-resolution point cloud data through an automated workflow, leveraging cutting-edge deep learning techniques. A key step in this process requires training a segmentation network on real-world coloured data, augmented with synthetic uncoloured data. Colourising this synthetic data holds potential for enhancing the quality of training datasets, leading to improved segmentation and overall scene reconstruction accuracy. Although this task has been successfully explored for 2D images, point-cloud colourisation raises a more difficult challenge that remains largely underexplored, particularly at the scene level. Many uncertainties remain regarding the effective evaluation of the task, suitable neural network architectures, and mitigation strategies for common artefacts, such as desaturation or limited semantic understanding. To address these challenges, point cloud segmentation architectures have been repurposed to the colourisation task. These backbones are built on a fully convolutional encoder-decoder structure that employs downsampling for feature extraction and upsampling for reconstruction, while incorporating skip connections to preserve spatial information (in line with the U-Net paradigm). Numerous metrics have been followed to assess colourisation performances. Furthermore, the generative adversarial network (GAN) framework has been investigated to overcome the limitations of the U-Net paradigm, potentially enhancing the generation of more realistic output. Overall, the results suggest that U-Net-like architectures are prone to uneven performance across different extractors, with some models exhibiting desaturation or difficulties in efficiently addressing the task ambiguity. Although GAN show improvements in managing multimodal uncertainty, they remain undermined by recurring issues of desaturation and training instability, limiting their overall performance. Furthermore, both quantitative and qualitative evaluation often fail to fully represent the model’s true capabilities.