Multimodal Pre-training with Language Alignment for Outdoor Scenarios : Aligning 3D LiDAR, Image & Language for Zero-Shot Outdoor 3D Object Recognition

Sammanfattning

Advances in 3D deep learning have improved LiDAR point cloud analysis, benefiting autonomous driving, robotics, and geospatial tasks. In the field of foundation model research, the alignment of sensor features with language features has allowed open-vocabulary perception, which has far-reaching applications. However, research in 3D language alignment is still at the nascent stages, mainly due to a lack of large, high-quality annotated datasets, limiting generalization and transfer learning. This line of research is virtually unexplored for outdoor scenarios, where sensor inputs are sparse, have large domain gaps, and incur occlusions. With this background in mind, the goal in this thesis is to attempt language alignment for sparse 3D outdoor scenes. Concretely, the aim is to align a 3D encoder to predict CLIP tokens for LiDAR-detected outdoor objects. This alignment with the multimodal space of CLIP (image & text) enables zero-shot 3D object understanding, improving generalization to unseen categories. Transferring 2D model knowledge to 3D perception compensates for limited 3D datasets. The significance of this work in the industrial environment is, for instance, enabling intelligent tagging of raw data logs captured in outdoor scenes at Scania. Mapping LiDAR to CLIP tokens allows dynamic, scalable tagging without retraining, making it ideal for large-scale driving datasets where new object classes and semantic changes frequently emerge.