Enhancing Deep Learning-Based 3D Face Reconstructions with Consumer-Grade Depth Data

Sammanfattning

Deep learning models have significantly improved the advancement of monocular 3D face reconstruction models. However, such models often struggle to capture person-specific high-frequency details. To address this issue, this report proposes a hybrid reconstruction pipeline that enhances DECA produced 3D face models by incorporating RGB-D data captured with the consumer grade iPhone TrueDepth camera. The final pipeline uses DECA output, synchronized RGB-D, face segmentation, landmark detection, and ICP. Starting from a DECA-generated coarse mesh, we fit an independent mesh copy to each depth scan, constrain vertex displacements to the surface normal, and compute the per-vertex mean across all fitted meshes to obtain a refined, personalized shape. This method allows for correcting 3D facial models to better reflect individual characteristics. The corrected reconstructions show an improved detail correspondence to the target person, having converged towards the mean shape of several depth scans. Since this method only modifies existing meshes, the resulting personalized mesh is still morphable.