DG-Recon: Depth-Guided Neural 3D Scene Reconstruction

PubDate: Oct 2023

Teams: Qualcomm Technologies

Writers: Jihong Ju Ching-Wei Tseng Oleksandr Bailo Georgi Dikov Mohsen Ghafoorian

PDF: DG-Recon: Depth-Guided Neural 3D Scene Reconstruction

Abstract

A key challenge in neural 3D scene reconstruction from monocular images is to fuse features back projected from various views without any depth or occlusion information. We address this by leveraging monocular depth priors, which effectively guide the fusion to improve surface prediction and skip over irrelevant, ambiguous, or occluded features. Furthermore, we revisit the averagebased fusion used by most neural 3D reconstruction methods and propose two alternatives, a variance-based and a cross-attention-based fusion module, that are more efficient and effective than the average-based and self-attentionbased counterparts. Compared to the NeuralRecon baseline, the proposed DG-Recon models significantly improve the reconstruction quality and completeness while remaining in real-time. Our method achieves state-of-the-art online reconstruction results on the ScanNet dataset and is on par with the current best offline method, which repeatedly accesses keyframes from the entire video sequence. Our ScanNet-trained model also generalizes robustly to the challenging 7-Scenes dataset and a subset of SUN3D containing scenes as big as an entire floor.