High-Res Facial Appearance Capture from Polarized Smartphone Images

PubDate: Jun 2023

Teams: Technical University of Munich 2Meta Reality Labs 3Max Planck Institute for Intelligent Systems

Writers: Dejan Azinovic, Olivier Maury, Christophe Hery, Matthias Nießner, Justus Thies

PDF: High-Res Facial Appearance Capture from Polarized Smartphone Images

Abstract

We propose a novel method for high-quality facial texture reconstruction from RGB images using a novel capturing routine based on a single smartphone which we equip with an inexpensive polarization foil. Specifically, we turn the flashlight into a polarized light source and add a polarization filter on top of the camera. Leveraging this setup, we capture the face of a subject with cross-polarized and parallel-polarized light. For each subject, we record two short sequences in a dark environment under flash illumination with different light polarization using the modified smartphone. Based on these observations, we reconstruct an explicit surface mesh of the face using structure from motion. We then exploit the camera and light colocation within a differentiable renderer to optimize the facial textures using an analysis-by-synthesis approach. Our method optimizes for high-resolution normal textures, diffuse albedo, and specular albedo using a coarse-to-fine optimization scheme. We show that the optimized textures can be used in a standard rendering pipeline to synthesize high-quality photo-realistic 3D digital humans in novel environments.