AI-Mediated 3D Video Conferencing

PubDate: Aug 2023

Teams: US San Diego；UNC Chapel Hill；NVIDIA

Writers: Michael Stengel；Koki Nagano；Chao Liu；Matthew Chan；Alex Trevithick (US San Diego)；Shalini De Mello；Jonghyun Kim；David Luebke；Amrita Mazumdar；Shengze Wang (UNC Chapel Hill)；Mayoore Jaiswal (NVIDIA)

PDF: AI-Mediated 3D Video Conferencing

Project: AI-Mediated 3D Video Conferencing

Abstract

We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method that encodes a given 2D input into an efficient triplanar neural representation of the user, which can be rendered from novel viewpoints in real-time. Our AI-based techniques drastically reduce the cost for 3D capture, while providing a high-fidelity 3D representation on the receiver’s end at the cost of traditional 2D video streaming. Additional advantages of our AI-based approach include the ability to accommodate both photorealistic and stylized avatars, and the ability to enable mutual eye contact in multi-directional video conferencing. We demonstrate our system using a tracked stereo display for a personal viewing experience as well as a light field display for a multi-viewer experience.