VRComm: an end-to-end web system for real-time photorealistic social VR communication

PubDate: July 2021

Teams: TNO

Writers: Simon N. B. Gunkel；Rick Hindriks；Karim M. El Assal；Hans M. Stokking；Sylvie Dijkstra-Soudarissanane；Frank ter Haar;Omar Niamut

PDF: VRComm: an end-to-end web system for real-time photorealistic social VR communication

Abstract

Tools and platforms that enable remote communication and collaboration provide a strong contribution to societal challenges. Virtual meetings and conferencing, in particular, can help to reduce commutes and lower our ecological footprint, and can alleviate physical distancing measures in case of global pandemics. In this paper, we outline how to bridge the gap between common video conferencing systems and emerging social VR platforms to allow immersive communication in Virtual Reality (VR). We present a novel VR communication framework that enables remote communication in virtual environments with real-time photorealistic user representation based on colour-and-depth (RGBD) cameras and web browser clients, deployed on common off-the-shelf hardware devices. The paper’s main contribution is threefold: (a) a new VR communication framework, (b) a novel approach for real-time depth data transmitting as a 2D grayscale for 3D user representation, including a central MCU-based approach for this new format and (c) a technical evaluation of the system with respect to processing delay, CPU and GPU usage.