Deep dive: deep-neural-network-based video extension for immersive head-mounted display experiences

PubDate: June 2019

Teams: The University of Tokyo

Writers: Naoki Kimura；Michinari Kono；Jun Rekimoto

PDF: Deep dive: deep-neural-network-based video extension for immersive head-mounted display experiences

Abstract

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed. Head-mounted displays (HMDs) are home-friendly pervasive devices, which can provide an immersive video experience owing to their wide field-of-view (FoV) and separation of users from the outside environment. They are often used for viewing panoramic and stereoscopic recorded videos or virtually generated environments, but the demand for viewing standard plane videos with HMDs has increased. However, the theater mode, which restricts the FoV, is basically used for viewing plane videos. Thus, the advantages of HMDs are not fully utilized. Therefore, we explored a method for viewing plane videos by an HMD, in combination with view augmentation by LED implants to the HMD. We have constructed a system for viewing plane videos using an HMD with a deep neural network (DNN) model optimized for generating and extending images for peripheral vision and wide FoV customization. We found that enlarging the original video and extending the video with our DNN model can improve the user experience. However, our method provided more comfortable viewing by preventing motion sickness in a first-person-view video.