User Navigation Modeling, Rate-Distortion Analysis, and End-to-End Optimization for Viewport-Driven 360∘ Video Streaming
PubDate: Aug 2022
Teams: EPFL；Cesson-Sévigné；Synmedia；Adobe
Writers: Jacob Chakareski; Xavier Corbillon; Gwendal Simon; Vishwanathan Swaminathan
Abstract
The emerging technologies of Virtual Reality (VR) and 360∘ video introduce new challenges for state-of-the-art video communication systems. Enormous data volume and spatial user navigation are unique characteristics of 360∘ videos that necessitate a space-time effective allocation of the available network streaming bandwidth over the 360∘ video content to maximize the Quality of Experience (QoE) delivered to the user. Towards this objective, we investigate a framework for viewport-driven rate-distortion optimized 360∘ video streaming that integrates the user view navigation patterns and the spatiotemporal rate-distortion characteristics of the 360∘ video content to maximize the delivered user viewport video quality, for the given network/system resources. The framework comprises a methodology for assigning dynamic navigation likelihoods over the 360∘ video spatiotemporal panorama, induced by the user navigation patterns, an analysis and characterization of the 360∘ video panorama's spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the content, and an optimization problem formulation and solution that capture and aim to maximize the delivered expected viewport video quality, given a user's navigation patterns, the 360∘ video encoding/streaming decisions, and the available system/network resources. We formulate a Markov model to capture the navigation patterns of a user over the 360∘ video panorama and simultaneously extend our actual navigation datasets by synthesizing additional realistic navigation data. Moreover, we investigate the impact of using two different tile sizes for equirectangular tiling of the 360∘ video panorama. Our experimental results demonstrate the advantages of our framework over the conventional approach of streaming a monolithic uniformly-encoded 360∘ video and a state-of-the-art navigation-speed based reference method.
