On the Number of Participants Needed for Subjective Quality Assessment of 360° Videos

PubDate: February 2020

Teams: Blekinge Institute of Technology

Writers: Hans-Jürgen Zepernick; Majed Elwardy; Yan Hu; Veronica Sundstedt

PDF: On the Number of Participants Needed for Subjective Quality Assessment of 360° Videos

Abstract

Immersive multimedia applications such as 360° videos and virtual reality (VR) live broadcasting are expected to receive increased popularity in the consumer markets. These applications provide three-dimensional spatial information and require to develop the related advanced wired and wireless ecosystems ranging from head-mounted displays (HMDs) and processors to network connectivity. To develop immersive multimedia systems that provide acceptable user experiences, it is essential to conduct subjective experiments that support assessing subjective video quality as perceived by the users. In this paper, we focus on the question of how many participants are needed in subjective experiments for 360° videos such that a statistically meaningful subjective quality assessment is supported. For this purpose, a subjective experiment for 360° videos has been conducted using the modified absolute category rating with hidden reference (M-ACR-HR) method. A statistical analysis of the opinion scores given by the participants to the visual stimuli shown on an HMD is provided. In particular, mean opinion scores (MOS), differential MOS (DMOS), and the associated margin of error (MoE) and differential MoE are obtained. It is shown that 21 to 26 participants would be sufficient to drive the differential MoE of the DMOS below the given thresholds of ε =1% and ε = 0.75%, respectively.