Let’s Go There: Combining Voice and Pointing in VR

PubDate: July 2020

Teams: University of Toronto；University of Toronto Mississauga

Writers: Jaisie Sin, Cosmin Munteanu

PDF: Let’s Go There: Combining Voice and Pointing in VR

Abstract

Hand-tracking has been advertised as a natural means to engage with a virtual environment that also enhances the feeling of presence in and lowers the barriers to entry to virtual reality. We seek to explore combining hand-tracking with voice input (which is then processed with automatic speech recognition) for a novel multimodal experience. Thus, we created Let’s Go There, which explores this joint-input method for four functions in virtual reality environments: positioning, object identification, information mapping, and disambiguation. This combination may serve as a more intuitive means for users to communicate and navigate in virtual environments. We expect there to be multiple potential applications of this multimodal form of interaction across numerous domains including training, education, teamwork, and games.