EXController: enhancing interaction capability for VR handheld controllers using real-time vision sensing

PubDate: November 2018

Teams: University of Tsukuba

Writers: Junjian Zhang；Yaohao Chen；Satoshi Hashizume；Naoya Muramatsu；Kotaro Omomo；Riku Iwasaki；Kaji Wataru；Yoichi Ochiai

PDF: EXController: enhancing interaction capability for VR handheld controllers using real-time vision sensing

Abstract

This paper presents EXController, a new controller-mounted finger posture recognition device specially designed for VR handheld controllers. We seek to provide additional input through real-time vision sensing by attaching a near infrared (NIR) camera onto the controller. We designed and implemented an exploratory prototype with a HTC Vive controller. The NIR camera is modified from a traditional webcam and applied with a data-driven Convolutional Neural Network (CNN) classifier. We designed 12 different finger gestures and trained the CNN classifier with a dataset from 20 subjects, achieving an average accuracy of 86.17% across – subjects, and, approximately more than 92% on three of the finger postures, and more than 89% on the top-4 accuracy postures. We also developed a Unity demo that shows matched finger animations, running at approximately 27 fps in real-time.