VibVid: VIBration Estimation from VIDeo by using Neural Network

Title: VibVid: VIBration Estimation from VIDeo by using Neural Network

Teams: Graduate School of Information Science and Technology, The University of Tokyo

Writers: Yoshida, Kentaro; Inoue, Seki ; Makino, Yasutoshi; Shinoda, Hiroyuki

Publication date: Nov 2017

Abstract

Along with advances in video technology in recent years, there is an increasing need for adding tactile sensation to it. Many researches on models for estimating appropriate tactile information from images and sounds contained in videos have been reported. In this paper, we propose a method named VibVid that uses machine learning for estimating the tactile signal from video with audio that can deal with the kind of video where video and tactile information are not so obviously related. As an example, we evaluated by estimating and imparting the vibration transmitted to the tennis racket from the first-person view video of tennis. As a result, the waveform generated by VibVid was almost in line with the actual vibration waveform. Then we conducted a subject experiment including 20 participants, and it showed good results in four evaluation criteria of harmony, fun, immersiveness, and realism etc.