STMG: A Machine Learning Microgesture Recognition System for Supporting Thumb-Based VR/AR Input

Date：May 2024

Teams:Reality Labs

Writers:Kenrick Kin,Chengde Wan,Ken Koh,Andrei Marin,Necati Cihan Camgöz,Yubo Zhang,Yujun Cai,Fedor Kovalev,Moshe Ben-Zacharia,Shannon Hoople,Marcos Nunes-Ueno,Mariel Sanchez-Rodriguez,Ayush Bhargava,Robert Wang,Eric Sauser,Shugao Ma

PDF:STMG: A Machine Learning Microgesture Recognition System for Supporting Thumb-Based VR/AR Input

Abstract

AR/VR devices have started to adopt hand tracking, in lieu of controllers, to support user interaction. However, today’s hand input rely primarily on one gesture: pinch. Moreover, current mappings of hand motion to use cases like VR locomotion and content scrolling involve more complex and larger arm motions than joystick or trackpad usage. STMG increases the gesture space by recognizing additional small thumb-based microgestures from skeletal tracking running on a headset. We take a machine learning approach and achieve a 95.1% recognition accuracy across seven thumb gestures performed on the index finger surface: four directional thumb swipes (left, right, forward, backward), thumb tap, and fingertip pinch start and pinch end. We detail the components to our machine learning pipeline and highlight our design decisions and lessons learned in producing a well generalized model. We then demonstrate how these microgestures simplify and reduce arm motions for hand-based locomotion and scrolling interactions.