Seamless Bare-Hand Interaction in Mixed Reality

PubDate: April 2019

Teams: Fondazione Bruno Kessler

Writers: Caterina Battisti; Stefano Messelodi; Fabio Poiesi

PDF: Seamless Bare-Hand Interaction in Mixed Reality

Abstract

Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth, estimated with an auxiliary infrared stereo camera used for hand tracking, to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors, we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared, and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.