Towards Real Time Object Recognition For Context Awareness in Mixed Reality: A Machine Learning Approach
PubDate: May 2020
Teams: Virginia Tech
Writers: Archi Dasgupta; Mark Manuel; Rifat Sabbir Mansur; Nabil Nowak; Denis Gračanin
Abstract
Mixed Reality (MR) technologies provide users with an environment that incorporates virtual objects and metadata into their physical surroundings. This opens up exciting new possibilities for applications in various domains including: education, training, healthcare, and Computer Supported Cooperative Work (CSCW). Recognizing physical objects in the environment provides contextual clues that can improve user task performance and influence effective cognitive load for complex tasks. However, this facet of MR is still relatively nascent because of hardware limitations. Presently, MR headsets can only detect physical surfaces; they can not distinguish between physical artifacts that a user might need in their work environment. Machine Learning (ML) based object recognition solutions can help overcome this limitation. However, current commercially available MR devices have insufficient computational capabilities and are unable to support the state-of-the-art ML based object recognition solutions. To address these challenges, we describe a novel approach and the corresponding framework by introducing a hardware level separation between the two tasks of object recognition and rendering virtual components. Our approach leverages ML based object recognition, MR, and a portable IoT edge computing platform to support contextual awareness. We describe the implementation of this approach and provide a case study. The prototype implementation uses an edge device to process and recognize objects to get information about the user’s physical environment and provides the user with situational awareness through the MR device. Our case study uses a Christmas tree decoration activity to demonstrate the ability of the proposed framework to support assembly-type tasks. The case study establishes the applicability of the proposed approach to a variety of problem domains. Object recognition and context aware capabilities, combined with virtual indicators and instructions, provide a solution that increases usability, helps reduce human error, and improves the overall task performance.
