Teams: 1 Massachusetts Institute of Technology, 2 University of Michigan

Writers: Tara Boroushaki 1, Maisy Lam 1, Laura Dodds 1, Aline Eid 1,2, Fadel Adib 1

PDF: Augmenting Augmented Reality with Non-Line-of-Sight Perception

Abstract

We present the design, implementation, and evaluation of X-AR, an augmented reality (AR) system with non-line-of-sight perception. X-AR augments AR headsets with RF sensing to enable users to see things that are otherwise invisible to the human eye or to stateof-the-art AR systems. Our design introduces three main innovations: the first is an AR-conformal antenna that tightly matches the shape of the AR headset visor while providing excellent radiation and bandwidth capabilities for RF sensing. The second is an RF-visual synthetic aperture localization algorithm that leverages natural human mobility to localize RF-tagged objects in line-ofsight and non-line-of-sight settings. Finally, the third is an RF-visual verification primitive that fuses RF and vision to deliver actionable tasks to end users such as picking verification. We built an end-to-end prototype of our design by integrating it into a Microsoft Hololens 2 AR headset and evaluated it in line-of-sight and nonline-of-sight environments. Our results demonstrate that X-AR achieves decimeter-level RF localization (median of 9.8 cm) of fully-occluded items and can perform RFvisual picking verification with over 95% accuracy (FScore) when extracting RFID-tagged items.These results show that X-AR is successful in extending AR systems to non-line-of-sight perception, with important implications to manufacturing, warehousing, and smart home applications. Demo video: y2u.be/bdUN21ft7G0

Augmenting Augmented Reality with Non-Line-of-Sight Perception最先出现在Nweon Paper。

Teams: Ben-Gurion University of the Negev,Meta;International Audio Laboratories Erlangen;University of Huddersfield;, Microsoft Research;The Australian National University

Writers: Boaz Rafaely, Vladimir Tourbabin, Emanuel Habets, Zamir Ben-Hur, Hyunkook Lee, Hannes Gamper, Lior Arbel, Lachlan Birnie, Thushara Abhayapala, Prasanga Samarasinghe

PDF: Spatial audio signal processing for binaural reproduction of recorded acoustic scenes – review and challenges

Abstract

Spatial audio has been studied for several decades, but has seen much renewed interest recently due to advances in both software and hardware for capture and playback, and the emergence of applications such as virtual reality and augmented reality. This renewed interest has led to the investment of increasing efforts in developing signal processing algorithms for spatial audio, both for capture and for playback. In particular, due to the popularity of headphones and earphones, many spatial audio signal processing methods have dealt with binaural reproduction based on headphone listening. Among these new developments, processing spatial audio signals recorded in real environments using microphone arrays plays an important role. Following this emerging activity, this paper aims to provide a scientific review of recent developments and an outlook for future challenges. This review also proposes a generalized framework for describing spatial audio signal processing for the binaural reproduction of recorded sound. This framework helps to understand the collective progress of the research community, and to identify gaps for future research. It is composed of five main blocks, namely: the acoustic scene, recording, processing, reproduction, and perception and evaluation. First, each block is briefly presented, and then, a comprehensive review of the processing block is provided. This includes topics from simple binaural recording to Ambisonics and perceptually motivated approaches, which focus on careful array configuration and design. Beamforming and parametric-based processing afford more flexible designs and shift the focus to processing and modeling of the sound field. Then, emerging machine- and deep-learning approaches, which take a further step towards flexibility in design, are described. Finally, specific methods for signal transformations such as rotation, translation and enhancement, enabling additional flexibility in reproduction and improvement in the quality of the binaural signal, are presented. The review concludes by highlighting directions for future research.

Spatial audio signal processing for binaural reproduction of recorded acoustic scenes – review and challenges最先出现在Nweon Paper。

Teams: Max Planck Institute for Intelligent Systems;Microsoft Mixed Reality & AI Lab

Writers: Balamurugan Thambiraja, Ikhsanul Habibie, Sadegh Aliakbarian, Darren Cosker, Christian Theobalt, Justus Thies

PDF: Imitator: Personalized Speech-driven 3D Facial Animation

Abstract

Speech-driven 3D facial animation has been widely explored, with applications in gaming, character animation, virtual reality, and telepresence systems. State-of-the-art methods deform the face topology of the target actor to sync the input audio without considering the identity-specific speaking style and facial idiosyncrasies of the target actor, thus, resulting in unrealistic and inaccurate lip movements. To address this, we present Imitator, a speech-driven facial expression synthesis method, which learns identity-specific details from a short input video and produces novel facial expressions matching the identity-specific speaking style and facial idiosyncrasies of the target actor. Specifically, we train a style-agnostic transformer on a large facial expression dataset which we use as a prior for audio-driven facial expressions. Based on this prior, we optimize for identity-specific speaking style based on a short reference video. To train the prior, we introduce a novel loss function based on detected bilabial consonants to ensure plausible lip closures and consequently improve the realism of the generated expressions. Through detailed experiments and a user study, we show that our approach produces temporally coherent facial expressions from input audio while preserving the speaking style of the target actors.

Imitator: Personalized Speech-driven 3D Facial Animation最先出现在Nweon Paper。

Teams:  Beihang University

Writers: Zhimin Wang; Yuxin Zhao; Feng Lu

PDF: Gaze-Vergence-Controlled See-Through Vision in Augmented Reality

Abstract

Gaze-Vergence-Controlled See-Through Vision in Augmented Reality最先出现在Nweon Paper。

Teams: ETH Zurich；Microsoft Mixed Reality & AI Lab

Writers: Paul-Edouard Sarlin, Mihai-Alexandru Dusmanu, Johannes L. Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, Marc Pollefeys

PDF: LaMAR: Benchmarking Localization and Mapping for Augmented Reality

Abstract

Localization and mapping is the foundational technology for augmented reality (AR) that enables sharing and persistence of digital content in the real world. While significant progress has been made, researchers are still mostly driven by unrealistic benchmarks not representative of real-world AR scenarios. In particular, benchmarks are often based on small-scale datasets with low scene diversity, captured from stationary cameras, and lacking other sensor inputs like inertial, radio, or depth data. Furthermore, ground-truth (GT) accuracy is mostly insufficient to satisfy AR requirements. To close this gap, we introduce a new benchmark with a comprehensive capture and GT pipeline, which allow us to co-register realistic AR trajectories in diverse scenes and from heterogeneous devices at scale. To establish accurate GT, our pipeline robustly aligns the captured trajectories against laser scans in a fully automatic manner. Based on this pipeline, we publish a benchmark dataset of diverse and large-scale scenes recorded with head-mounted and hand-held AR devices. We extend several state-of-the-art methods to take advantage of the AR specific setup and evaluate them on our benchmark. Based on the results, we present novel insights on current research gaps to provide avenues for future work in the community.

LaMAR: Benchmarking Localization and Mapping for Augmented Reality最先出现在Nweon Paper。

Teams: Microsoft

Writers: Erroll Wood Tadas Baltrušaitis Charlie Hewitt Matthew Johnson Jingjing Shen Nikola Milosavljevic Daniel Wilde Stephan Garbin Chirag Raman Jamie Shotton Toby Sharp Ivan Stojiljkovic Thomas J. Cashman Julien Valentin

PDF: 3D Face Reconstruction with Dense Landmarks

Abstract

Landmarks often play a key role in face analysis, but many aspects of identity or expression cannot be represented by sparse landmarks alone. Thus, in order to reconstruct faces more accurately, landmarks are often combined with additional signals like depth images or techniques like differentiable rendering.

Can we keep things simple by just using more landmarks?

In answer, we present the first method that accurately predicts ten times as many landmarks as usual, covering the whole head, including the eyes and teeth. This is accomplished using synthetic training data, which guarantees perfect landmark annotations. By fitting a morphable model to these dense landmarks, we achieve state-of-the-art results for monocular 3D face reconstruction in the wild. We show that dense landmarks are an ideal signal for integrating face shape information across frames by demonstrating accurate and expressive facial performance capture in both monocular and multi-view scenarios. Finally, our method is highly efficient: we can predict dense landmarks and fit our 3D face model at over 150FPS on a single CPU thread.

3D Face Reconstruction with Dense Landmarks最先出现在Nweon Paper。

Teams: Royal Netherlands Aerospace Centre

Writers: Jürgen Teutsch; Tanja Bos; Marcel van Apeldoorn; Lansenou Camara

PDF: Attention Guidance for Tower ATC Using Augmented Reality Devices

Abstract

In 2021 Royal NLR carried out innovative technology experiments on their high-fidelity real-time air traffic control simulation and validation platform, NARSIM. These experiments were part of the SESAR 2020 project Digital Technologies for Tower (DTT). The technology option that was investigated focused on advanced HMI interaction modes for aerodrome tower controllers. More particular, Attention Capturing and Guidance strategies with an Augmented Reality device, the Microsoft HoloLens 2™, were evaluated inside an aerodrome control tower environment for Amsterdam Airport Schiphol, one of the major European hub airports.The NARSIM environment consisted of a realistic but downscaled presentation of the airport with two tower controller working positions emulating current tower systems. Such a set-up allowed researchers to focus their work on the application of Augmented Reality with the introduction of (virtual) aircraft labels as well as special symbology and auditory cues for capturing and guiding tower controller attention in the case of critical events. Several typical attention-critical events that may occur at an airport, such as go-around operations and runway incursions, were orchestrated by a team of NLR experts and presented to the tower controllers while they were operating traffic as usual. Human performance and ATC operational experts observed and analyzed the simulations.This paper describes the steps taken and the challenges encountered when integrating the HoloLens inside the NARSIM Tower environment. Furthermore, it explores the proposed operational concept for Attention Capturing and Guidance with the HoloLens and how it was realized inside the device. The results of the technical evaluation activity with two experienced air traffic controllers are described in detail. These results came to the conclusion that the device in combination with the concept was a favorable addition to the controller working environment. While desired technical performance improvements, mostly related to user comfort and general adjustments, depend on further vendor development, the used HoloLens was seen as a technically useful device for implementing prototype applications for Attention Capturing and Guidance with aural and visual cues.In the final sections of the paper, an outlook into the expected future use of Augmented Reality devices in conventional control tower environments is given.

Attention Guidance for Tower ATC Using Augmented Reality Devices最先出现在Nweon Paper。

Teams: Beihang University

Writers: Zhimin Wang; Yuxin Zhao; Feng Lu

PDF: Control with Vergence Eye Movement in Augmented Reality See-Through Vision

Abstract

Augmented Reality (AR) see-through vision has become a recent research focus since it enables the user to see through a wall and see the occluded objects. Most existing works only used common modalities to control the display for see-through vision, e.g., button clicking and speech control. However, we use visual system to observe see-through vision. Using an addition interaction channel will distract the user and degrade the user experience. In this paper, we propose a novel interaction method using vergence eye movement for controlling see-through vision in AR. Specifically, we first customize eye cameras and design gaze depth estimation method for Microsoft HoloLens 2. With our algorithm, fixation depth can be computed from the vergence, and used to manage the see-through vision. We also propose two control techniques of gaze vergence. The experimental results show that the gaze depth estimation method is efficient. The difference cannot be found between these two modalities in terms of completion time and the number of successes.

Control with Vergence Eye Movement in Augmented Reality See-Through Vision最先出现在Nweon Paper。

Teams: Colorado State University;Microsoft Research

Writers: Adam S. Williams, Xiaoyan Zhou, Michel Pahud, Francisco R. Ortega

PDF: A Pilot Study on The Impact of Stereoscopic Display Type on User Interactions Within A Immersive Analytics Environment

Abstract

Immersive Analytics (IA) and consumer adoption of augmented reality (AR) and virtual reality (VR) head-mounted displays (HMDs) are both rapidly growing. When used in conjunction, stereoscopic IA environments can offer improved user understanding and engagement; however, it is unclear how the choice of stereoscopic display impacts user interactions within an IA environment. This paper presents a pilot study that examines the impact of stereoscopic display type on object manipulation and environmental navigation using consumer-available AR and VR displays. This work finds that the display type can impact how users manipulate virtual content, how they navigate the environment, and how able they are to answer questions about the represented data.

A Pilot Study on The Impact of Stereoscopic Display Type on User Interactions Within A Immersive Analytics Environment最先出现在Nweon Paper。

Teams: Ecole de technologie sup´erieure

Writers: Michael J. McGuffin (ETS), Ryan Servera (ETS), Marie Forest (ETS)

PDF: Path Tracing in 2D, 3D, and Physicalized Networks

Abstract

It is common to advise against using 3D to visualize abstract data such as networks, however Ware and Mitchell’s 2008 study showed that path tracing in a network is less error prone in 3D than in 2D. It is unclear, however, if 3D retains its advantage when the 2D presentation of a network is improved using edge-routing, and when simple interaction techniques for exploring the network are available. We address this with two studies of path tracing under new conditions. The first study was preregistered, involved 34 users, and compared 2D and 3D layouts that the user could rotate and move in virtual reality with a handheld controller. Error rates were lower in 3D than in 2D, despite the use of edge-routing in 2D and the use of mouse-driven interactive highlighting of edges. The second study involved 12 users and investigated data physicalization, comparing 3D layouts in virtual reality versus physical 3D printouts of networks augmented with a Microsoft HoloLens headset. No difference was found in error rate, but users performed a variety of actions with their fingers in the physical condition which can inform new interaction techniques.

Path Tracing in 2D, 3D, and Physicalized Networks最先出现在Nweon Paper。

Teams: Georgia Institute of Technology;Microsoft Research

Writers: Nolan Wagener, Andrey Kolobov, Felipe Vieira Frujeri, Ricky Loynd, Ching-An Cheng, Matthew Hausknecht

PDF: MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control

Abstract

Simulated humanoids are an appealing research domain due to their physical capabilities. Nonetheless, they are also challenging to control, as a policy must drive an unstable, discontinuous, and high-dimensional physical system. One widely studied approach is to utilize motion capture (MoCap) data to teach the humanoid agent low-level skills (e.g., standing, walking, and running) that can then be re-used to synthesize high-level behaviors. However, even with MoCap data, controlling simulated humanoids remains very hard, as MoCap data offers only kinematic information. Finding physical control inputs to realize the demonstrated motions requires computationally intensive methods like reinforcement learning. Thus, despite the publicly available MoCap data, its utility has been limited to institutions with large-scale compute. In this work, we dramatically lower the barrier for productive research on this topic by training and releasing high-quality agents that can track over three hours of MoCap data for a simulated humanoid in the dm_control physics-based environment. We release MoCapAct (Motion Capture with Actions), a dataset of these expert agents and their rollouts, which contain proprioceptive observations and actions. We demonstrate the utility of MoCapAct by using it to train a single hierarchical policy capable of tracking the entire MoCap dataset within dm_control and show the learned low-level component can be re-used to efficiently learn downstream high-level tasks. Finally, we use MoCapAct to train an autoregressive GPT model and show that it can control a simulated humanoid to perform natural motion completion given a motion prompt.
Videos of the results and links to the code and dataset are available at this https URL.

MoCapAct: A Multi-Task Dataset for Simulated Humanoid Control最先出现在Nweon Paper。

Teams: Vanderbilt University；University of Utah

Writers: Haley Adams; Jeanine Stefanucci; Sarah Creem-Regehr; Bobby Bodenheimer

PDF: Depth Perception in Augmented Reality: The Effects of Display, Shadow, and Position

Abstract

Although it is commonly accepted that depth perception in augmented reality (AR) displays is distorted, we have yet to isolate which properties of AR affect people’s ability to correctly perceive virtual objects in real spaces. From prior research on depth perception in commercial virtual reality, it is likely that ergonomic properties and graphical limitations impact visual perception in head-mounted displays (HMDs). However, an insufficient amount of research has been conducted in augmented reality HMDs for us to begin isolating pertinent factors in this family of displays. To this end, in the current research, we evaluate absolute measures of distance perception in the Microsoft HoloLens 2, an optical see-through AR display, and the Varjo XR-3, a video see-through AR display. The current work is the first to evaluate either device using absolute distance perception as a measure. For each display, we asked participants to verbally report distance judgments to both grounded and floating targets that were rendered either with or without a cast shadow along the ground. Our findings suggest that currently available video see-through displays may induce more distance underestimation than their optical see-through counterparts. We also find that the vertical position of an object and the presence of a cast shadow influence depth perception.

Depth Perception in Augmented Reality: The Effects of Display, Shadow, and Position最先出现在Nweon Paper。

Teams: Nanjing University of Aeronautics and Astronautics；UNSW

Writers: Xiong Lu; Yuxing Yan; Beibei Qi; Huang Qian; Junbin Sun; Aaron Quigley

PDF: Contactless Haptic Display Through Magnetic Field Control

Abstract

Haptic rendering enables people to touch, perceive, and manipulate virtual objects in a virtual environment. Using six cascaded identical hollow disk electromagnets and a small permanent magnet attached to an operator’s finger, this paper proposes and develops an untethered haptic interface through magnetic field control. The concentric hole inside the six cascaded electromagnets provides the workspace, where the 3D position of the permanent magnet is tracked with a Microsoft Kinect sensor. The driving currents of six cascaded electromagnets are calculated in real-time for generating the desired magnetic force. Offline data from an FEA (finite element analysis) based simulation, determines the relationship between the magnetic force, the driving currents, and the position of the permanent magnet. A set of experiments including the virtual object recognition experiment, the virtual surface identification experiment, and the user perception evaluation experiment were conducted to demonstrate the proposed system, where Microsoft HoloLens holographic glasses are used for visual rendering. The proposed magnetic haptic display leads to an untethered and non-contact interface for natural haptic rendering applications, which overcomes the constraints of mechanical linkages in tool-based traditional haptic devices.

Contactless Haptic Display Through Magnetic Field Control最先出现在Nweon Paper。

Teams: Rice University

Writers: Daniel Gorczynski,Lydia Beaudrot

PDF: Measuring understorey vegetation structure using a novel mixed-reality device

Abstract

Most ecological studies of vegetation structure have relied on manual field measurements that are labour-intensive and time-consuming. Many current alternatives to classical measurements are expensive or difficult to transport to field settings.
Here we evaluated a new method for measuring understorey vegetation with a novel mixed-reality, remote sensing device, the Microsoft HoloLens. We developed a vegetation sensing application called VegSense that allows the HoloLens user to control the device’s environmental scanners to measure understorey vegetation. Using VegSense, we tested the ability of the Microsoft HoloLens relative to classical field measurements to (a) detect trees and saplings, (b) measure diameter at breast height (DBH), (c) detect individual understorey vegetation structures and (d) estimate understorey vegetation complexity replicating the rod-transect method.
We found that VegSense performed well in detecting and measuring trees with a DBH of 17 cm or more and estimating vegetation complexity and performed moderately at detecting understorey vegetation.
Our results indicate that the HoloLens is a suitable alternative for multiple classical field measurements of understorey vegetation. This method costs much less than typical terrestrial LiDAR systems, and can facilitate efficient, high-quality environmental data collection. Further software development has the potential to reveal additional ways in which this device can be harnessed for applications to ecology and evolution.

Measuring understorey vegetation structure using a novel mixed-reality device最先出现在Nweon Paper。

Teams: University of Maryland;Microsoft Research

Writers: Shoken Kaneko; Hannes Gamper

PDF: A Fast Forest Reverberator Using Single Scattering Cylinders

Abstract

Simulating forest acoustics has important applications for rendering forest sound scenes in mixed and virtual reality, developing wildlife monitoring systems that use microphone arrays distributed in a forest, or as an artistic effect. Previously proposed methods for forest impulse response (IR) synthesis are limited to small or sparse forests because of their cubic asymptotic complexity with respect to the number of trees. Here we propose a simple and efficient parametric forest IR generation algorithm that relies on a multitude of single scattering cylinders to approximate scattering caused by tree trunks. The proposed method was compared to measured forest IRs in terms of the IR echo density, energy decay, reverberation time (T60), and clarity (C50). Experimental results indicate that the proposed algorithm generates forest reverb with acoustic characteristics similar to real forest IRs at a low computational cost.

A Fast Forest Reverberator Using Single Scattering Cylinders最先出现在Nweon Paper。

Teams: Shanghai Branch of National Computer network Emergency Response technical Team

Writers: Xindan Wang; Qu Chen; Zhi Li

PDF: A 3D Reconstruction Method for Augmented Reality Sandbox Based on Depth Sensor

Abstract

This paper builds an Augmented Reality Sandbox (AR Sandbox) system based on augmented reality technology, and performs a 3D reconstruction for the sandbox terrain using the depth sensor Microsoft Kinect in the AR Sandbox, as an entry point to pave the way for later development of related metaverse applications, such as the metaverse architecting and visual interactive modeling. The innovation of this paper is that for the AR Sandbox scene, a 3D reconstruction method based on depth sensor is proposed, which can automatically cut off the edge of the sandbox table in Kinect field of view, and accurately and completely reconstruct the sandbox terrain in Matlab.

A 3D Reconstruction Method for Augmented Reality Sandbox Based on Depth Sensor最先出现在Nweon Paper。

Teams: Bangladesh University of Engineering and Technology

Writers: Md Mushfiqur Rahman; S. M. Mahbubur Rahman

PDF: Synthesis of User-Level 3D Head Avatar from Smartphone Video for Virtual Reality

Abstract

The purpose of an avatar is to intensify the sense of reality in a simulated environment such as a virtual reality system. This is done by creating a replica of a real person in virtual reality, as intricately as possible. Implementation of 3D scanners (i.e., Microsoft Kinect) produced vast opportunities for creating 3D models of celebrities and other people who can afford them. These humanoid 3D models as well as fictional human avatars opened the door of perceiving a more realistic 3D system than before. However, a simplified approach of fully rigged 3D head avatar synthesis may help to expand the application of 3D avatars allowing anyone to feel a personalized experience of their own avatar. In this work, we propose a system of 3D head avatar synthesis at the user-end, using an easily accessible and low-cost sensor (smartphone camera). Our proposed methodology consists of data acquisition of the concerned person as video, feature extraction and matching, image registration, formation of structure form motion and post-processing. Without requiring any training data, our method provides a satisfactory performance based on the experimental results. Additionally, we release a public database for avatar synthesis from smartphone video and introduce a set of attributes to evaluate the synthesized 3D avatars.

Synthesis of User-Level 3D Head Avatar from Smartphone Video for Virtual Reality最先出现在Nweon Paper。

Teams: Coburg University of Applied Sciences；Microsoft Research；University of Cambridge；University of Primorska

Writers: Verena Biener, Snehanjali Kalamkar, Negar Nouri, Eyal Ofek, Michel Pahud, John J. Dudley, Jinghui Hu, Per Ola Kristensson, Maheshya Weerasinghe, Klen Čopič Pucihar, Matjaž Kljun, Stephan Streuber, Jens Grubert

PDF: Quantifying the Effects of Working in VR for One Week

Abstract

Virtual Reality (VR) provides new possibilities for modern knowledge work. However, the potential advantages of virtual work environments can only be used if it is feasible to work in them for an extended period of time. Until now, there are limited studies of long-term effects when working in VR. This paper addresses the need for understanding such long-term effects. Specifically, we report on a comparative study (n=16), in which participants were working in VR for an entire week – for five days, eight hours each day – as well as in a baseline physical desktop environment. This study aims to quantify the effects of exchanging a desktop-based work environment with a VR-based environment. Hence, during this study, we do not present the participants with the best possible VR system but rather a setup delivering a comparable experience to working in the physical desktop environment. The study reveals that, as expected, VR results in significantly worse ratings across most measures. Among other results, we found concerning levels of simulator sickness, below average usability ratings and two participants dropped out on the first day using VR, due to migraine, nausea and anxiety. Nevertheless, there is some indication that participants gradually overcame negative first impressions and initial discomfort. Overall, this study helps lay the groundwork for subsequent research, by clearly highlighting current shortcomings and identifying opportunities for improving the experience of working in VR.

Quantifying the Effects of Working in VR for One Week最先出现在Nweon Paper。

Teams: University of Science and Technology of China;Microsoft Research Asia;Imperial College London;Microsoft Azure Speech;University of Surrey;South China University of Technology

Writers: Yichong Leng, Zehua Chen, Junliang Guo, Haohe Liu, Jiawei Chen, Xu Tan, Danilo Mandic, Lei He, Xiang-Yang Li, Tao Qin, Sheng Zhao, Tie-Yan Liu

PDF: BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis

Abstract

Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective of two-stage synthesis with advanced generative models (i.e., the diffusion models),the proposed BinauralGrad is able to generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics (Wave L2: 0.128 vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples are available online.

BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis最先出现在Nweon Paper。

Teams: Microsoft Research

Writers: Balasaravanan Thoravi Kumaravel；Andrew D Wilson

PDF: DreamStream: Immersive and Interactive Spectating in VR

Abstract

Today spectating and streaming virtual reality (VR) activities typically involves spectators viewing a 2D stream of the VR user’s view. Streaming 2D videos of the game play is popular and well-supported by platforms such as Twitch. However, the generic streaming of full 3D representations is less explored. Thus, while the VR player’s experience may be fully immersive, spectators are limited to 2D videos. This asymmetry lessens the overall experience for spectators, who themselves may be eager to spectate in VR. DreamStream puts viewers in the virtual environment of the VR application, allowing them to look “over the shoulder” of the VR player. Spectators can view streamed VR content immersively in 3D, independently explore the VR scene beyond what the VR player sees and ultimately cohabit the virtual environment alongside the VR player. For the VR player, DreamStream provides a spatial awareness of all their spectators. DreamStream retrofits and works with existing VR applications. We discuss the design and implementation of DreamStream, and carry out three qualitative informal evaluations. These evaluations shed light on the strengths and weakness of using DreamStream for the purpose of interactive spectating. Our participants found that DreamStream’s VR viewer interface offered increased immersion, and made it easier to communicate and interact with the VR player.

DreamStream: Immersive and Interactive Spectating in VR最先出现在Nweon Paper。

Teams: Imperial College London

Writers: Xue Hu; Ferdinando Rodriguez y Baena; Fabrizio Cutolo

PDF: Rotation-constrained optical see-through headset calibration with bare-hand alignment

Abstract

The inaccessibility of user-perceived reality remains an open issue in pursuing the accurate calibration of optical see-through (OST) head-mounted displays (HMDs). Manual user alignment is usually required to collect a set of virtual-to-real correspondences, so that a default or an offline display calibration can be updated to account for the user’s eye position(s). Current alignment-based calibration procedures usually require point-wise alignments between rendered image point(s) and associated physical landmark(s) of a target calibration tool. As each alignment can only provide one or a few correspondences, repeated alignments are required to ensure calibration quality. This work presents an accurate and tool-less online OST calibration method to update an offline-calibrated eye-display model. The user’s bare hand is markerlessly tracked by a commercial RGBD camera anchored to the OST headset to generate a user-specific cursor for correspondence collection. The required alignment is object-wise, and can provide thousands of unordered corresponding points in tracked space. The collected correspondences are registered by a proposed rotation-constrained iterative closest point (rcICP) method to optimise the viewpoint-related calibration parameters. We implemented such a method for the Microsoft HoloLens 1. The resiliency of the proposed procedure to noisy data was evaluated through simulated tests and real experiments performed with an eye-replacement camera. According to the simulation test, the rcICP registration is robust against possible user-induced rotational misalignment. With a single alignment, our method achieves 8.81 arcmin (1.37 mm) positional error and 1. 76° rotational error by camera-based tests in the arm-reach distance, and 10.79 arcmin (7.71 pixels) reprojection error by user tests.

Rotation-constrained optical see-through headset calibration with bare-hand alignment最先出现在Nweon Paper。

Teams: Mississippi State University；University of Nebraska–Lincoln

Writers: Farzana Alam Khan; Veera Venkata Ram Murali Krishna Rao Muvva; Dennis Wu; Mohammed Safayet Arefin; Nate Phillips; J. Edward Swan

PDF: Measuring the Perceived Three-Dimensional Location of Virtual Objects in Optical See-Through Augmented Reality

Abstract

For optical see-through augmented reality (AR), a new method for measuring the perceived three-dimensional location of virtual objects is presented, where participants verbally report a virtual object’s location relative to both a vertical and horizontal grid. The method is tested with a small (1.95 × 1.95 × 1.95 cm) virtual object at distances of 50 to 80 cm, viewed through a Microsoft HoloLens 1 st generation AR display. Two experiments examine two different virtual object designs, whether turning in a circle between reported object locations disrupts HoloLens tracking, and whether accuracy errors, including a rightward bias and underestimated depth, might be due to systematic errors that are restricted to a particular display. Turning in a circle did not disrupt HoloLens tracking, and testing with a second display did not suggest systematic errors restricted to a particular display. Instead, the experiments are consistent with the hypothesis that, when looking downwards at a horizontal plane, HoloLens 1 st generation displays exhibit a systematic rightward perceptual bias. Precision analysis suggests that the method could measure the perceived location of a virtual object within an accuracy of less than 1 mm.

Measuring the Perceived Three-Dimensional Location of Virtual Objects in Optical See-Through Augmented Reality最先出现在Nweon Paper。

Teams: University of Utah；Vanderbilt University

Writers: Holly C. Gagnon; Dun Na; Keith Heiner; Jeanine Stefanucci; Sarah Creem-Regehr; Bobby Bodenheimer

PDF: Walking Through Walls: The Effect of Collision-Based Feedback on Affordance Judgments in Augmented Reality

Abstract

Feedback about actions in augmented reality (AR) is limited and can be ambi due to the nature of interacting with virtual objects. AR devices also have a restricted field of view (FOV), limiting the amount of available visual information that can be used to perform an action or provide feedback during or after an action. We used the Microsoft HoloLens 1 to investigate whether perceptual-motor, collision-based outcome feedback calibrates judgments of whether one can pass through an aperture in AR. Additionally, we manipulated the amount of information available within the FOV by having participants view the aperture at two different distances. Feedback calibrated passing-through judgments at both distances but resulted in an overestimation of the just-passable aperture width. Moreover, the far viewing condition had more overestimation of just-passable aperture width than the near viewing condition.

Walking Through Walls: The Effect of Collision-Based Feedback on Affordance Judgments in Augmented Reality最先出现在Nweon Paper。

Teams: Microsoft Mixed Reality and AI Lab；ETH Zurich

Writers: Jeffrey Delmerico, Roi Poranne, Federica Bogo, Helen Oleynikova, Eric Vollenweider, Stelian Coros, Juan Nieto, Marc Pollefeys

PDF: Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together

Abstract

Spatial computing – the ability of devices to be aware of their surroundings and to represent this digitally – offers novel capabilities in human-robot interaction. In particular, the combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning, which offers exciting new possibilities for collaboration between humans and robots. This paper presents several human-robot systems that utilize these capabilities to enable novel robot use cases: mission planning for inspection, gesture-based control, and immersive teleoperation. These works demonstrate the power of mixed reality as a tool for human-robot interaction, and the potential of spatial computing and mixed reality to drive the future of human-robot interaction.

Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together最先出现在Nweon Paper。

Teams: Coburg University of Applied Sciences and Arts；University of Primorska；Microsoft Research；University of Cambridge

Writers: Daniel Schneider, Verena Biener, Alexander Otte, Travis Gesslein, Philipp Gagel, Cuauhtli Campos, Klen Čopič Pucihar, Matjaž Kljun, Eyal Ofek, Michel Pahud, Per Ola Kristensson, Jens Grubert

PDF: Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays

Abstract

An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual reality (AR/VR) are capable of finger and hand tracking. We report on the accuracy of off-the-shelf VR and AR HMDs when used for touch-based tasks such as pointing or drawing. Specifically, we report on the finger tracking accuracy of the VR head-mounted displays Oculus Quest, Vive Pro and the Leap Motion controller, when attached to a VR HMD, as well as the finger tracking accuracy of the AR head-mounted displays Microsoft HoloLens 2 and Magic Leap. We present the results of two experiments in which we compare the accuracy for absolute and relative pointing tasks using both human participants and a robot. The results suggest that HTC Vive has a lower spatial accuracy than the Oculus Quest and Leap Motion and that the Microsoft HoloLens 2 provides higher spatial accuracy than Magic Leap One. These findings can serve as decision support for researchers and practitioners in choosing which systems to use in the future.

Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays最先出现在Nweon Paper。

Teams: Imperial College London;University of Pisa

Writers: Xue Hu, Ferdinando Rodriguez y Baena, Fabrizio Cutolo

PDF: Rotation-constrained optical see-through headset calibration withbare-hand alignment

Abstract

The inaccessibility of user-perceived reality remains an open issue in pursuing the accurate calibration of optical see-through (OST) head-mounted displays (HMDs). Manual user alignment is usually required to collect a set of virtual-to-real correspondences, so that a default or an offline display calibration can be updated to account for the user’s eye position(s). Current alignment-based calibration procedures usually require point-wise alignments between rendered image point(s) and associated physical landmark(s) of a target calibration tool. As each alignment can only provide one or a few correspondences, repeated alignments are required to ensure calibration quality.
This work presents an accurate and tool-less online OST calibration method to update an offline-calibrated eye-display model. The user’s bare hand is markerlessly tracked by a commercial RGBD camera anchored to the OST headset to generate a user-specific cursor for correspondence collection. The required alignment is object-wise, and can provide thousands of unordered corresponding points in tracked space. The collected correspondences are registered by a proposed rotation-constrained iterative closest point (rcICP) method to optimise the viewpoint-related calibration parameters. We implemented such a method for the Microsoft HoloLens 1. The resiliency of the proposed procedure to noisy data was evaluated through simulated tests and real experiments performed with an eye-replacement camera. According to the simulation test, the rcICP registration is robust against possible user-induced rotational misalignment. With a single alignment, our method achieves 8.81 arcmin (1.37 mm) positional error and 1.76 degree rotational error by camera-based tests in the arm-reach distance, and 10.79 arcmin (7.71 pixels) reprojection error by user tests.

Rotation-constrained optical see-through headset calibration withbare-hand alignment最先出现在Nweon Paper。

Teams: NICT and Osaka University

Writers: Michal Joachimczak; Juan Liu; Hiroshi Ando

PDF: Downsizing: The Effect of Mixed-Reality Person Representations on Stress and Presence in Telecommunication

Abstract

We study how mixed-reality (MR) telepresence can enhance long-distance human interaction and how altering three-dimensional (3D) representations of a remote person can be used to modulate stress and anxiety during social interactions. To do so, we developed an MR telepresence system employing commodity depth sensors and Microsoft’s Hololens. A textured, polygonal 3D model of a person was reconstructed in real time and transmitted over network for rendering in remote location using Hololens. In this pilot study, we used mock job interview paradigm to induce stress in human-subjects interacting with an interviewer presented as an MR hologram. Participants were exposed to three different types of real-time reconstructed virtual holograms of the interviewer, a natural-sized 3D reconstruction (NR), a miniature 3D reconstruction (SR) and a 2D-display representation (LCD). Participants reported their subjective experience through questionnaires, while their biophysical responses were recorded. We found that the size of 3D representation of a remote interviewer had a significant effect on participants’ stress levels and their sense of presence. NR condition induced more stress and presence than SR condition and was significantly different from LCD condition.

Downsizing: The Effect of Mixed-Reality Person Representations on Stress and Presence in Telecommunication最先出现在Nweon Paper。

Teams: The University of Electro-Communications

Writers: Jiazhen Guo; Peng Chen; Yinlai Jiang; Hiroshi Yokoi; Shunta Togo

PDF: Real-time Object Detection with Deep Learning for Robot Vision on Mixed Reality Device

Abstract

Mixed reality device sensing capabilities are valuable for robots, for example, the inertial measurement unit (IMU) sensor and time-of-flight (TOF) depth sensor can support the robot in navigating its environment. This paper demonstrates a deep learning (YOLO model) background, realtime object detection system implemented on mixed reality device. The goal of the system is to create a real-time communication system between HoloLens and Ubuntu systems to enable real-time object detection using the YOLO model. The experimental results show that the proposed method has a fast speed to achieve real-time object detection using HoloLens. This enables Microsoft HoloLens as a device for robot vision. To enhance human-robot interaction, we will apply it to a wearable robot arm system to automatically grasp objects in the future.

Real-time Object Detection with Deep Learning for Robot Vision on Mixed Reality Device最先出现在Nweon Paper。

Teams: Microsoft Research

Writers: Dhruv Jain Sasa Junuzovic Eyal Ofek Mike Sinclair John Porter Chris Yoon Swetha Machanavajhala Meredith Ringel Morris

PDF: A Taxonomy of Sounds in Virtual Reality

Abstract

Virtual reality (VR) leverages human sight, hearing and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, information conveyed through sound may not be accessible. To help with future design of accessible VR sound representations for DHH users, this paper contributes a consistent language and structure for representing sounds in VR. Using two studies, we report on the design and evaluation of a novel taxonomy for VR sounds. Study 1 included interviews with 10 VR sound designers to develop our taxonomy along two dimensions: sound source and intent. To evaluate this taxonomy, we conducted another study (Study 2) where eight HCI researchers used our taxonomy to document sounds in 33 VR apps. We found that our taxonomy was able to successfully categorize nearly all sounds (265/267) in these apps. We also uncovered additional insights for designing accessible visual and haptic-based sound substitutes for DHH users.

A Taxonomy of Sounds in Virtual Reality最先出现在Nweon Paper。

Teams: Inria；INSA Rennes

Writers: Yoren Gaffary; Benoît Le Gouis; Maud Marchal; Ferran Argelaguet; Bruno Arnaldi; Anatole Lécuyer

PDF: AR Feels “Softer” than VR: Haptic Perception of Stiffness in Augmented versus Virtual Reality

Abstract

Does it feel the same when you touch an object in Augmented Reality (AR) or in Virtual Reality (VR)? In this paper we study and compare the haptic perception of stiffness of a virtual object in two situations: (1) a purely virtual environment versus (2) a real and augmented environment. We have designed an experimental setup based on a Microsoft HoloLens and a haptic force-feedback device, enabling to press a virtual piston, and compare its stiffness successively in either Augmented Reality (the virtual piston is surrounded by several real objects all located inside a cardboard box) or in Virtual Reality (the same virtual piston is displayed in a fully virtual scene composed of the same other objects). We have conducted a psychophysical experiment with 12 participants. Our results show a surprising bias in perception between the two conditions. The virtual piston is on average perceived stiffer in the VR condition compared to the AR condition. For instance, when the piston had the same stiffness in AR and VR, participants would select the VR piston as the stiffer one in 60% of cases. This suggests a psychological effect as if objects in AR would feel ”softer” than in pure VR. Taken together, our results open new perspectives on perception in AR versus VR, and pave the way to future studies aiming at characterizing potential perceptual biases.

AR Feels “Softer” than VR: Haptic Perception of Stiffness in Augmented versus Virtual Reality最先出现在Nweon Paper。