Nweon Paper https://paper.nweon.com 映维网,影响力虚拟现实(VR)、增强现实(AR)产业信息数据平台 Thu, 13 May 2021 13:07:25 +0000 zh-CN hourly 1 https://wordpress.org/?v=4.8.17 https://paper.nweon.com/wp-content/uploads/2021/04/nweon-icon.png Nweon Paper https://paper.nweon.com 32 32 Enhancing Photorealism Enhancement https://paper.nweon.com/9890 Thu, 13 May 2021 13:07:25 +0000 https://paper.nweon.com/9890 PubDate: May 2021

Enhancing Photorealism Enhancement最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: Intel Intelligent Systems Lab

Writers: Stephan R. Richter, Hassan Abu AlHaija, Vladlen Koltun

PDF: Enhancing Photorealism Enhancement

Abstract

We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines.

Enhancing Photorealism Enhancement最先出现在Nweon Paper

]]>
Towards usable underwater virtual reality systems https://paper.nweon.com/9888 Thu, 13 May 2021 07:37:42 +0000 https://paper.nweon.com/9888 PubDate: April 2017

Towards usable underwater virtual reality systems最先出现在Nweon Paper

]]>
PubDate: April 2017

Teams: University of Texas at San Antonio;Kennesaw State University

Writers: Raphael Costa; Rongkai Guo; John Quarles

PDF: Towards usable underwater virtual reality systems

Abstract

The objective of this research is to compare the effectiveness of different tracking devices underwater. There have been few works in aquatic virtual reality (VR) – i.e., VR systems that can be used in a real underwater environment. Moreover, the works that have been done have noted limitations on tracking accuracy. Our initial test results suggest that inertial measurement units work well underwater for orientation tracking but a different approach is needed for position tracking. Towards this goal, we have waterproofed and evaluated several consumer tracking systems intended for gaming to determine the most effective approaches. First, we informally tested infrared systems and fiducial marker based systems, which demonstrated significant limitations of optical approaches. Next, we quantitatively compared inertial measurement units (IMU) and a magnetic tracking system both above water (as a baseline) and underwater. By comparing the devices rotation data, we have discovered that the magnetic tracking system implemented by the Razer Hydra is more accurate underwater as compared to a phone-based IMU. This suggests that magnetic tracking systems should be further explored for underwater VR applications.

Towards usable underwater virtual reality systems最先出现在Nweon Paper

]]>
A Light-Field Journey to Virtual Reality https://paper.nweon.com/9886 Thu, 13 May 2021 06:25:36 +0000 https://paper.nweon.com/9886 PubDate: May 201

A Light-Field Journey to Virtual Reality最先出现在Nweon Paper

]]>
PubDate: May 201

Teams: University of Delaware

Writers: Jingyi Yu

PDF: A Light-Field Journey to Virtual Reality

Abstract

Producing ultra-high-quality content is the next grand challenge for the VR industry. Creating a virtual reality that even the human eye cannot distinguish from the real world will require light-field technology–3D imaging technology that captures the light rays people perceive from different locations and directions. When combined with computer vision and machine learning, light-field technology provides a viable path for producing low-cost, high-quality VR content.

A Light-Field Journey to Virtual Reality最先出现在Nweon Paper

]]>
Coordinating attention and cooperation in multi-user virtual reality narratives https://paper.nweon.com/9884 Thu, 13 May 2021 05:37:54 +0000 https://paper.nweon.com/9884 PubDate: April 2017

Coordinating attention and cooperation in multi-user virtual reality narratives最先出现在Nweon Paper

]]>
PubDate: April 2017

Teams: Texas A&M University

Writers: Cullen Brown; Ghanshyam Bhutra; Mohamed Suhail; Qinghong Xu; Eric D. Ragan

PDF: Coordinating attention and cooperation in multi-user virtual reality narratives

Abstract

Limited research has been performed attempting to handle multiuser storytelling environments in virtual reality. As such, a number of questions about handling story progression and maintaining user presence in a multi-user virtual environment have yet to be answered. We created a multi-user virtual reality story experience in which we intend to study a set of guided camera techniques and a set of gaze distractor techniques to determine how best to attract disparate users to the same story. Additionally, we describe our preliminary work and plans to study the effectiveness of these techniques, their effect on user presence, and generally how multiple users feel their actions affect the outcome of a story.

Coordinating attention and cooperation in multi-user virtual reality narratives最先出现在Nweon Paper

]]>
Optical FIow, Perturbation Velocities and Postural Response In Virtual Reality https://paper.nweon.com/9880 Thu, 13 May 2021 04:17:19 +0000 https://paper.nweon.com/9880 PubDate: May 2020

Optical FIow, Perturbation Velocities and Postural Response In Virtual Reality最先出现在Nweon Paper

]]>
PubDate: May 2020

Teams: University of Florida;Montclair State University

Writers: Markus Santoso; David Phillips

PDF: Optical FIow, Perturbation Velocities and Postural Response In Virtual Reality

Abstract

The purpose of this study was to investigate the effect of optical flow velocity in a virtual reality (VR) environment on user’s postural control. We hypothesized that the velocity of the optical flow will perturb user’s balance indicated by an increase center of pressure (COP) distance travelled and speed. Seventeen young, healthy participants were tested in one-foot support stances. Our study showed the visual perturbations increased COP distance in one-foot support and the slowest perturbation velocity induced the highest response. For VR communities, developers could use this information to raise their awareness that any sudden shift in the virtual environment at any velocity could reduce a user’s postural stability and place them at risk of falling, particularly at slower perturbation velocities.

Optical FIow, Perturbation Velocities and Postural Response In Virtual Reality最先出现在Nweon Paper

]]>
Gaze-Pinch Menu: Performing Multiple Interactions Concurrently in Mixed Reality https://paper.nweon.com/9878 Thu, 13 May 2021 02:16:29 +0000 https://paper.nweon.com/9878 PubDate: May 2021

Gaze-Pinch Menu: Performing Multiple Interactions Concurrently in Mixed Reality最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: Beihang University

Writers: Lu Lu; Pengshuai Duan; Xukun Shen; Shijin Zhang; Huiyan Feng; Yong Flu

PDF: Gaze-Pinch Menu: Performing Multiple Interactions Concurrently in Mixed Reality

Abstract

Performing an interaction using gaze and pinch has been certified as an efficient interactive method in Mixed Reality, for such techniques can provide users concise and natural experiences. However, executing a task with individual interactions gradually is inefficient in some application scenarios. In this paper, we propose the Hand-Pinch Menu, which core concept is to reduce unnecessary operations by combining several interactions. Users can continuously perform multiple interactions on a selected object concurrently without changing gestures by using this technique. The user study results show that our Gaze-Pinch Menu can improve operational efficiency effectively.

Gaze-Pinch Menu: Performing Multiple Interactions Concurrently in Mixed Reality最先出现在Nweon Paper

]]>
GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation https://paper.nweon.com/9876 Wed, 12 May 2021 06:04:20 +0000 https://paper.nweon.com/9876 PubDate: May 2021

GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: KAIST;Microsoft Research

Writers: Neung Ryu Hye-Young Jo Michel Pahud Mike Sinclair Andrea Bianchi

PDF: GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation

Abstract

Virtual Reality experiences, such as games and simulations, typically support the usage of bimanual controllers to interact with virtual objects. To recreate the haptic sensation of holding objects of various shapes and behaviors with both hands, previous researchers have used mechanical linkages between the controllers that render adjustable stiffness. However, the linkage cannot quickly adapt to simulate dynamic objects, nor it can be removed to support free movements. This paper introduces GamesBond, a pair of 4-DoF controllers without physical linkage but capable to create the illusion of being connected as a single device, forming a virtual bond. The two controllers work together by dynamically displaying and physically rendering deformations of hand grips, and so allowing users to perceive a single connected object between the hands, such as a jumping rope. With a user study and various applications we show that GamesBond increases the realism, immersion, and enjoyment of bimanual interaction.

GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip Deformation最先出现在Nweon Paper

]]>
Deep Relightable Appearance Models for Animatable Faces https://paper.nweon.com/9874 Wed, 12 May 2021 05:37:39 +0000 https://paper.nweon.com/9874 PubDate: August 9, 2021

Deep Relightable Appearance Models for Animatable Faces最先出现在Nweon Paper

]]>
PubDate: August 9, 2021

Teams: University of California;Facebook Reality Labs

Writers: Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, Jason Saragih

PDF: Deep Relightable Appearance Models for Animatable Faces

Abstract

We present a method for building high-fidelity animatable 3D face models that can be posed and rendered with novel lighting environments in real-time. Our main insight is that relightable models trained to produce an image lit from a single light direction can generalize to natural illumination conditions but are computationally expensive to render. On the other hand, efficient, high-fidelity face models trained with point-light data do not generalize to novel lighting conditions. We leverage the strengths of each of these two approaches. We first train an expensive but generalizable model on point-light illuminations, and use it to generate a training set of high-quality synthetic face images under natural illumination conditions. We then train an efficient model on this augmented dataset, reducing the generalization ability requirements. As the efficacy of this approach hinges on the quality of the synthetic data we can generate, we present a study of lighting pattern combinations for dynamic captures and evaluate their suitability for learning generalizable relightable models. Towards achieving the best possible quality, we present a novel approach for generating dynamic relightable faces that exceeds state-of-the-art performance. Our method is capable of capturing subtle lighting effects and can even generate compelling near-field relighting despite being trained exclusively with far-field lighting data. Finally, we motivate the utility of our model by animating it with images captured from VR-headset mounted cameras, demonstrating the first system for face-driven interactions in VR that uses a photorealistic relightable face model.

Deep Relightable Appearance Models for Animatable Faces最先出现在Nweon Paper

]]>
Mixture of Volumetric Primitives for Efficient Neural Rendering https://paper.nweon.com/9872 Wed, 12 May 2021 04:34:25 +0000 https://paper.nweon.com/9872 PubDate: August 9, 2021

Mixture of Volumetric Primitives for Efficient Neural Rendering最先出现在Nweon Paper

]]>
PubDate: August 9, 2021

Teams: Facebook Reality Labs

Writers: Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, Jason Saragih

PDF: Mixture of Volumetric Primitives for Efficient Neural Rendering

Abstract

Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with our work. Triangle meshes have difficulty modeling thin structures like hair, volumetric representations like Neural Volumes are too low-resolution given a reasonable memory budget, and high-resolution implicit representations like Neural Radiance Fields are too slow for use in real-time applications. We present Mixture of Volumetric Primitives (MVP), a representation for rendering dynamic 3D content that combines the completeness of volumetric representations with the efficiency of primitive-based rendering, e.g., point-based or mesh-based methods. Our approach achieves this by leveraging spatially shared computation with a convolutional architecture and by minimizing computation in empty regions of space with volumetric primitives that can move to cover only occupied regions. Our parameterization supports the integration of correspondence and tracking constraints, while being robust to areas where classical tracking fails, such as around thin or translucent structures and areas with large topological variability. MVP is a hybrid that generalizes both volumetric and primitive-based representations. Through a series of extensive experiments we demonstrate that it inherits the strengths of each, while avoiding many of their limitations. We also compare our approach to several state-of-the-art methods and demonstrate that MVP produces superior results in terms of quality and runtime performance.

Mixture of Volumetric Primitives for Efficient Neural Rendering最先出现在Nweon Paper

]]>
Multi-Channel Speech Enhancement Using Graph Neural Networks https://paper.nweon.com/9870 Wed, 12 May 2021 03:01:39 +0000 https://paper.nweon.com/9870 PubDate: June 6, 2021

Multi-Channel Speech Enhancement Using Graph Neural Networks最先出现在Nweon Paper

]]>
PubDate: June 6, 2021

Teams: Facebook Reality Labs Research

Writers: Panagiotis Tzirakis, Anurag Kumar, Jacob Donley

PDF: Multi-Channel Speech Enhancement Using Graph Neural Networks

Abstract

Multi-channel speech enhancement aims to extract clean speech from a noisy mixture using signals captured from multiple microphones. Recently proposed methods tackle this problem by incorporating deep neural network models with spatial filtering techniques such as the minimum variance distortionless response (MVDR) beamformer. In this paper, we introduce a different research direction by viewing each audio channel as a node lying in a non-Euclidean space and, specifically, a graph. This formulation allows us to apply graph neural networks (GNN) to find spatial correlations among the different channels (nodes). We utilize graph convolution networks (GCN) by incorporating them in the embedding space of a U-Net architecture. We use LibriSpeech dataset and simulate room acoustics data to extensively experiment with our approach using different array types, and number of microphones. Results indicate the superiority of our approach when compared to prior state-of-the-art method.

Multi-Channel Speech Enhancement Using Graph Neural Networks最先出现在Nweon Paper

]]>
StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR https://paper.nweon.com/9868 Wed, 12 May 2021 02:19:29 +0000 https://paper.nweon.com/9868 PubDate: May 8, 2021

StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR最先出现在Nweon Paper

]]>
PubDate: May 8, 2021

Teams: Chatham Labs;Canada and University of Toronto

Writers: Sunggeun Ahn, Stephanie Santosa, Mark Parent, Daniel Wigdor, Tovi Grossman, Marcello Giordano

PDF: StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR

Abstract

This work explores the design of marking menus for gaze-based AR/VR menu selection by expert and novice users. It first identifies and explains the challenges inherent in ocular motor control and current eye tracking hardware, including overshooting, incorrect selections, and false activations. Through three empirical studies, we optimized and validated design parameters to mitigate these errors while reducing completion time, task load, and eye fatigue. Based on the findings from these studies, we derived a set of design guidelines to support gaze-based marking menus in AR/VR. To overcome the overshoot errors found with eye-based expert marking menu behavior, we developed StickyPie, a marking menu technique that enables scale-independent marking input by estimating saccade landing positions. An evaluation of StickyPie revealed that StickyPie was easier to learn than the traditional technique (i.e., RegularPie) and was 10% more efficient after 3 sessions.

StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR最先出现在Nweon Paper

]]>
A Multichannel Pneumatic Analog Control System for Haptic Displays https://paper.nweon.com/9866 Wed, 12 May 2021 01:43:39 +0000 https://paper.nweon.com/9866 PubDate: May 9, 2021

A Multichannel Pneumatic Analog Control System for Haptic Displays最先出现在Nweon Paper

]]>
PubDate: May 9, 2021

Teams: Facebook Reality Labs Research

Writers: Benjamin Stephens-Fripp, Ali Israr, Carine Rognon

PDF: A Multichannel Pneumatic Analog Control System for Haptic Displays

Abstract

Soft robotics use in haptic devices continues to grow, with pneumatic control being a common actuation source. Typical control systems used, however, rely on a digital on/of control allowing only inflation/deflation at a set rate. This limits the degrees of freedom available when designing haptic experiences. We present an alternative system to allow the use of analog control of the pneumatic waveform profiles to design and experiment with haptic devices, and to determine the optimum wave profile for the desired experience. Using a combination of of-the-shelf components and a user interface, our system allows for rapid experimentation with various pressure levels, and the ability to control waveform profiles in a common format such as attack-sustain-release. In this paper, we demonstrate that by altering the attack and release profiles we can create a more pleasant pulsing sensation on the wrist, and a more continuous sensation for communicating movement around the wrist.

A Multichannel Pneumatic Analog Control System for Haptic Displays最先出现在Nweon Paper

]]>
Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality https://paper.nweon.com/9864 Tue, 11 May 2021 07:46:56 +0000 https://paper.nweon.com/9864 PubDate: May 8, 2021

Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality最先出现在Nweon Paper

]]>
PubDate: May 8, 2021

Teams: University of Cambridge;Facebook Reality Labs

Writers: Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, Per Ola Kristensson

PDF: Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality

Abstract

Typing with ten fingers on a virtual keyboard in virtual or augmented reality exposes a challenging input interpretation problem. There are many sources of noise in this interaction context and these exacerbate the challenge of accurately translating human actions into text. A particularly challenging input noise source arises from the physiology of the hand. Intentional finger movements can produce unintentional coactivations in other fingers. On a physical keyboard, the resistance of the keys alleviates this issue. On a virtual keyboard, coactivations are likely to introduce spurious input events under a naive solution to input detection. In this paper we examine the features that discriminate intentional activations from coactivations. Based on this analysis, we demonstrate three alternative coactivation detection strategies with high discrimination power. Finally, we integrate coactivation detection into a probabilistic decoder and demonstrate its ability to further reduce uncorrected character error rates by approximately 10% relative and 0.9% absolute.

Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality最先出现在Nweon Paper

]]>
Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality https://paper.nweon.com/9862 Tue, 11 May 2021 07:22:26 +0000 https://paper.nweon.com/9862 PubDate: May 8, 2021

Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality最先出现在Nweon Paper

]]>
PubDate: May 8, 2021

Teams: Chatham Labs & University of Toronto;Facebook Reality Labs

Writers: Zhen Li, Joannes Chan, Joshua Walton, Hrvoje Benko, Daniel Wigdor, Michael Glueck

PDF: Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality

Abstract

In virtual reality (VR) environments, asymmetric bimanual interaction techniques can increase users’ input bandwidth by complementing their perceptual and motor systems (e.g., using the dominant hand to select 3D UI controls anchored around the nondominant arm). However, it is unclear how to optimize the layout of such 3D UI controls for near-body and mid-air interactions. We evaluate the performance and limitations of non-dominant armanchored 3D UIs in VR environments through a bimanual pointing study. Results demonstrated that targets appearing closer to the skin, located around the wrist, or placed on the medial side of the forearm could be selected more quickly than targets farther away from the skin, located around the elbow, or on the lateral side of the forearm. Based on these results, we developed Armstrong guidelines, demonstrated through a Unity plugin to enable designers to create performance-optimized arm-anchored 3D UI layouts.

Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality最先出现在Nweon Paper

]]>
Revisiting Milgram and Kishino’s Reality-Virtuality Continuum https://paper.nweon.com/9860 Tue, 11 May 2021 07:01:40 +0000 https://paper.nweon.com/9860 PubDate: March 26, 2021

Revisiting Milgram and Kishino’s Reality-Virtuality Continuum最先出现在Nweon Paper

]]>
PubDate: March 26, 2021

Teams: La Trobe University;University of North Carolina

Writers: Richard Skarbez, Missie Smith, Mary C. Whitton

PDF: Revisiting Milgram and Kishino’s Reality-Virtuality Continuum

Abstract

Since its introduction in 1994, Milgram and Kishino’s reality-virtuality (RV) continuum has been used to frame virtual and augmented reality research and development. While originally, the RV continuum and the three dimensions of the supporting taxonomy (extent of world knowledge, reproduction fidelity, and extent of presence metaphor) were intended to characterize the capabilities of visual display technology, researchers have embraced the RV continuum while largely ignoring the taxonomy. Considering the leaps in technology made over the last 25 years, revisiting the RV continuum and taxonomy is timely. In reexamining Milgram and Kishino’s ideas, we realized, first, that the RV continuum is actually discontinuous; perfect virtual reality cannot be reached. Secondly, mixed reality is broader than previously believed, and, in fact, encompasses conventional virtual reality experiences. Finally, our revised taxonomy adds coherence, accounting for the role of users, which is critical to assessing modern mixed reality experiences. The 3D space created by our taxonomy incorporates familiar constructs such as presence and immersion, and also proposes new constructs that may be important as mixed reality technology matures.

Revisiting Milgram and Kishino’s Reality-Virtuality Continuum最先出现在Nweon Paper

]]>
Heterogeneous Dataflow Accelerators for Multi-DNN Workloads https://paper.nweon.com/9858 Tue, 11 May 2021 01:17:54 +0000 https://paper.nweon.com/9858 PubDate: March 3, 2021

Heterogeneous Dataflow Accelerators for Multi-DNN Workloads最先出现在Nweon Paper

]]>
PubDate: March 3, 2021

Teams: Georgia Insitute of Technology;Facebook

Writers: Hyoukjun Kwon, Liangzhen La, Michael Pellauer, Tushar Krishna, Yu-Hsin Chen, Vikas Chandra

PDF: Heterogeneous Dataflow Accelerators for Multi-DNN Workloads

Abstract

Emerging AI-enabled applications such as augmented and virtual reality (AR/VR) leverage multiple deep neural network (DNN) models for various sub-tasks such as object detection, image segmentation, eye-tracking, speech recognition, and so on. Because of the diversity of the sub-tasks, the layers within and across the DNN models are highly heterogeneous in operation and shape. Diverse layer operations and shapes are major challenges for a fixed dataflow accelerator (FDA) that employs a fixed dataflow strategy on a single DNN accelerator substrate since each layer prefers different dataflows (computation order and parallelization) and tile sizes. Reconfigurable DNN accelerators (RDAs) have been proposed to adapt their dataflows to diverse layers to address the challenge. However, the dataflow flexibility in RDAs is enabled at the cost of expensive hardware structures (switches, interconnects, controller, etc.) and requires per-layer reconfiguration, which introduces considerable energy costs.

Alternatively, this work proposes a new class of accelerators, heterogeneous dataflow accelerators (HDAs), which deploy multiple accelerator substrates (i.e., sub-accelerators), each supporting a different dataflow. HDAs enable coarser-grained dataflow flexibility than RDAs with higher energy efficiency and lower area cost comparable to FDAs. To exploit such benefits, hardware resource partitioning across sub-accelerators and layer execution schedule need to be carefully optimized. Therefore, we also present Herald, a framework for co-optimizing hardware partitioning and layer scheduling. Using Herald on a suite of AR/VR and MLPerf workloads, we identify a promising HDA architecture, Maelstrom, which demonstrates 65.3% lower latency and 5.0% lower energy compared to the best fixed dataflow accelerators and 22.0% lower energy at the cost of 20.7% higher latency compared to a state-of-the-art reconfigurable DNN accelerator (RDA). The results suggest that HDA is an alternative class of Pareto-optimal accelerators to RDA with strength in energy, which can be a better choice than RDAs depending on the use cases.

Heterogeneous Dataflow Accelerators for Multi-DNN Workloads最先出现在Nweon Paper

]]>
NASGEM: Neural Architecture Search via Graph Embedding Method https://paper.nweon.com/9856 Tue, 11 May 2021 00:47:47 +0000 https://paper.nweon.com/9856 PubDate: February 9, 2021

NASGEM: Neural Architecture Search via Graph Embedding Method最先出现在Nweon Paper

]]>
PubDate: February 9, 2021

Teams: Duke University;Facebook;Tsinghua University;University of Nevada

Writers: Hsin-Pai Cheng, Tunhou Zhang, Yixing Zhang, Shiyu Liang, Feng Liang, Feng Yan, Meng Li, Vikas Chandra, Hai Li, Yiran Chen

PDF: NASGEM: Neural Architecture Search via Graph Embedding Method

Abstract

Neural Architecture Search (NAS) automates and prospers the design of neural networks. Estimator-based NAS has been proposed recently to model the relationship between architectures and their performance to enable scalable and flexible search. However, existing estimator-based methods encode the architecture into a latent space without considering graph similarity. Ignoring graph similarity in node-based search space may induce a large inconsistency between similar graphs and their distance in the continuous encoding space, leading to inaccurate encoding representation and/or reduced representation capacity that can yield sub-optimal search results. To preserve graph correlation information in encoding, we propose NASGEM which stands for Neural Architecture Search via Graph Embedding Method. NASGEM is driven by a novel graph embedding method equipped with similarity measures to capture the graph topology information. By precisely estimating the graph distance and using an auxiliary Weisfeiler-Lehman kernel to guide the encoding, NASGEM can utilize additional structural information to get more accurate graph representation to improve the search efficiency. GEMNet, a set of networks discovered by NASGEM, consistently outperforms networks crafted by existing search methods in classification tasks, i.e., with 0.4%-3.6% higher accuracy while having 11%- 21% fewer Multiply-Accumulates. We further transfer GEMNet for COCO object detection. In both one-stage and two- stage detectors, our GEMNet surpasses its manually-crafted and automatically-searched counterparts.

NASGEM: Neural Architecture Search via Graph Embedding Method最先出现在Nweon Paper

]]>
Thinking Outside the Lab: VR Size & Depth Perception in the Wild https://paper.nweon.com/9854 Mon, 10 May 2021 07:34:54 +0000 https://paper.nweon.com/9854 PubDate: May 2021

Thinking Outside the Lab: VR Size & Depth Perception in the Wild最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: University of Toronto

Writers: Rahul Arora, Jiannan Li, Gongyi Shi, Karan Singh

PDF: Thinking Outside the Lab: VR Size & Depth Perception in the Wild

Abstract

Size and distance perception in Virtual Reality (VR) have been widely studied, albeit in a controlled laboratory setting with a small number of participants. We describe a fully remote perceptual study with a gamified protocol to encourage participant engagement, which allowed us to quickly collect high-quality data from a large, diverse participant pool (N=60). Our study aims to understand medium-field size and egocentric distance perception in real-world usage of consumer VR devices. We utilized two perceptual matching tasks – distance bisection and size matching – at the same target distances of 1–9 metres. While the bisection protocol indicated a near-universal trend of nonlinear distance compression, the size matching estimates were more equivocal. Varying eye-height from the floor plane showed no significant effect on the judgements. We also discuss the pros and cons of a fully remote perceptual study in VR, the impact of hardware variation, and measures needed to ensure high-quality data.

Thinking Outside the Lab: VR Size & Depth Perception in the Wild最先出现在Nweon Paper

]]>
Spatial Privacy-aware VR streaming https://paper.nweon.com/9852 Mon, 10 May 2021 07:02:18 +0000 https://paper.nweon.com/9852 PubDate: Apr 2021

Spatial Privacy-aware VR streaming最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: Beihang University

Writers: Xing Wei, Chenyang Yang

PDF: Spatial Privacy-aware VR streaming

Abstract

Proactive tile-based virtual reality (VR) video streaming employs the current tracking data of a user to predict future requested tiles, then renders and delivers the predicted tiles before playback. Very recently, privacy protection in proactive VR video streaming starts to raise concerns. However, existing privacy protection may fail even with privacy-preserve federated learning. This is because when the future requested tiles can be predicted accurately, the user-behavior-related data can still be recovered from the predicted tiles. In this paper, we consider how to protect privacy even with accurate predictors and investigate the impact of privacy requirement on the quality of experience (QoE). To this end, we first add extra \textit{camouflaged} tile requests to the real tile requests and model the privacy requirement as the \textit{spatial degree of privacy} (sDoP). By ensuring sDoP, the real tile requests can be hidden and privacy can be protected. Then, we jointly optimize the durations for prediction, computing, and transmitting, aimed at maximizing the privacy-aware QoE given arbitrary predictor and configured resources. From the obtained optimal closed-form solution, we find that the impacts of sDoP on the QoE are two sides of the same coin. On the one side the increase of sDoP improves the capability of communication and computing hence improves QoE. On the other side it degrades the prediction performance hence degrades the QoE. The overall impact depends on which factor dominates the QoE. Simulation with two predictors on a real dataset verifies the analysis and shows that the overall impact of sDoP is to improve the QoE.

Spatial Privacy-aware VR streaming最先出现在Nweon Paper

]]>
A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics https://paper.nweon.com/9850 Mon, 10 May 2021 06:16:38 +0000 https://paper.nweon.com/9850 PubDate: May 2021

A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: Stanford University

Writers: Brooke Krajancich, Petr Kellnhofer, Gordon Wetzstein

PDF: A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics

Abstract

Virtual and augmented reality (VR/AR) displays strive to provide a resolution, framerate and field of view that matches the perceptual capabilities of the human visual system, all while constrained by limited compute budgets and transmission bandwidths of wearable computing systems. Foveated graphics techniques have emerged that could achieve these goals by exploiting the falloff of spatial acuity in the periphery of the visual field. However, considerably less attention has been given to temporal aspects of human vision, which also vary across the retina. This is in part due to limitations of current eccentricity-dependent models of the visual system. We introduce a new model, experimentally measuring and computationally fitting eccentricity-dependent critical flicker fusion thresholds jointly for both space and time. In this way, our model is unique in enabling the prediction of temporal information that is imperceptible for a certain spatial frequency, eccentricity, and range of luminance levels. We validate our model with an image quality user study, and use it to predict potential bandwidth savings 7x higher than those afforded by current spatial-only foveated models. As such, this work forms the enabling foundation for new temporally foveated graphics techniques.

A Perceptual Model for Eccentricity-dependent Spatio-temporal Flicker Fusion and its Applications to Foveated Graphics最先出现在Nweon Paper

]]>
Points2Sound: From mono to binaural audio using 3D point cloud scenes https://paper.nweon.com/9848 Mon, 10 May 2021 05:34:31 +0000 https://paper.nweon.com/9848 PubDate: Apr 2021

Points2Sound: From mono to binaural audio using 3D point cloud scenes最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: University of Music and Performing Arts Vienna

Writers: Francesc Lluís, Vasileios Chatziioannou, Alex Hofmann

PDF: Points2Sound: From mono to binaural audio using 3D point cloud scenes

Abstract

Binaural sound that matches the visual counterpart is crucial to bring meaningful and immersive experiences to people in augmented reality (AR) and virtual reality (VR) applications. Recent works have shown the possibility to generate binaural audio from mono using 2D visual information as guidance. Using 3D visual information may allow for a more accurate representation of a virtual audio scene for VR/AR applications. This paper proposes Points2Sound, a multi-modal deep learning model which generates a binaural version from mono audio using 3D point cloud scenes. Specifically, Points2Sound consist of a vision network which extracts visual features from the point cloud scene to condition an audio network, which operates in the waveform domain, to synthesize the binaural version. Both quantitative and perceptual evaluations indicate that our proposed model is preferred over a reference case, based on a recent 2D mono-to-binaural model.

Points2Sound: From mono to binaural audio using 3D point cloud scenes最先出现在Nweon Paper

]]>
Semi-Autonomous Planning and Visualization in Virtual Reality https://paper.nweon.com/9846 Mon, 10 May 2021 05:11:06 +0000 https://paper.nweon.com/9846 PubDate: Apr 2021

Semi-Autonomous Planning and Visualization in Virtual Reality最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: University of Massachusetts

Writers: Gregory LeMasurier, Jordan Allspaw, Holly A. Yanco

PDF: Semi-Autonomous Planning and Visualization in Virtual Reality

Abstract

Virtual reality (VR) interfaces for robots provide a three-dimensional (3D) view of the robot in its environment, which allows people to better plan complex robot movements in tight or cluttered spaces. In our prior work, we created a VR interface to allow for the teleoperation of a humanoid robot. As detailed in this paper, we have now focused on a human-in-the-loop planner where the operator can send higher level manipulation and navigation goals in VR through functional waypoints, visualize the results of a robot planner in the 3D virtual space, and then deny, alter or confirm the plan to send to the robot. In addition, we have adapted our interface to also work for a mobile manipulation robot in addition to the humanoid robot. For a video demonstration please see the accompanying video at this https URL.

Semi-Autonomous Planning and Visualization in Virtual Reality最先出现在Nweon Paper

]]>
Flow-based Video Segmentation for Human Head and Shoulders https://paper.nweon.com/9844 Thu, 06 May 2021 08:01:45 +0000 https://paper.nweon.com/9844 PubDate: Apr 2021

Flow-based Video Segmentation for Human Head and Shoulders最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: University of Alberta

Writers: Zijian Kuang, Xinran Tie

PDF: Flow-based Video Segmentation for Human Head and Shoulders

Abstract

Video segmentation for the human head and shoulders is essential in creating elegant media for videoconferencing and virtual reality applications. The main challenge is to process high-quality background subtraction in a real-time manner and address the segmentation issues under motion blurs, e.g., shaking the head or waving hands during conference video. To overcome the motion blur problem in video segmentation, we propose a novel flow-based encoder-decoder network (FUNet) that combines both traditional Horn-Schunck optical-flow estimation technique and convolutional neural networks to perform robust real-time video segmentation. We also introduce a video and image segmentation dataset: ConferenceVideoSegmentationDataset.

Flow-based Video Segmentation for Human Head and Shoulders最先出现在Nweon Paper

]]>
Privacy-aware VR streaming https://paper.nweon.com/9842 Thu, 06 May 2021 07:22:31 +0000 https://paper.nweon.com/9842 PubDate:

Privacy-aware VR streaming最先出现在Nweon Paper

]]>
PubDate:

Teams: Beihang University

Writers: Xing Wei, Chenyang Yang

PDF: Privacy-aware VR streaming

Abstract

Proactive tile-based virtual reality (VR) video streaming employs the current tracking data of a user to predict future requested tiles, then renders and delivers the predicted tiles to be requested before playback. The quality of experience (QoE) depends on the overall performance of prediction, computing (i.e., rendering) and communication. All prior works neglect that users may have privacy requirement, i.e., not all the current tracking data are allowed to be uploaded. In this paper, we investigate the privacy-aware VR streaming. We first establish a dataset that collects the privacy requirement of 66 users among 18 panoramic videos. The dataset shows that the privacy requirements of 360∘ videos are heterogeneous. Only 41\% of the total watched videos have no privacy requirement. Based on these findings, we formulate the privacy requirement as the \textit{degree of privacy} (DoP), and investigate the impact of DoP on the proactive VR streaming. First, we find that with DoP, the length of the observation window and prediction window of a tile predictor should be variable. Then, we jointly optimize the durations for computing and transmitting the selected tiles as well as the computing and communication capability, aimed at maximizing the QoE given arbitrary predictor and configured resources. From the obtained optimal closed-form solution, we find a resource-saturated region where DoP has no impact on the QoE and a resource-unsaturated region where the two-fold impacts of DoP are contradictory. On the one hand, the increase of DoP will degrade the prediction performance and thus degrade the QoE. On the other hand, the increase of DoP will improve the capability of computing and communication and thus improve the QoE. Simulation results using two predictors and a real dataset validate the analysis and demonstrate the overall impact of DoP on the QoE.

Privacy-aware VR streaming最先出现在Nweon Paper

]]>
SelectVisAR: Selective Visualisation of Virtual Environments in Augmented Reality https://paper.nweon.com/9840 Thu, 06 May 2021 07:01:31 +0000 https://paper.nweon.com/9840 PubDate: May 2021

SelectVisAR: Selective Visualisation of Virtual Environments in Augmented Reality最先出现在Nweon Paper

]]>
PubDate: May 2021

Teams: KU Leuven

Writers: Robbe Cools, Jihae Han, Adalberto L. Simeone

PDF: SelectVisAR: Selective Visualisation of Virtual Environments in Augmented Reality

Abstract

When establishing a visual connection between a virtual reality user and an augmented reality user, it is important to consider whether the augmented reality user faces a surplus of information. Augmented reality, compared to virtual reality, involves two, not one, planes of information: the physical and the virtual. We propose SelectVisAR, a selective visualisation system of virtual environments in augmented reality. Our system enables an augmented reality spectator to perceive a co-located virtual reality user in the context of four distinct visualisation conditions: Interactive, Proximity, Everything, and Dollhouse. We explore an additional two conditions, Context and Spotlight, in a follow-up study. Our design uses a human-centric approach to information filtering, selectively visualising only parts of the virtual environment related to the interactive possibilities of a virtual reality user. The research investigates how selective visualisations can be helpful or trivial for the augmented reality user when observing a virtual reality user.

SelectVisAR: Selective Visualisation of Virtual Environments in Augmented Reality最先出现在Nweon Paper

]]>
Inverse design enables large-scale high-performance meta-optics reshaping virtual reality https://paper.nweon.com/9838 Thu, 06 May 2021 06:19:23 +0000 https://paper.nweon.com/9838 PubDate: Apr 2021

Inverse design enables large-scale high-performance meta-optics reshaping virtual reality最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: Harvard University;Massachusetts Institute of Technology;, Korea Institute of Science and Technology;National University of Singapore

Writers: Zhaoyi Li, Raphaël Pestourie, Joon-Suh Park, Yao-Wei Huang, Steven G. Johnson, Federico Capasso

PDF: Inverse design enables large-scale high-performance meta-optics reshaping virtual reality

Abstract

Meta-optics has achieved major breakthroughs in the past decade; however, conventional forward design faces challenges as functionality complexity and device size scale up. Inverse design aims at optimizing meta-optics design but has been currently limited by expensive brute-force numerical solvers to small devices, which are also difficult to realize experimentally. Here, we present a general inverse design framework for aperiodic large-scale complex meta-optics in three dimensions, which alleviates computational cost for both simulation and optimization via a fast-approximate solver and an adjoint method, respectively. Our framework naturally accounts for fabrication constraints via a surrogate model. In experiments, we demonstrate, for the first time, aberration-corrected metalenses working in the visible with high numerical aperture, poly-chromatic focusing, and large diameter up to centimeter scale. Such large-scale meta-optics opens a new paradigm for applications, and we demonstrate its potential for future virtual-reality platforms by using a meta-eyepiece and a laser back-illuminated micro-Liquid Crystal Display.

Inverse design enables large-scale high-performance meta-optics reshaping virtual reality最先出现在Nweon Paper

]]>
Combining the Projective Consciousness Model and Virtual Humans to assess ToM capacity in Virtual Reality: a proof-of-concept https://paper.nweon.com/9836 Thu, 06 May 2021 05:43:53 +0000 https://paper.nweon.com/9836 PubDate: Apr 2021

Combining the Projective Consciousness Model and Virtual Humans to assess ToM capacity in Virtual Reality: a proof-of-concept最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: University of Geneva

Writers: David Rudrauf, Grégoire Sergeant-Perthuis, Yvain Tisserand, Teerawat Monnor, Olivier Belli

PDF: Combining the Projective Consciousness Model and Virtual Humans to assess ToM capacity in Virtual Reality: a proof-of-concept

Abstract

Relating explicit psychological mechanisms and observable behaviours is a central aim of psychological and behavioural science. We implemented the principles of the Projective Consciousness Model into artificial agents embodied as virtual humans, as a proof-of-concept for a methodological framework aimed at simulating behaviours and assessing underlying psychological parameters, in the context of experiments in virtual reality. We focus on simulating the role of Theory of Mind (ToM) in the choice of strategic behaviours of approach and avoidance to optimise the satisfaction of agents’ preferences. We designed an experiment in a virtual environment that could be used with real humans, allowing us to classify behaviours as a function of order of ToM, up to the second order. We show that our agents demonstrate expected behaviours with consistent parameters of ToM in this experiment. We also show that the agents can be used to estimate correctly each other order of ToM. A similar approach could be used with real humans in virtual reality experiments not only to enable human participants to interact with parametric, virtual humans as stimuli, but also as a mean of inference to derive model-based psychological assessments of the participants.

Combining the Projective Consciousness Model and Virtual Humans to assess ToM capacity in Virtual Reality: a proof-of-concept最先出现在Nweon Paper

]]>
Spatial Knowledge Acquisition in Virtual and Physical Reality: A Comparative Evaluation https://paper.nweon.com/9834 Thu, 06 May 2021 05:16:28 +0000 https://paper.nweon.com/9834 PubDate: Apr 2021

Spatial Knowledge Acquisition in Virtual and Physical Reality: A Comparative Evaluation最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: Xi’an Jiaotong-Liverpool University;Nanyang Technological University

Writers: Diego Monteiro, Xian Wang, Hai-Ning Liang, Yiyu Cai

PDF: Spatial Knowledge Acquisition in Virtual and Physical Reality: A Comparative Evaluation

Abstract

Virtual Reality (VR) head-mounted displays (HMDs) have been studied widely as tools for the most diverse kinds of training activities. One special kind that is the basis for many real-world applications is spatial knowledge acquisition and navigation. For example, knowing well by heart escape routes can be an important factor for firefighters and soldiers. Prior research on how well knowledge acquired in virtual worlds translates to real, physical one has had mixed results, with some suggesting spatial learning in VR is akin to using a regular 2D display. However, VR HMDs have evolved drastically in the last decade, and little is known about how spatial training skills in a simulated environment using up-to-date VR HMDs compares to training in the real world. In this paper, we aim to investigate how people trained in a VR maze compare against those trained in a physical maze in terms of recall of the position of items inside the environment. While our results did not find significant differences in time performance for people who experienced the physical and those who trained in VR, other behavioural factors were different.

Spatial Knowledge Acquisition in Virtual and Physical Reality: A Comparative Evaluation最先出现在Nweon Paper

]]>
Optimal Transmission of Multi-Quality Tiled 360 VR Video in MIMO-OFDMA Systems https://paper.nweon.com/9832 Thu, 06 May 2021 04:38:03 +0000 https://paper.nweon.com/9832 PubDate: Apr 2021

Optimal Transmission of Multi-Quality Tiled 360 VR Video in MIMO-OFDMA Systems最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: Shanghai Jiao Tong University;University of Electro-Communications

Writers: Chengjun Guo, Ying Cui, Zhi Liu, Derrick Wing Kwan Ng

PDF: Optimal Transmission of Multi-Quality Tiled 360 VR Video in MIMO-OFDMA Systems

Abstract

In this paper, we study the optimal transmission of a multi-quality tiled 360 virtual reality (VR) video from a multi-antenna server (e.g., access point or base station) to multiple single-antenna users in a multiple-input multiple-output (MIMO)-orthogonal frequency division multiple access (OFDMA) system. We minimize the total transmission power with respect to the subcarrier allocation constraints, rate allocation constraints, and successful transmission constraints, by optimizing the beamforming vector and subcarrier, transmission power and rate allocation. The formulated resource allocation problem is a challenging mixed discrete-continuous optimization problem. We obtain an asymptotically optimal solution in the case of a large antenna array, and a suboptimal solution in the general case. As far as we know, this is the first work providing optimization-based design for 360 VR video transmission in MIMO-OFDMA systems. Finally, by numerical results, we show that the proposed solutions achieve significant improvement in performance compared to the existing solutions.

Optimal Transmission of Multi-Quality Tiled 360 VR Video in MIMO-OFDMA Systems最先出现在Nweon Paper

]]>
Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing https://paper.nweon.com/9830 Thu, 06 May 2021 04:13:28 +0000 https://paper.nweon.com/9830 PubDate: Apr 2021

Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing最先出现在Nweon Paper

]]>
PubDate: Apr 2021

Teams: Monash University

Writers: Khoa Cong Hoang, Wesley P. Chan, Steven Lay, Akansel Cosgun, Elizabeth Croft

PDF: Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing

Abstract

Safety is a fundamental requirement in any human-robot collaboration scenario. To ensure the safety of users for such scenarios, we propose a novel Virtual Barrier system facilitated by an augmented reality interface. Our system provides two kinds of Virtual Barriers to ensure safety: 1) a Virtual Person Barrier which encapsulates and follows the user to protect them from colliding with the robot, and 2) Virtual Obstacle Barriers which users can spawn to protect objects or regions that the robot should not enter. To enable effective human-robot collaboration, our system includes an intuitive robot programming interface utilizing speech commands and hand gestures, and features the capability of automatic path re-planning when potential collisions are detected as a result of a barrier intersecting the robot’s planned path. We compared our novel system with a standard 2D display interface through a user study, where participants performed a task mimicking an industrial manufacturing procedure. Results show that our system increases the user’s sense of safety and task efficiency, and makes the interaction more intuitive.

Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing最先出现在Nweon Paper

]]>