Nweon Paper https://paper.nweon.com 映维网,影响力虚拟现实(VR)、增强现实(AR)产业信息数据平台 Thu, 14 Oct 2021 08:01:40 +0000 en-US hourly 1 https://wordpress.org/?v=4.8.17 https://paper.nweon.com/wp-content/uploads/2021/04/nweon-icon.png Nweon Paper https://paper.nweon.com 32 32 3D Hand Pose and Shape Estimation from RGB Images for Improved Keypoint-Based Hand-Gesture Recognition https://paper.nweon.com/11334 Thu, 14 Oct 2021 08:01:40 +0000 https://paper.nweon.com/11334 PubDate: Sep 2021

3D Hand Pose and Shape Estimation from RGB Images for Improved Keypoint-Based Hand-Gesture Recognition最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Sapienza University;Universita di Udine

Writers: Danilo Avola, Luigi Cinque, Alessio Fagioli, Gian Luca Foresti, Adriano Fragomeni, Daniele Pannone

PDF: 3D Hand Pose and Shape Estimation from RGB Images for Improved Keypoint-Based Hand-Gesture Recognition

Abstract

Estimating the 3D hand pose from a 2D image is a well-studied problem and a requirement for several real-life applications such as virtual reality, augmented reality, and hand-gesture recognition. Currently, good estimations can be computed starting from single RGB images, especially when forcing the system to also consider, through a multi-task learning approach, the hand shape when the pose is determined. However, when addressing the aforementioned real-life tasks, performances can drop considerably depending on the hand representation, thus suggesting that stable descriptions are required to achieve satisfactory results. As a consequence, in this paper we present a keypoint-based end-to-end framework for the 3D hand and pose estimation, and successfully apply it to the hand-gesture recognition task as a study case. Specifically, after a pre-processing step where the images are normalized, the proposed pipeline comprises a multi-task semantic feature extractor generating 2D heatmaps and hand silhouettes from RGB images; a viewpoint encoder predicting hand and camera view parameters; a stable hand estimator producing the 3D hand pose and shape; and a loss function designed to jointly guide all of the components during the learning phase. To assess the proposed framework, tests were performed on a 3D pose and shape estimation benchmark dataset, obtaining state-of-the-art performances. What is more, the devised system was also evaluated on 2 hand-gesture recognition benchmark datasets, where the framework significantly outperforms other keypoint-based approaches; indicating that the presented method is an effective solution able to generate stable 3D estimates for the hand pose and shape.

3D Hand Pose and Shape Estimation from RGB Images for Improved Keypoint-Based Hand-Gesture Recognition最先出现在Nweon Paper

]]>
Here To Stay: Measuring Hologram Stability in Markerless Smartphone Augmented Reality https://paper.nweon.com/11332 Thu, 14 Oct 2021 07:22:58 +0000 https://paper.nweon.com/11332 PubDate: Sep 2021

Here To Stay: Measuring Hologram Stability in Markerless Smartphone Augmented Reality最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Duke University;University of California

Writers: Tim Scargill, Jiasi Chen, Maria Gorlatova

PDF:

Abstract

Markerless augmented reality (AR) has the potential to provide engaging experiences and improve outcomes across a wide variety of industries; the overlaying of virtual content, or holograms, onto a view of the real world without the need for predefined markers provides great convenience and flexibility. However, unwanted hologram movement frequently occurs in markerless smartphone AR due to challenging visual conditions or device movement, and resulting error in device pose tracking. We develop a method for measuring hologram positional errors on commercial smartphone markerless AR platforms, implement it as an open-source AR app, HoloMeasure, and use the app to conduct systematic quantitative characterizations of hologram stability across 6 different user actions, 3 different smartphone models, and over 200 different environments. Our study demonstrates significant levels of spatial instability in holograms in all but the simplest settings, and underscores the need for further enhancements to pose tracking algorithms for smartphone-based markerless AR.

Here To Stay: Measuring Hologram Stability in Markerless Smartphone Augmented Reality最先出现在Nweon Paper

]]>
Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse https://paper.nweon.com/11330 Thu, 14 Oct 2021 07:01:23 +0000 https://paper.nweon.com/11330 PubDate: Sep 2021

Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: iZotope;Facebook Reality Labs;Magic Leap;Brian Schmidt Studios

Writers: Jean-Marc Jot, Rémi Audfray, Mark Hertensteiner, Brian Schmidt

PDF: Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse

Abstract

Interactive audio spatialization technology previously developed for video game authoring and rendering has evolved into an essential component of platforms enabling shared immersive virtual experiences for future co-presence, remote collaboration and entertainment applications. New wearable virtual and augmented reality displays employ real-time binaural audio computing engines rendering multiple digital objects and supporting the free navigation of networked participants or their avatars through a juxtaposition of environments, real and virtual, often referred to as the Metaverse. These applications require a parametric audio scene programming interface to facilitate the creation and deployment of shared, dynamic and realistic virtual 3D worlds on mobile computing platforms and remote servers.
We propose a practical approach for designing parametric 6-degree-of-freedom object-based interactive audio engines to deliver the perceptually relevant binaural cues necessary for audio/visual and virtual/real congruence in Metaverse experiences. We address the effects of room reverberation, acoustic reflectors, and obstacles in both the virtual and real environments, and discuss how such effects may be driven by combinations of pre-computed and real-time acoustic propagation solvers. We envision an open scene description model distilled to facilitate the development of interoperable applications distributed across multiple platforms, where each audio object represents, to the user, a natural sound source having controllable distance, size, orientation, and acoustic radiation properties.

Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse最先出现在Nweon Paper

]]>
A 4-DoF Parallel Origami Haptic Device for Normal, Shear, and Torsion Feedback https://paper.nweon.com/11328 Thu, 14 Oct 2021 06:25:20 +0000 https://paper.nweon.com/11328 PubDate: Sep 2021

A 4-DoF Parallel Origami Haptic Device for Normal, Shear, and Torsion Feedback最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Stanford University;Intuitive Surgical

Writers: Sophia R. Williams, Jacob M. Suchoski, Zonghe Chua, Allison M. Okamura

PDF: A 4-DoF Parallel Origami Haptic Device for Normal, Shear, and Torsion Feedback

Abstract

We present a mesoscale finger-mounted 4-degree-of-freedom (DoF) haptic device that is created using origami fabrication techniques. The 4-DoF device is a parallel kinematic mechanism capable of delivering normal, shear, and torsional haptic feedback to the fingertip. Traditional methods of robot fabrication are not well suited for designing small robotic devices because it is challenging and expensive to manufacture small, low-friction joints. Our device uses origami manufacturing principles to reduce complexity and the device footprint. We characterize the bandwidth, workspace, and force output of the device. The capabilities of the torsion-DoF are demonstrated in a virtual reality scenario. Our results show that the device can deliver haptic feedback in 4-DoFs with an effective operational workspace of 0.64cm3 with ±30∘ rotation at every location. The maximum forces and torques the device can apply in the x-, y-, z-, and θ-directions, are ±1.5N, ±1.5N, 2N, and 5N⋅mm, respectively, and the device has an operating bandwidth of 9Hz.

A 4-DoF Parallel Origami Haptic Device for Normal, Shear, and Torsion Feedback最先出现在Nweon Paper

]]>
Multi-View Video-Based 3D Hand Pose Estimation https://paper.nweon.com/11326 Thu, 14 Oct 2021 06:01:19 +0000 https://paper.nweon.com/11326 PubDate: Sep 2021

Multi-View Video-Based 3D Hand Pose Estimation最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Queen’s University

Writers: Leyla Khaleghi, Alireza Sepas Moghaddam, Joshua Marshall, Ali Etemad

PDF: Multi-View Video-Based 3D Hand Pose Estimation

Abstract

Hand pose estimation (HPE) can be used for a variety of human-computer interaction applications such as gesture-based control for physical or virtual/augmented reality devices. Recent works have shown that videos or multi-view images carry rich information regarding the hand, allowing for the development of more robust HPE systems. In this paper, we present the Multi-View Video-Based 3D Hand (MuViHand) dataset, consisting of multi-view videos of the hand along with ground-truth 3D pose labels. Our dataset includes more than 402,000 synthetic hand images available in 4,560 videos. The videos have been simultaneously captured from six different angles with complex backgrounds and random levels of dynamic lighting. The data has been captured from 10 distinct animated subjects using 12 cameras in a semi-circle topology where six tracking cameras only focus on the hand and the other six fixed cameras capture the entire body. Next, we implement MuViHandNet, a neural pipeline consisting of image encoders for obtaining visual embeddings of the hand, recurrent learners to learn both temporal and angular sequential information, and graph networks with U-Net architectures to estimate the final 3D pose information. We perform extensive experiments and show the challenging nature of this new dataset as well as the effectiveness of our proposed method. Ablation studies show the added value of each component in MuViHandNet, as well as the benefit of having temporal and sequential information in the dataset.

Multi-View Video-Based 3D Hand Pose Estimation最先出现在Nweon Paper

]]>
Physics-Informed Neural Networks (PINNs) for Sound Field Predictions with Parameterized Sources and Impedance Boundaries https://paper.nweon.com/11324 Thu, 14 Oct 2021 05:28:19 +0000 https://paper.nweon.com/11324 PubDate: Sep 2021

Physics-Informed Neural Networks (PINNs) for Sound Field Predictions with Parameterized Sources and Impedance Boundaries最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: University of Denmark

Writers: Nikolas Borrel-Jensen, Allan P. Engsig-Karup, Cheol-Ho Jeong

PDF: Physics-Informed Neural Networks (PINNs) for Sound Field Predictions with Parameterized Sources and Impedance Boundaries

Abstract

Realistic sound is essential in virtual environments, such as computer games and mixed reality. Efficient and accurate numerical methods for pre-calculating acoustics have been developed over the last decade, however, pre-calculating acoustics makes handling dynamic scenes with moving sources challenging and requires intractable memory storage. A Physics-Informed Neural Networks (PINNS) method is presented, learning a compact and efficient surrogate model with parameterized moving sources and impedance boundaries, satisfying a system of coupled equations. The trained model shows relative mean errors below 2%/0.2 dB, indicating that acoustics with moving sources and impedance boundaries can be predicted in real-time using PINNs.

Physics-Informed Neural Networks (PINNs) for Sound Field Predictions with Parameterized Sources and Impedance Boundaries最先出现在Nweon Paper

]]>
Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs https://paper.nweon.com/11322 Thu, 14 Oct 2021 05:01:28 +0000 https://paper.nweon.com/11322 PubDate: Sep 2021

Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: The University of Texas At Austin

Writers: Abduallah Mohamed, Huancheng Chen, Zhangyang Wang, Christian Claudel

PDF: Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs

Abstract

Several applications such as autonomous driving, augmented reality and virtual reality require a precise prediction of the 3D human pose. Recently, a new problem was introduced in the field to predict the 3D human poses from observed 2D poses. We propose Skeleton-Graph, a deep spatio-temporal graph CNN model that predicts the future 3D skeleton poses in a single pass from the 2D ones. Unlike prior works, Skeleton-Graph focuses on modeling the interaction between the skeleton joints by exploiting their spatial configuration. This is being achieved by formulating the problem as a graph structure while learning a suitable graph adjacency kernel. By the design, Skeleton-Graph predicts the future 3D poses without divergence in the long-term, unlike prior works. We also introduce a new metric that measures the divergence of predictions in the long term. Our results show an FDE improvement of at least 27% and an ADE of 4% on both the GTA-IM and PROX datasets respectively in comparison with prior works. Also, we are 88% and 93% less divergence on the long-term motion prediction in comparison with prior works on both GTA-IM and PROX datasets.

Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs最先出现在Nweon Paper

]]>
Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays https://paper.nweon.com/11320 Thu, 14 Oct 2021 04:26:14 +0000 https://paper.nweon.com/11320 PubDate: Sep 2021

Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Coburg University of Applied Sciences and Arts;University of Primorska;Microsoft Research;University of Cambridge

Writers: Daniel Schneider, Verena Biener, Alexander Otte, Travis Gesslein, Philipp Gagel, Cuauhtli Campos, Klen Čopič Pucihar, Matjaž Kljun, Eyal Ofek, Michel Pahud, Per Ola Kristensson, Jens Grubert

PDF: Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays

Abstract

An increasing number of consumer-oriented head-mounted displays (HMD) for augmented and virtual reality (AR/VR) are capable of finger and hand tracking. We report on the accuracy of off-the-shelf VR and AR HMDs when used for touch-based tasks such as pointing or drawing. Specifically, we report on the finger tracking accuracy of the VR head-mounted displays Oculus Quest, Vive Pro and the Leap Motion controller, when attached to a VR HMD, as well as the finger tracking accuracy of the AR head-mounted displays Microsoft HoloLens 2 and Magic Leap. We present the results of two experiments in which we compare the accuracy for absolute and relative pointing tasks using both human participants and a robot. The results suggest that HTC Vive has a lower spatial accuracy than the Oculus Quest and Leap Motion and that the Microsoft HoloLens 2 provides higher spatial accuracy than Magic Leap One. These findings can serve as decision support for researchers and practitioners in choosing which systems to use in the future.

Accuracy Evaluation of Touch Tasks in Commodity Virtual and Augmented Reality Head-Mounted Displays最先出现在Nweon Paper

]]>
VRMenuDesigner: A toolkit for automatically generating and modifying VR menus https://paper.nweon.com/11318 Thu, 14 Oct 2021 03:01:22 +0000 https://paper.nweon.com/11318 PubDate: Sep 2021

VRMenuDesigner: A toolkit for automatically generating and modifying VR menus最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Shandong University of Science and Technology;University of South Australia

Writers: Shengzhe Hou, Bruce H. Thomas

PDF: VRMenuDesigner: A toolkit for automatically generating and modifying VR menus

Abstract

With the rapid development of Virtual Reality (VR) technology, the research of User Interface (UI), especially menus, in the VR environment has attracted more and more attention. However, it is very tedious for researchers to develop UI from scratch or modify existing functions and there are no easy-to-use tools for efficient development. This paper aims to present VRMenuDesigner, a flexible and modular toolkit for automatically generating/modifying VR menus. This toolkit is provided as open-source library and easy to extend to adapt to various requirements. The main contribution of this work is to organize the menus and functions with object-oriented thinking, which makes the system very understandable and extensible. VRMenuDesigner includes two key tools: Creator and Modifier for quickly generating and modifying elements. Moreover, we developed several built-in menus and discussed their usability. After a brief review and taxonomy of 3D menus, the architecture and implementation of the toolbox are introduced.

VRMenuDesigner: A toolkit for automatically generating and modifying VR menus最先出现在Nweon Paper

]]>
Architecture and Performance Evaluation of Distributed Computation Offloading in Edge Computing https://paper.nweon.com/11316 Thu, 14 Oct 2021 02:25:22 +0000 https://paper.nweon.com/11316 PubDate: Sep 2021

Architecture and Performance Evaluation of Distributed Computation Offloading in Edge Computing最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: National Research Council, Pisa

Writers: Claudio Cicconetti, Marco Conti, Andrea Passarella

PDF: Architecture and Performance Evaluation of Distributed Computation Offloading in Edge Computing

Abstract

Edge computing is an emerging paradigm to enable low-latency applications, like mobile augmented reality, because it takes the computation on processing devices that are closer to the users. On the other hand, the need for highly scalable execution of stateless tasks for cloud systems is driving the definition of new technologies based on serverless computing. In this paper, we propose a novel architecture where the two converge to enable low-latency applications: this is achieved by offloading short-lived stateless tasks from the user terminals to edge nodes. Furthermore, we design a distributed algorithm that tackles the research challenge of selecting the best executor, based on real-time measurements and simple, yet effective, prediction algorithms. Finally, we describe a new performance evaluation framework specifically designed for an accurate assessment of algorithms and protocols in edge computing environments, where the nodes may have very heterogeneous networking and processing capabilities. The proposed framework relies on the use of real components on lightweight virtualization mixed with simulated computation and is well-suited to the analysis of several applications and network environments. Using our framework, we evaluate our proposed architecture and algorithms in small- and large-scale edge computing scenarios, showing that our solution achieves similar or better delay performance than a centralized solution, with far less network utilization.

Architecture and Performance Evaluation of Distributed Computation Offloading in Edge Computing最先出现在Nweon Paper

]]>
Virtual Reality Gaming on the Cloud: A Reality Check https://paper.nweon.com/11314 Thu, 14 Oct 2021 02:01:22 +0000 https://paper.nweon.com/11314 PubDate: Sep 2021

Virtual Reality Gaming on the Cloud: A Reality Check最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Ryerson University

Writers: Sihao Zhao, Hatem Abou-zeid, Ramy Atawia, Yoga Suhas Kuruba Manjunath, Akram Bin Sediq, Xiao-Ping Zhang

PDF: Virtual Reality Gaming on the Cloud: A Reality Check

Abstract

Cloud virtual reality (VR) gaming traffic characteristics such as frame size, inter-arrival time, and latency need to be carefully studied as a first step toward scalable VR cloud service provisioning. To this end, in this paper we analyze the behavior of VR gaming traffic and Quality of Service (QoS) when VR rendering is conducted remotely in the cloud. We first build a VR testbed utilizing a cloud server, a commercial VR headset, and an off-the-shelf WiFi router. Using this testbed, we collect and process cloud VR gaming traffic data from different games under a number of network conditions and fixed and adaptive video encoding schemes. To analyze the application-level characteristics such as video frame size, frame inter-arrival time, frame loss and frame latency, we develop an interval threshold based identification method for video frames. Based on the frame identification results, we present two statistical models that capture the behaviour of the VR gaming video traffic. The models can be used by researchers and practitioners to generate VR traffic models for simulations and experiments – and are paramount in designing advanced radio resource management (RRM) and network optimization for cloud VR gaming services. To the best of the authors’ knowledge, this is the first measurement study and analysis conducted using a commercial cloud VR gaming platform, and under both fixed and adaptive bitrate streaming. We make our VR traffic data-sets publicly available for further research by the community.

Virtual Reality Gaming on the Cloud: A Reality Check最先出现在Nweon Paper

]]>
Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display https://paper.nweon.com/11312 Wed, 13 Oct 2021 07:43:24 +0000 https://paper.nweon.com/11312 PubDate: Sep 2021

Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: 1Princeton University2Facebook Reality Labs Research3KAUST

Writers: Seung-Hwan Baek, Ethan Tseng, Andrew Maimone, Nathan Matsuda, Grace Kuo, Qiang Fu, Wolfgang Heidrich, Douglas Lanman, Felix Heide

PDF: Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display

Abstract

Holographic displays can generate light fields by dynamically modulating the wavefront of a coherent beam of light using a spatial light modulator, promising rich virtual and augmented reality applications. However, the limited spatial resolution of existing dynamic spatial light modulators imposes a tight bound on the diffraction angle. As a result, today’s holographic displays possess low étendue, which is the product of the display area and the maximum solid angle of diffracted light. The low étendue forces a sacrifice of either the field of view (FOV) or the display size. In this work, we lift this limitation by presenting neural étendue expanders. This new breed of optical elements, which is learned from a natural image dataset, enables higher diffraction angles for ultra-wide FOV while maintaining both a compact form factor and the fidelity of displayed contents to human viewers. With neural étendue expanders, we achieve 64× étendue expansion of natural images with reconstruction quality (measured in PSNR) over 29dB on simulated retinal-resolution images. As a result, the proposed approach with expansion factor 64× enables high-fidelity ultra-wide-angle holographic projection of natural images using an 8K-pixel SLM, resulting in a 18.5 mm eyebox size and 2.18 steradians FOV, covering 85\% of the human stereo FOV.

Neural Étendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display最先出现在Nweon Paper

]]>
FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold https://paper.nweon.com/11310 Wed, 13 Oct 2021 07:16:56 +0000 https://paper.nweon.com/11310 PubDate: Sep 2021

FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Université Côte d’Azur and Inria

Writers: Thomas Leimkühler, George Drettakis

PDF:

Abstract

Current Generative Adversarial Networks (GANs) produce photorealistic renderings of portrait images. Embedding real images into the latent space of such models enables high-level image editing. While recent methods provide considerable semantic control over the (re-)generated images, they can only generate a limited set of viewpoints and cannot explicitly control the camera. Such 3D camera control is required for 3D virtual and mixed reality applications. In our solution, we use a few images of a face to perform 3D reconstruction, and we introduce the notion of the GAN camera manifold, the key element allowing us to precisely define the range of images that the GAN can reproduce in a stable manner. We train a small face-specific neural implicit representation network to map a captured face to this manifold and complement it with a warping scheme to obtain free-viewpoint novel-view synthesis. We show how our approach – due to its precise camera control – enables the integration of a pre-trained StyleGAN into standard 3D rendering pipelines, allowing e.g., stereo rendering or consistent insertion of faces in synthetic 3D environments. Our solution proposes the first truly free-viewpoint rendering of realistic faces at interactive rates, using only a small number of casual photos as input, while simultaneously allowing semantic editing capabilities, such as facial expression or lighting changes.

FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold最先出现在Nweon Paper

]]>
Rectangle-based Approximation for Rendering Glossy Interreflections https://paper.nweon.com/11308 Wed, 13 Oct 2021 07:01:21 +0000 https://paper.nweon.com/11308 PubDate: Sep 2021

Rectangle-based Approximation for Rendering Glossy Interreflections最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Chunbiao Guo

Writers: Chunbiao Guo

PDF: Rectangle-based Approximation for Rendering Glossy Interreflections

Abstract

This study introduces an approximation for rendering one bounce glossy interreflection in real time. The solution is based on the most representative point (MRP) and extends to a sampling disk near the MRP. Our algorithm represents geometry as rectangle proxies and specular reflections using a spherical Gaussian. The reflected radiance from the disk was efficiently approximated by selecting a representative attenuation axis in the sampling disk. We provide an efficient approximation of the glossy interreflection and can efficiently perform the approximation at runtime. Our method uses forward rendering (without using GBuffer), which is more suitable for platforms that favor forward rendering, such as mobile applications and virtual reality.

Rectangle-based Approximation for Rendering Glossy Interreflections最先出现在Nweon Paper

]]>
CrowdDriven: A New Challenging Dataset for Outdoor Visual Localization https://paper.nweon.com/11306 Wed, 13 Oct 2021 06:28:32 +0000 https://paper.nweon.com/11306 PubDate: Sep 2021

CrowdDriven: A New Challenging Dataset for Outdoor Visual Localization最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: 1Chalmers University of Technology 2Facebook 3Czech Technical University in Prague

Writers: Ara Jafarzadeh, Manuel Lopez Antequera, Pau Gargallo, Yubin Kuang, Carl Toft, Fredrik Kahl, Torsten Sattler

PDF: CrowdDriven: A New Challenging Dataset for Outdoor Visual Localization

Abstract

Visual localization is the problem of estimating the position and orientation from which a given image (or a sequence of images) is taken in a known scene. It is an important part of a wide range of computer vision and robotics applications, from self-driving cars to augmented/virtual reality systems. Visual localization techniques should work reliably and robustly under a wide range of conditions, including seasonal, weather, illumination and man-made changes. Recent benchmarking efforts model this by providing images under different conditions, and the community has made rapid progress on these datasets since their inception. However, they are limited to a few geographical regions and often recorded with a single device. We propose a new benchmark for visual localization in outdoor scenes, using crowd-sourced data to cover a wide range of geographical regions and camera devices with a focus on the failure cases of current algorithms. Experiments with state-of-the-art localization approaches show that our dataset is very challenging, with all evaluated methods failing on its hardest parts. As part of the dataset release, we provide the tooling used to generate it, enabling efficient and effective 2D correspondence annotation to obtain reference poses.

CrowdDriven: A New Challenging Dataset for Outdoor Visual Localization最先出现在Nweon Paper

]]>
Comfort and Sickness while Virtually Aboard an Autonomous Telepresence Robot https://paper.nweon.com/11304 Wed, 13 Oct 2021 05:46:22 +0000 https://paper.nweon.com/11304 PubDate: Sep 2021

Comfort and Sickness while Virtually Aboard an Autonomous Telepresence Robot最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: University of Oulu;Centro de Investigacion en Matematicas

Writers: Markku Suomalainen, Katherine J. Mimnaugh, Israel Becerra, Eliezer Lozano, Rafael Murrieta-Cid, Steven M. LaValle

PDF: Comfort and Sickness while Virtually Aboard an Autonomous Telepresence Robot

Abstract

In this paper, we analyze how different path aspects affect a user’s experience, mainly VR sickness and overall comfort, while immersed in an autonomously moving telepresence robot through a virtual reality headset. In particular, we focus on how the robot turns and the distance it keeps from objects, with the goal of planning suitable trajectories for an autonomously moving immersive telepresence robot in mind; rotational acceleration is known for causing the majority of VR sickness, and distance to objects modulates the optical flow. We ran a within-subjects user study (n = 36, women = 18) in which the participants watched three panoramic videos recorded in a virtual museum while aboard an autonomously moving telepresence robot taking three different paths varying in aspects such as turns, speeds, or distances to walls and objects. We found a moderate correlation between the users’ sickness as measured by the SSQ and comfort on a 6-point Likert scale across all paths. However, we detected no association between sickness and the choice of the most comfortable path, showing that sickness is not the only factor affecting the comfort of the user. The subjective experience of turn speed did not correlate with either the SSQ scores or comfort, even though people often mentioned turning speed as a source of discomfort in the open-ended questions. Through exploring the open-ended answers more carefully, a possible reason is that the length and lack of predictability also play a large role in making people observe turns as uncomfortable. A larger subjective distance from walls and objects increased comfort and decreased sickness both in quantitative and qualitative data. Finally, the SSQ subscales and total weighted scores showed differences by age group and by gender.

Comfort and Sickness while Virtually Aboard an Autonomous Telepresence Robot最先出现在Nweon Paper

]]>
Rethinking Immersive Virtual Reality and Empathy https://paper.nweon.com/11302 Wed, 13 Oct 2021 03:01:26 +0000 https://paper.nweon.com/11302 PubDate: Sep 2021

Rethinking Immersive Virtual Reality and Empathy最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: University of Waterloo

Writers: Ken Jen Lee, Edith Law

PDF: Rethinking Immersive Virtual Reality and Empathy

Abstract

In this position paper, we aim to spark more discussions surrounding the use of empathy as the intended outcome of many studies on immersive virtual reality experiences. As a construct, empathy has many significant flaws that may lead to unintended and negative outcomes, going against our original goal of employing these technologies for the betterment of society. We highlight the possible advantages of designing for rational compassion instead, and propose alternative research directions and outcome measurements for immersive virtual reality that urgently warrant our attention.

Rethinking Immersive Virtual Reality and Empathy最先出现在Nweon Paper

]]>
Eye-Tracking-Based Design of Mixed Reality Learning Environments in STEM https://paper.nweon.com/11300 Wed, 13 Oct 2021 02:13:36 +0000 https://paper.nweon.com/11300 PubDate: Sep 2021

Eye-Tracking-Based Design of Mixed Reality Learning Environments in STEM最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Technische Universität Braunschweig

Writers: Dörte Sonntag, Oliver Bodensiek

PDF: Eye-Tracking-Based Design of Mixed Reality Learning Environments in STEM

Abstract

With the advent of commercially available Mixed-Reality(MR)-headsets in recent years MR-assisted learning started to play a vital role in educational research, especially related to STEM (science, technology, engineering and mathematics) education. Along with these developments it seems viable to further frameworks and structured design processes for MR-based learning environments. Instead of a widely applicable framework for designing educational MR applications, we here consider the case of virtually enhancing physical hands-on experiments in STEM, where students are given a certain problem to solve, and how to design these. For this focused realm, we suggest an empirically driven problemand user-centred design process for MR applications to get novices to act more like experts and exemplify it for a specific experiment and problem set containing a non-trivial electric circuit with capacitors and coils.

Eye-Tracking-Based Design of Mixed Reality Learning Environments in STEM最先出现在Nweon Paper

]]>
Horizontal and Vertical Collaboration for VR Delivery in MEC-Enabled Small-Cell Networks https://paper.nweon.com/11298 Wed, 13 Oct 2021 01:37:24 +0000 https://paper.nweon.com/11298 PubDate: Sep 2021

Horizontal and Vertical Collaboration for VR Delivery in MEC-Enabled Small-Cell Networks最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Zhuojia Gu, Hancheng Lu, Chenkai Zou

Writers: Zhuojia Gu, Hancheng Lu, Chenkai Zou

PDF: Horizontal and Vertical Collaboration for VR Delivery in MEC-Enabled Small-Cell Networks

Abstract

Due to the large bandwidth, low latency and computationally intensive features of virtual reality (VR) video applications, the current resource-constrained wireless and edge networks cannot meet the requirements of on-demand VR delivery. In this letter, we propose a joint horizontal and vertical collaboration architecture in mobile edge computing (MEC)-enabled small-cell networks for VR delivery. In the proposed architecture, multiple MEC servers can jointly provide VR head-mounted devices (HMDs) with edge caching and viewpoint computation services, while the computation tasks can also be performed at HMDs or on the cloud. Power allocation at base stations (BSs) is considered in coordination with horizontal collaboration (HC) and vertical collaboration (VC) of MEC servers to obtain lower end-to-end latency of VR delivery. A joint caching, power allocation and task offloading problem is then formulated, and a discrete branch-reduce-and-bound (DBRB) algorithm inspired by monotone optimization is proposed to effectively solve the problem. Simulation results demonstrate the advantage of the proposed architecture and algorithm in terms of existing ones.

Horizontal and Vertical Collaboration for VR Delivery in MEC-Enabled Small-Cell Networks最先出现在Nweon Paper

]]>
Exploratory Design of a Hands-free Video Game Controller for a Quadriplegic Individual https://paper.nweon.com/11296 Tue, 12 Oct 2021 06:01:20 +0000 https://paper.nweon.com/11296 PubDate: Sep 2021

Exploratory Design of a Hands-free Video Game Controller for a Quadriplegic Individual最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: University of California;Palo Alto Senior High School

Writers: Atieh Taheri, Ziv Weissman, Misha Sra

PDF: Exploratory Design of a Hands-free Video Game Controller for a Quadriplegic Individual

Abstract

From colored pixels to hyper-realistic 3D landscapes of virtual reality, video games have evolved immensely over the last few decades. However, video game input still requires two-handed dexterous finger manipulations for simultaneous joystick and trigger or mouse and keyboard presses. In this work, we explore the design of a hands-free game control method using realtime facial expression recognition for individuals with neurological and neuromuscular diseases who are unable to use traditional game controllers. Similar to other Assistive Technologies (AT), our facial input technique is also designed and tested in collaboration with a graduate student who has Spinal Muscular Atrophy. Our preliminary evaluation shows the potential of facial expression recognition for augmenting the lives of quadriplegic individuals by enabling them to accomplish things like walking, running, flying or other adventures that may not be so attainable otherwise.

Exploratory Design of a Hands-free Video Game Controller for a Quadriplegic Individual最先出现在Nweon Paper

]]>
Realistic Hands: A Hybrid Model for 3D Hand Reconstruction https://paper.nweon.com/11294 Tue, 12 Oct 2021 05:22:21 +0000 https://paper.nweon.com/11294 PubDate: Aug 2021

Realistic Hands: A Hybrid Model for 3D Hand Reconstruction最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: ETH Zurich;University of Haifa

Writers: Michael Seeber, Martin R. Oswald, Roi Poranne

PDF: Realistic Hands: A Hybrid Model for 3D Hand Reconstruction

Abstract

Estimating 3D hand meshes from RGB images robustly is a highly desirable task, made challenging due to the numerous degrees of freedom, and issues such as self similarity and occlusions. Previous methods generally either use parametric 3D hand models or follow a model-free approach. While the former can be considered more robust, e.g. to occlusions, they are less expressive. We propose a hybrid approach, utilizing a deep neural network and differential rendering based optimization to demonstrably achieve the best of both worlds. In addition, we explore Virtual Reality (VR) as an application. Most VR headsets are nowadays equipped with multiple cameras, which we can leverage by extending our method to the egocentric stereo domain. This extension proves to be more resilient to the above mentioned issues. Finally, as a use-case, we show that the improved image-model alignment can be used to acquire the user’s hand texture, which leads to a more realistic virtual hand representation.

Realistic Hands: A Hybrid Model for 3D Hand Reconstruction最先出现在Nweon Paper

]]>
Optimal Latency-Oriented Scheduling in Parallel Queuing Systems https://paper.nweon.com/11292 Tue, 12 Oct 2021 05:01:20 +0000 https://paper.nweon.com/11292 PubDate: Aug 2021

Optimal Latency-Oriented Scheduling in Parallel Queuing Systems最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: University of Padova; Aalborg University; Nokia Bell Labs

Writers: Andrea Bedin, Federico Chiariotti, Stepan Kucera, Andrea Zanella

PDF: Optimal Latency-Oriented Scheduling in Parallel Queuing Systems

Abstract

Today, more and more interactive applications, such as augmented/virtual reality, haptic Internet, and Industrial Internet of Things, require communication services with guaranteed end-to-end latency limits, which are difficult to provide over shared communication networks, particularly in the presence of wireless links. Robustness against disturbances affecting individual links can be obtained by coding the information flow in multiple streams to be forwarded across parallel transmission links. This approach, however, requires coding and scheduling algorithms that can adapt to the state of links to take full advantage of path diversity and avoid self-induced congestion on some links. To gain some fundamental insights on this challenging problem, in this paper we resort to Markov Decision Process (MDP) theory and abstract the parallel paths as independent queuing systems, whose arrival processes are managed by a common controller that determines the amount of redundancy to be applied to the source messages and the number of (coded) packets to be sent to each queue. The objective is to find the joint coding and scheduling policy that maximizes a certain utility function, e.g., the fraction of source blocks delivered to the destination within a predetermined deadline, despite the variability of the individual connections. We find the optimal redundancy and scheduling strategies by using policy iteration methods. We then analyze the optimal policy in a series of scenarios, highlighting its most important aspects and analyzing ways to improve existing heuristics from the literature.

Optimal Latency-Oriented Scheduling in Parallel Queuing Systems最先出现在Nweon Paper

]]>
Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80% of Your Bandwidth https://paper.nweon.com/11290 Tue, 12 Oct 2021 04:19:19 +0000 https://paper.nweon.com/11290 PubDate: Sep 2021

Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80% of Your Bandwidth最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: Stanford Universit

Writers: Luke Hsiao, Brooke Krajancich, Philip Levis, Gordon Wetzstein, Keith Winstein

PDF: Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80% of Your Bandwidth

Abstract

Virtual reality systems today cannot yet stream immersive, retina-quality virtual reality video over a network. One of the greatest challenges to this goal is the sheer data rates required to transmit retina-quality video frames at high resolutions and frame rates. Recent work has leveraged the decay of visual acuity in human perception in novel gaze-contingent video compression techniques. In this paper, we show that reducing the motion-to-photon latency of a system itself is a key method for improving the compression ratio of gaze-contingent compression. Our key finding is that a client and streaming server system with sub-15ms latency can achieve 5x better compression than traditional techniques while also using simpler software algorithms than previous work.

Towards Retina-Quality VR Video Streaming: 15ms Could Save You 80% of Your Bandwidth最先出现在Nweon Paper

]]>
Rotation-constrained optical see-through headset calibration withbare-hand alignment https://paper.nweon.com/11288 Tue, 12 Oct 2021 03:01:19 +0000 https://paper.nweon.com/11288 PubDate: Aug 2021

Rotation-constrained optical see-through headset calibration withbare-hand alignment最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: Imperial College London;University of Pisa

Writers: Xue Hu, Ferdinando Rodriguez y Baena, Fabrizio Cutolo

PDF: Rotation-constrained optical see-through headset calibration withbare-hand alignment

Abstract

The inaccessibility of user-perceived reality remains an open issue in pursuing the accurate calibration of optical see-through (OST) head-mounted displays (HMDs). Manual user alignment is usually required to collect a set of virtual-to-real correspondences, so that a default or an offline display calibration can be updated to account for the user’s eye position(s). Current alignment-based calibration procedures usually require point-wise alignments between rendered image point(s) and associated physical landmark(s) of a target calibration tool. As each alignment can only provide one or a few correspondences, repeated alignments are required to ensure calibration quality.
This work presents an accurate and tool-less online OST calibration method to update an offline-calibrated eye-display model. The user’s bare hand is markerlessly tracked by a commercial RGBD camera anchored to the OST headset to generate a user-specific cursor for correspondence collection. The required alignment is object-wise, and can provide thousands of unordered corresponding points in tracked space. The collected correspondences are registered by a proposed rotation-constrained iterative closest point (rcICP) method to optimise the viewpoint-related calibration parameters. We implemented such a method for the Microsoft HoloLens 1. The resiliency of the proposed procedure to noisy data was evaluated through simulated tests and real experiments performed with an eye-replacement camera. According to the simulation test, the rcICP registration is robust against possible user-induced rotational misalignment. With a single alignment, our method achieves 8.81 arcmin (1.37 mm) positional error and 1.76 degree rotational error by camera-based tests in the arm-reach distance, and 10.79 arcmin (7.71 pixels) reprojection error by user tests.

Rotation-constrained optical see-through headset calibration withbare-hand alignment最先出现在Nweon Paper

]]>
Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments https://paper.nweon.com/11286 Tue, 12 Oct 2021 02:19:34 +0000 https://paper.nweon.com/11286 PubDate: Aug 2021

Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: University of Missouri-Columbia, *Columbia College, +Rutgers University

Writers: Samaikya Valluripally, Vaibhav Akashe, Michael Fisher, David Falana, Khaza Anuarul Hoque, Prasad Calyam

PDF: Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments

Abstract

Social virtual reality learning environments (VRLEs) provide immersive experience to users with increased accessibility to remote learning. Lack of maintaining high-performance and secured data delivery in critical VRLE application domains (e.g., military training, manufacturing) can disrupt application functionality and induce cybersickness. In this paper, we present a novel rule-based 3QS-adaptation framework that performs risk and cost aware trade-off analysis to control cybersickness due to performance/security anomaly events during a VRLE session. Our framework implementation in a social VRLE viz., vSocial monitors performance/security anomaly events in network/session data. In the event of an anomaly, the framework features rule-based adaptations that are triggered by using various decision metrics. Based on our experimental results, we demonstrate the effectiveness of our rule-based 3QS-adaptation framework in reducing cybersickness levels, while maintaining application functionality. Using our key findings, we enlist suitable practices for addressing performance and security issues towards a more high-performing and robust social VRLE.

Rule-based Adaptations to Control Cybersickness in Social Virtual Reality Learning Environments最先出现在Nweon Paper

]]>
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots https://paper.nweon.com/11284 Tue, 12 Oct 2021 01:40:21 +0000 https://paper.nweon.com/11284 PubDate: Aug 2021

HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daneil Leithinger, Mar Gonzalez-Franco

Writers: Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daneil Leithinger, Mar Gonzalez-Franco

PDF: HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots

Abstract

HapticBots introduces a novel encountered-type haptic approach for Virtual Reality (VR) based on multiple tabletop-size shape-changing robots. These robots move on a tabletop and change their height and orientation to haptically render various surfaces and objects on-demand. Compared to previous encountered-type haptic approaches like shape displays or robotic arms, our proposed approach has an advantage in deployability, scalability, and generalizability – these robots can be easily deployed due to their compact form factor. They can support multiple concurrent touch points in a large area thanks to the distributed nature of the robots. We propose and evaluate a novel set of interactions enabled by these robots which include: 1) rendering haptics for VR objects by providing just-in-time touch-points on the user’s hand, 2) simulating continuous surfaces with the concurrent height and position change, and 3) enabling the user to pick up and move VR objects through graspable proxy objects. Finally, we demonstrate HapticBots with various applications, including remote collaboration, education and training, design and 3D modeling, and gaming and entertainment.

HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots最先出现在Nweon Paper

]]>
EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers https://paper.nweon.com/11282 Tue, 12 Oct 2021 00:46:21 +0000 https://paper.nweon.com/11282 PubDate: Oct 2021

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers最先出现在Nweon Paper

]]>
PubDate: Oct 2021

Teams: ethz

Writers: Manuel Kaufmann, Yi Zhao, Chengcheng Tang, Lingling Tao, Christopher Twigg, Jie Song, Robert Wang, Otmar Hilliges

PDF: EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

Abstract

Fully immersive experiences in AR/VR critically depend on reconstructing the full body pose of the user without restricting their motion. In this paper we study the use of body-worn electromagnetic (EM) field-based sensing for the task of 3D human pose reconstruction. To this end, we present a method to estimate SMPL parameters from 6-12 EM sensors. We leverage a customized wearable system consisting of wireless EM sensors measuring time-synchronized 6D poses at 120 Hz. To provide accurate poses even with little user instrumentation, we adopt a recently proposed hybrid framework, learned gradient descent (LGD), to iteratively estimate SMPL pose and shape from our input measurements. This allows us to harness powerful pose priors to cope with the idiosyncrasies of the input data and achieve accurate pose estimates. The proposed method uses AMASS to synthesize virtual EM-sensor data and we show that it generalizes well to a newly captured real dataset consisting of a total of 36 minutes of motion from 5 subjects. We achieve reconstruction errors as low as 31.8 mm and 13.3 degrees, outperforming both pure learning- and pure optimization-based methods.

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers最先出现在Nweon Paper

]]>
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting https://paper.nweon.com/11280 Mon, 11 Oct 2021 05:01:20 +0000 https://paper.nweon.com/11280 PubDate: Sep 2021

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting最先出现在Nweon Paper

]]>
PubDate: Sep 2021

Teams: NVIDIA;University of Toronto;Vector Institute

Writers: Zian Wang, Jonah Philion, Sanja Fidler, Jan Kautz

PDF: Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

Abstract

In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. However, indoor scenes contain complex 3D light transport where a 2D representation is insufficient. In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting. Inspired by classic volume rendering techniques, we propose a novel Volumetric Spherical Gaussian representation for lighting, which parameterizes the exitant radiance of the 3D scene surfaces on a voxel grid. We design a physics based differentiable renderer that utilizes our 3D lighting representation, and formulates the energy-conserving image formation process that enables joint training of all intrinsic properties with the re-rendering constraint. Our model ensures physically correct predictions and avoids the need for ground-truth HDR lighting which is not easily accessible. Experiments show that our method outperforms prior works both quantitatively and qualitatively, and is capable of producing photorealistic results for AR applications such as virtual object insertion even for highly specular objects.

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting最先出现在Nweon Paper

]]>
The “Kinesthetic HMD”: Enhancing Self-Motion Sensations in VR with Head-Based Force Feedback https://paper.nweon.com/11278 Sat, 09 Oct 2021 08:01:54 +0000 https://paper.nweon.com/11278 PubDate: Aug 2021

The “Kinesthetic HMD”: Enhancing Self-Motion Sensations in VR with Head-Based Force Feedback最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: Univ Rennes

Writers: Antoine Costes, Anatole Lécuyer

PDF: The “Kinesthetic HMD”: Enhancing Self-Motion Sensations in VR with Head-Based Force Feedback

Abstract

The sensation of self-motion is essential in many virtual reality applications, from entertainment to training, such as flying and driving simulators. If the common approach used in amusement parks is to actuate the seats with cumbersome systems, multisensory integration can also be leveraged to get rich effects from lightweight solutions. In this short paper, we introduce a novel approach called the “Kinesthetic HMD”: actuating a head-mounted display with force feedback in order to provide sensations of self-motion. We discuss its design considerations and demonstrate an augmented flight simulator use case with a proof-of-concept prototype. We conducted a user study assessing our approach’s ability to enhance self-motion sensations. Taken together, our results show that our Kinesthetic HMD provides significantly stronger and more egocentric sensations than a visual-only self-motion experience. Thus, by providing congruent vestibular and proprioceptive cues related to balance and self-motion, the Kinesthetic HMD represents a promising approach for a variety of virtual reality applications in which motion sensations are prominent.

The “Kinesthetic HMD”: Enhancing Self-Motion Sensations in VR with Head-Based Force Feedback最先出现在Nweon Paper

]]>
ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones https://paper.nweon.com/11276 Sat, 09 Oct 2021 07:22:20 +0000 https://paper.nweon.com/11276 PubDate: Aug 2021

ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones最先出现在Nweon Paper

]]>
PubDate: Aug 2021

Teams: JD.COM Inc, Beihang University;Beijing Advanced Innovation Center for Big Data-Based Precision Medicine

Writers: Shan An, Guangfu Che, Jinghao Guo, Haogang Zhu, Junjie Ye, Fangru Zhou, Zhaoqi Zhu, Dong Wei, Aishan Liu, Wei Zhang

PDF: ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones

Abstract

Virtual try-on technology enables users to try various fashion items using augmented reality and provides a convenient online shopping experience. However, most previous works focus on the virtual try-on for clothes while neglecting that for shoes, which is also a promising task. To this concern, this work proposes a real-time augmented reality virtual shoe try-on system for smartphones, namely ARShoe. Specifically, ARShoe adopts a novel multi-branch network to realize pose estimation and segmentation simultaneously. A solution to generate realistic 3D shoe model occlusion during the try-on process is presented. To achieve a smooth and stable try-on effect, this work further develop a novel stabilization method. Moreover, for training and evaluation, we construct the very first large-scale foot benchmark with multiple virtual shoe try-on task-related labels annotated. Exhaustive experiments on our newly constructed benchmark demonstrate the satisfying performance of ARShoe. Practical tests on common smartphones validate the real-time performance and stabilization of the proposed approach.

ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones最先出现在Nweon Paper

]]>