Teams: NVIDIA Research 2Johns Hopkins University

Writers: Zhaoshuo Li1,2;Thomas Müller1;Alex Evans1;Russell H. Taylor2;Mathias Unberath2;Ming-Yu Liu1;Chen-Hsuan Lin1

PDF: Neuralangelo: High-Fidelity Neural Surface Reconstruction

Project: Neuralangelo: High-Fidelity Neural Surface Reconstruction

Abstract

Neural surface reconstruction has shown to be powerful for recovering dense 3D surfaces via image-based neural rendering. However, current methods struggle to recover detailed structures of real-world scenes. To address the issue, we present Neuralangelo, which combines the representation power of multi-resolution 3D hash grids with neural surface rendering. Our approach is enabled by two key ingredients: (1) numerical gradients for computing higher-order derivatives as a smoothing operation and (2) coarseto-fine optimization on the hash grids controlling different levels of details. Even without auxiliary depth, Neuralangelo can effectively recover dense 3D surface structures from multi-view images with a fidelity that significantly surpasses previous methods, enabling detailed large-scale scene reconstruction from RGB video captures.

Neuralangelo: High-Fidelity Neural Surface Reconstruction最先出现在Nweon Paper。

Teams: Shandong University

Writers: Ruxiao Chen, Shuaishuai Guo

PDF: Look-Ahead Task Offloading for Multi-User Mobile Augmented Reality in Edge-Cloud Computing

Abstract

Mobile augmented reality (MAR) blends a real scenario with overlaid virtual content, which has been envisioned as one of the ubiquitous interfaces to the Metaverse. Due to the limited computing power and battery life of MAR devices, it is common to offload the computation tasks to edge or cloud servers in close proximity. However, existing offloading solutions developed for MAR tasks suffer from high migration overhead, poor scalability, and short-sightedness when applied in provisioning multi-user MAR services. To address these issues, a MAR service-oriented task offloading scheme is designed and evaluated in edge-cloud computing networks. Specifically, the task interdependency of MAR applications is firstly analyzed and modeled by using directed acyclic graphs. Then, we propose a look-ahead offloading scheme based on a modified Monte Carlo tree (MMCT) search, which can run several multi-step executions in advance to get an estimate of the long-term effect of immediate action. Experiment results show that the proposed offloading scheme can effectively improve the quality of service (QoS) in provisioning multi-user MAR services, compared to four benchmark schemes. Furthermore, it is also shown that the proposed solution is stable and suitable for applications in a highly volatile environment.

Look-Ahead Task Offloading for Multi-User Mobile Augmented Reality in Edge-Cloud Computing最先出现在Nweon Paper。

Teams: UC Berkeley;University of Wurzburg

Writers: Vivek Nair, Christian Rack, Wenbo Guo, Rui Wang, Shuixian Li, Brandon Huang, Atticus Cull, James F. O’Brien, Louis Rosenberg, Dawn Song

PDF: Inferring Private Personal Attributes of Virtual Reality Users from Head and Hand Motion Data

Abstract

Motion tracking “telemetry” data lies at the core of nearly all modern virtual reality (VR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to uniquely identify VR users. In this study, we go a step further, showing that a variety of private user information can be inferred just by analyzing motion data recorded by VR devices. We conducted a large-scale survey of VR users (N=1,006) with dozens of questions ranging from background and demographics to behavioral patterns and health information. We then collected VR motion samples of each user playing the game “Beat Saber,” and attempted to infer their survey responses using just their head and hand motion patterns. Using simple machine learning models, many of these attributes could accurately and consistently be inferred from VR motion data alone, highlighting the pressing need for privacy-preserving mechanisms in multi-user VR applications.

Inferring Private Personal Attributes of Virtual Reality Users from Head and Hand Motion Data最先出现在Nweon Paper。

Teams: University of Bristol

Writers: Adrian Azzarelli, Nantheera Anantrasirichai, David R Bull

PDF: Towards a Robust Framework for NeRF Evaluation

Abstract

Neural Radiance Field (NeRF) research has attracted significant attention recently, with 3D modelling, virtual/augmented reality, and visual effects driving its application. While current NeRF implementations can produce high quality visual results, there is a conspicuous lack of reliable methods for evaluating them. Conventional image quality assessment methods and analytical metrics (e.g. PSNR, SSIM, LPIPS etc.) only provide approximate indicators of performance since they generalise the ability of the entire NeRF pipeline. Hence, in this paper, we propose a new test framework which isolates the neural rendering network from the NeRF pipeline and then performs a parametric evaluation by training and evaluating the NeRF on an explicit radiance field representation. We also introduce a configurable approach for generating representations specifically for evaluation purposes. This employs ray-casting to transform mesh models into explicit NeRF samples, as well as to “shade” these representations. Combining these two approaches, we demonstrate how different “tasks” (scenes with different visual effects or learning strategies) and types of networks (NeRFs and depth-wise implicit neural representations (INRs)) can be evaluated within this framework. Additionally, we propose a novel metric to measure task complexity of the framework which accounts for the visual parameters and the distribution of the spatial data. Our approach offers the potential to create a comparative objective evaluation framework for NeRF methods.

Towards a Robust Framework for NeRF Evaluation最先出现在Nweon Paper。

Teams: Stanford University;Cornell University

Writers: Kyle T. Yoshida, Joel X. Kiernan, Allison M. Okamura, Cara M. Nunez

PDF: Exploring Human Response Times to Combinations of Audio, Haptic, and Visual Stimuli from a Mobile Device

Abstract

Auditory, haptic, and visual stimuli provide alerts, notifications, and information for a wide variety of applications ranging from virtual reality to wearable and hand-held devices. Response times to these stimuli have been used to assess motor control and design human-computer interaction systems. In this study, we investigate human response times to 26 combinations of auditory, haptic, and visual stimuli at three levels (high, low, and off). We developed an iOS app that presents these stimuli in random intervals and records response times on an iPhone 11. We conducted a user study with 20 participants and found that response time decreased with more types and higher levels of stimuli. The low visual condition had the slowest mean response time (mean +/- standard deviation, 528 +/- 105 ms) and the condition with high levels of audio, haptic, and visual stimuli had the fastest mean response time (320 +/- 43 ms). This work quantifies response times to multi-modal stimuli, identifies interactions between different stimuli types and levels, and introduces an app-based method that can be widely distributed to measure response time. Understanding preferences and response times for stimuli can provide insight into designing devices for human-machine interaction.

Exploring Human Response Times to Combinations of Audio, Haptic, and Visual Stimuli from a Mobile Device最先出现在Nweon Paper。

Teams: The University of Texas at Arlington

Writers: Minh Q. Tram, Joseph M. Cloud, William J. Beksi

PDF: Intuitive Robot Integration via Virtual Reality Workspaces

Abstract

As robots become increasingly prominent in diverse industrial settings, the desire for an accessible and reliable system has correspondingly increased. Yet, the task of meaningfully assessing the feasibility of introducing a new robotic component, or adding more robots into an existing infrastructure, remains a challenge. This is due to both the logistics of acquiring a robot and the need for expert knowledge in setting it up. In this paper, we address these concerns by developing a purely virtual simulation of a robotic system. Our proposed framework enables natural human-robot interaction through a visually immersive representation of the workspace. The main advantages of our approach are the following: (i) independence from a physical system, (ii) flexibility in defining the workspace and robotic tasks, and (iii) an intuitive interaction between the operator and the simulated environment. Not only does our system provide an enhanced understanding of 3D space to the operator, but it also encourages a hands-on way to perform robot programming. We evaluate the effectiveness of our method in applying novel automation assignments by training a robot in virtual reality and then executing the task on a real robot.

Intuitive Robot Integration via Virtual Reality Workspaces最先出现在Nweon Paper。

Teams: German Research Center for Artificial Intelligence;University of Oldenburg

Writers: Hannes Kath, Bengt Lüers, Thiago S. Gouvêa, Daniel Sonntag

PDF: A Virtual Reality Tool for Representing, Visualizing and Updating Deep Learning Models

Abstract

Deep learning is ubiquitous, but its lack of transparency limits its impact on several potential application areas. We demonstrate a virtual reality tool for automating the process of assigning data inputs to different categories. A dataset is represented as a cloud of points in virtual space. The user explores the cloud through movement and uses hand gestures to categorise portions of the cloud. This triggers gradual movements in the cloud: points of the same category are attracted to each other, different groups are pushed apart, while points are globally distributed in a way that utilises the entire space. The space, time, and forces observed in virtual reality can be mapped to well-defined machine learning concepts, namely the latent space, the training epochs and the backpropagation. Our tool illustrates how the inner workings of deep neural networks can be made tangible and transparent. We expect this approach to accelerate the autonomous development of deep learning applications by end users in novel areas.

A Virtual Reality Tool for Representing, Visualizing and Updating Deep Learning Models最先出现在Nweon Paper。

Teams: Technical University of Munich;University of Tubingen; University of Florida

Writers: Efe Bozkir, Süleyman Özdel, Mengdi Wang, Brendan David-John, Hong Gao, Kevin Butler, Eakta Jain, Enkelejda Kasneci

PDF: Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges

Abstract

Latest developments in computer hardware, sensor technologies, and artificial intelligence can make virtual reality (VR) and virtual spaces an important part of human everyday life. Eye tracking offers not only a hands-free way of interaction but also the possibility of a deeper understanding of human visual attention and cognitive processes in VR. Despite these possibilities, eye-tracking data also reveal privacy-sensitive attributes of users when it is combined with the information about the presented stimulus. To address these possibilities and potential privacy issues, in this survey, we first cover major works in eye tracking, VR, and privacy areas between the years 2012 and 2022. While eye tracking in the VR part covers the complete pipeline of eye-tracking methodology from pupil detection and gaze estimation to offline use and analyses, as for privacy and security, we focus on eye-based authentication as well as computational methods to preserve the privacy of individuals and their eye-tracking data in VR. Later, taking all into consideration, we draw three main directions for the research community by mainly focusing on privacy challenges. In summary, this survey provides an extensive literature review of the utmost possibilities with eye tracking in VR and the privacy implications of those possibilities.

Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges最先出现在Nweon Paper。

Teams: Tsinghua University;University of Science and Technology Beijing

Writers: Jingyuan Zhu, Huimin Ma, Jiansheng Chen, Jian Yuan

PDF: Few-shot 3D Shape Generation

Abstract

Realistic and diverse 3D shape generation is helpful for a wide variety of applications such as virtual reality, gaming, and animation. Modern generative models, such as GANs and diffusion models, learn from large-scale datasets and generate new samples following similar data distributions. However, when training data is limited, deep neural generative networks overfit and tend to replicate training samples. Prior works focus on few-shot image generation to produce high-quality and diverse results using a few target images. Unfortunately, abundant 3D shape data is typically hard to obtain as well. In this work, we make the first attempt to realize few-shot 3D shape generation by adapting generative models pre-trained on large source domains to target domains using limited data. To relieve overfitting and keep considerable diversity, we propose to maintain the probability distributions of the pairwise relative distances between adapted samples at feature-level and shape-level during domain adaptation. Our approach only needs the silhouettes of few-shot target samples as training data to learn target geometry distributions and achieve generated shapes with diverse topology and textures. Moreover, we introduce several metrics to evaluate the quality and diversity of few-shot 3D shape generation. The effectiveness of our approach is demonstrated qualitatively and quantitatively under a series of few-shot 3D shape adaptation setups.

Few-shot 3D Shape Generation最先出现在Nweon Paper。

Teams: Tsinghua University；University of Maryland；The University of Tokyo；Pingan Group

Writers: Tianhe Wu, Shuwei Shi, Haoming Cai, Mingdeng Cao, Jing Xiao, Yinqiang Zheng, Yujiu Yang

PDF: Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment

Abstract

Blind Omnidirectional Image Quality Assessment (BOIQA) aims to objectively assess the human perceptual quality of omnidirectional images (ODIs) without relying on pristine-quality image information. It is becoming more significant with the increasing advancement of virtual reality (VR) technology. However, the quality assessment of ODIs is severely hampered by the fact that the existing BOIQA pipeline lacks the modeling of the observer’s browsing process. To tackle this issue, we propose a novel multi-sequence network for BOIQA called Assessor360, which is derived from the realistic multi-assessor ODI quality assessment procedure. Specifically, we propose a generalized Recursive Probability Sampling (RPS) method for the BOIQA task, combining content and detailed information to generate multiple pseudo viewport sequences from a given starting point. Additionally, we design a Multi-scale Feature Aggregation (MFA) module with Distortion-aware Block (DAB) to fuse distorted and semantic features of each viewport. We also devise TMM to learn the viewport transition in the temporal domain. Extensive experimental results demonstrate that Assessor360 outperforms state-of-the-art methods on multiple OIQA datasets.

Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment最先出现在Nweon Paper。

Teams: Massachusetts Institute of Technology；Rice University

Writers: Kushagra Tiwary, Akshat Dave, Nikhil Behari, Tzofi Klinghoffer, Ashok Veeraraghavan, Ramesh Raskar

PDF: ORCa: Glossy Objects as Radiance Field Cameras

Abstract

Reflections on glossy objects contain valuable and hidden information about the surrounding environment. By converting these objects into cameras, we can unlock exciting applications, including imaging beyond the camera’s field-of-view and from seemingly impossible vantage points, e.g. from reflections on the human eye. However, this task is challenging because reflections depend jointly on object geometry, material properties, the 3D environment, and the observer viewing direction. Our approach converts glossy objects with unknown geometry into radiance-field cameras to image the world from the object’s perspective. Our key insight is to convert the object surface into a virtual sensor that captures cast reflections as a 2D projection of the 5D environment radiance field visible to the object. We show that recovering the environment radiance fields enables depth and radiance estimation from the object to its surroundings in addition to beyond field-of-view novel-view synthesis, i.e. rendering of novel views that are only directly-visible to the glossy object present in the scene, but not the observer. Moreover, using the radiance field we can image around occluders caused by close-by objects in the scene. Our method is trained end-to-end on multi-view images of the object and jointly estimates object geometry, diffuse radiance, and the 5D environment radiance field.

ORCa: Glossy Objects as Radiance Field Cameras最先出现在Nweon Paper。

Teams: Tsinghua University, 2Renmin University of China, 3ShengShu

Writers: Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu

PDF: ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Abstract

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512×512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page: this https URL

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation最先出现在Nweon Paper。

Teams: Brown University;Mitsubishi Electric Research Labs

Writers: Eric Rosen, Devesh K. Jha

PDF: A Virtual Reality Teleoperation Interface for Industrial Robot Manipulators

Abstract

We address the problem of teleoperating an industrial robot manipulator via a commercially available Virtual Reality (VR) interface. Previous works on VR teleoperation for robot manipulators focus primarily on collaborative or research robot platforms (whose dynamics and constraints differ from industrial robot arms), or only address tasks where the robot’s dynamics are not as important (e.g: pick and place tasks). We investigate the usage of commercially available VR interfaces for effectively teleoeprating industrial robot manipulators in a variety of contact-rich manipulation tasks. We find that applying standard practices for VR control of robot arms is challenging for industrial platforms because torque and velocity control is not exposed, and position control is mediated through a black-box controller. To mitigate these problems, we propose a simplified filtering approach to process command signals to enable operators to effectively teleoperate industrial robot arms with VR interfaces in dexterous manipulation tasks. We hope our findings will help robot practitioners implement and setup effective VR teleoperation interfaces for robot manipulators. The proposed method is demonstrated on a variety of contact-rich manipulation tasks which can also involve very precise movement of the robot during execution (videos can be found at this https URL)

A Virtual Reality Teleoperation Interface for Industrial Robot Manipulators最先出现在Nweon Paper。

Teams: Carnegie Mellon University

Writers: Abraham George, Alison Bartsch, Amir Barati Farimani

PDF: OpenVR: Teleoperation for Manipulation

Abstract

Across the robotics field, quality demonstrations are an integral part of many control pipelines. However, collecting high-quality demonstration trajectories remains time-consuming and difficult, often resulting in the number of demonstrations being the performance bottleneck. To address this issue, we present a method of Virtual Reality (VR) Teleoperation that uses an Oculus VR headset to teleoperate a Franka Emika Panda robot. Although other VR teleoperation methods exist, our code is open source, designed for readily available consumer hardware, easy to modify, agnostic to experimental setup, and simple to use.

OpenVR: Teleoperation for Manipulation最先出现在Nweon Paper。

Teams: The Hong Kong University of Science and Technology；

Writers: Jiadong Yu, Ahmad Alhilal, Tailin Zhou, Pan Hui, Danny H.K. Tsang

PDF: Attention-based QoE-aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality

Abstract

Metaverse applications such as virtual reality (VR) content streaming, require optimal resource allocation strategies for mobile edge computing (MEC) to ensure a high-quality user experience. In contrast to online reinforcement learning (RL) algorithms, which can incur substantial communication overheads and longer delays, the majority of existing works employ offline-trained RL algorithms for resource allocation decisions in MEC systems. However, they neglect the impact of desynchronization between the physical and digital worlds on the effectiveness of the allocation strategy. In this paper, we tackle this desynchronization using a continual RL framework that facilitates the resource allocation dynamically for MEC-enabled VR content streaming. We first design a digital twin-empowered edge computing (DTEC) system and formulate a quality of experience (QoE) maximization problem based on attention-based resolution perception. This problem optimizes the allocation of computing and bandwidth resources while adapting the attention-based resolution of the VR content. The continual RL framework in DTEC enables adaptive online execution in a time-varying environment. The reward function is defined based on the QoE and horizon-fairness QoE (hfQoE) constraints. Furthermore, we propose freshness prioritized experience replay – continual deep deterministic policy gradient (FPER-CDDPG) to enhance the performance of continual learning in the presence of time-varying DT updates. We test FPER-CDDPG using extensive experiments and evaluation. FPER-CDDPG outperforms the benchmarks in terms of average latency, QoE, and successful delivery rate as well as meeting the hfQoE requirements and performance over long-term execution while ensuring system scalability with the increasing number of users.

Attention-based QoE-aware Digital Twin Empowered Edge Computing for Immersive Virtual Reality最先出现在Nweon Paper。

Teams: University of Science and Technology of China

Writers: Yuang Chen, Hancheng Lu, Langtian Qin, Chang Wu, Chang Wen Chen

PDF: Streaming 360-degree VR Video with Statistical QoS Provisioning in mmWave Networks from Delay and Rate Perspectives

Abstract

Millimeter-wave(mmWave) technology has emerged as a promising enabler for unleashing the full potential of 360-degree virtual reality (VR). However, the explosive growth of VR services, coupled with the reliability issues of mmWave communications, poses enormous challenges in terms of wireless resource and quality-of-service (QoS) provisioning for mmWave-enabled 360-degree VR. In this paper, we propose an innovative 360-degree VR streaming architecture that addresses three under-exploited issues: overlapping field-of-views (FoVs), statistical QoS provisioning (SQP), and loss-tolerant active data discarding. Specifically, an overlapping FoV-based optimal joint unicast and multicast (JUM) task assignment scheme is designed to implement the non-redundant task assignments, thereby conserving wireless resources remarkably. Furthermore, leveraging stochastic network calculus, we develop a comprehensive SQP theoretical framework that encompasses two SQP schemes from delay and rate perspectives. Additionally, a corresponding optimal adaptive joint time-slot allocation and active-discarding (ADAPT-JTAAT) transmission scheme is proposed to minimize resource consumption while guaranteeing diverse statistical QoS requirements under loss-intolerant and loss-tolerant scenarios from delay and rate perspectives, respectively. Extensive simulations demonstrate the effectiveness of the designed overlapping FoV-based JUM optimal task assignment scheme. Comparisons with six baseline schemes validate that the proposed optimal ADAPTJTAAT transmission scheme can achieve superior SQP performance in resource utilization, flexible rate control, and robust queue behaviors.

Streaming 360-degree VR Video with Statistical QoS Provisioning in mmWave Networks from Delay and Rate Perspectives最先出现在Nweon Paper。

Teams: Jasmine Roberts

Writers: Jasmine Roberts

PDF: The AR/VR Technology Stack: A Central Repository of Software Development Libraries, Platforms, and Tools

Abstract

A comprehensive repository of software development libraries, platforms, and tools specifically to the domains of augmented, virtual, and mixed reality.

The AR/VR Technology Stack: A Central Repository of Software Development Libraries, Platforms, and Tools最先出现在Nweon Paper。

Teams: Samuel Greengard

Writers: Samuel Greengard

PDF: Better Algorithms through Faster Math

Abstract

The search for new algorithms that could reduce the time needed for multiplication is now at the center of data science.

Better Algorithms through Faster Math最先出现在Nweon Paper。

Teams: KTH Royal Institute of Technology

Writers: Marco Moletta, Maciej K. Wozniak, Michael C. Welle, Danica Kragic

PDF: A Virtual Reality Framework for Human-Robot Collaboration in Cloth Folding

Abstract

We present a virtual reality (VR) framework to automate the data collection process in cloth folding tasks. The framework uses skeleton representations to help the user define the folding plans for different classes of garments, allowing for replicating the folding on unseen items of the same class. We evaluate the framework in the context of automating garment folding tasks. A quantitative analysis is performed on 3 classes of garments, demonstrating that the framework reduces the need for intervention by the user. We also compare skeleton representations with RGB and binary images in a classification task on a large dataset of clothing items, motivating the use of the framework for other classes of garments.

A Virtual Reality Framework for Human-Robot Collaboration in Cloth Folding最先出现在Nweon Paper。

Teams: Nano-Institute Munich;Universidade Federal de Pernambuco;Ludwig-Maximilians-Universität München,;Monash University;Imperial College London

Writers: Wenzheng Lu, Leonarde de S. Menezes, Andreas Tittl, Haoran Ren, Stefan A. Maier

PDF: Active Huygens’ metasurface based on in-situ grown conductive polymer

Abstract

Active metasurfaces provide unique advantages for on-demand light manipulation at a subwavelength scale for emerging applications of 3D displays, augmented/virtual reality (AR/VR) glasses, holographic projectors and light detection and ranging (LiDAR). These applications put stringent requirements on switching speed, cycling duration, controllability over intermediate states, modulation contrast, optical efficiency and operation voltages. However, previous demonstrations focus only on particular subsets of these key performance requirements for device implementation, while the other performance metrics have remained too low for any practical use. Here, we demonstrate an active Huygens’ metasurface based on in-situ grown conductive polymer with holistic switching performance, including switching speed of 60 frames per second (fps), switching duration of more than 2000 switching cycles without noticeable degradation, hysteresis-free controllability over intermediate states, modulation contrast of over 1400%, optical efficiency of 28% and operation voltage range within 1 V. Our active metasurface design meets all foundational requirements for display applications and can be readily incorporated into other metasurface concepts to deliver high-reliability electrical control over its optical response, paving the way for compact and robust electro-optic metadevices.

Active Huygens’ metasurface based on in-situ grown conductive polymer最先出现在Nweon Paper。

Teams: University of California;Meta,;Georgia Institute of Technology;Harvard University,

Writers: Hyoukjun Kwon, Krishnakumar Nair, Jamin Seo, Jason Yik, Debabrata Mohapatra, Dongyuan Zhan, Jinook Song, Peter Capak, Peizhao Zhang, Peter Vajda, Colby Banbury, Mark Mazumder, Liangzhen Lai, Ashish Sirasao, Tushar Krishna, Harshit Khaitan, Vikas Chandra, Vijay Janapa Reddi

PDF: XRBench: An Extended Reality Machine Learning Benchmark Suite for the Metaverse

Abstract

Real-time multi-task multi-model (MTMM) workloads, a new form of deep learning inference workloads, are emerging for applications areas like extended reality (XR) to support metaverse use cases. These workloads combine user interactivity with computationally complex machine learning (ML) activities. Compared to standard ML applications, these ML workloads present unique difficulties and constraints. Real-time MTMM workloads impose heterogeneity and concurrency requirements on future ML systems and devices, necessitating the development of new capabilities. This paper begins with a discussion of the various characteristics of these real-time MTMM ML workloads and presents an ontology for evaluating the performance of future ML hardware for XR systems. Next, we present XRBENCH, a collection of MTMM ML tasks, models, and usage scenarios that execute these models in three representative ways: cascaded, concurrent, and cascaded-concurrent for XR use cases. Finally, we emphasize the need for new metrics that capture the requirements properly. We hope that our work will stimulate research and lead to the development of a new generation of ML systems for XR use cases. XRBench is available as an open-source project: https://github.com/XRBench

XRBench: An Extended Reality Machine Learning Benchmark Suite for the Metaverse最先出现在Nweon Paper。

Teams: Meta Reality Labs 2The University of Tokyo

Writers: Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, Cem Keskin

PDF: AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation

Abstract

We present AssemblyHand, a large-scale benchmark dataset with accurate 3D hand pose annotations, to facilitate the study of egocentric activities with challenging hand- object interactions. The dataset includes synchronized ego- centric and exocentric images sampled from the recent As- sembly101 dataset, in which participants assemble and dis- assemble take-apart toys. To obtain high-quality 3D hand pose annotations for the egocentric images, we develop an efficient pipeline, where we use an initial set of manual annotations to train a model to automatically annotate a much larger dataset. Our annotation model uses multi-view feature fusion and an iterative refinement scheme, and achieves an average keypoint error of 4.20 mm, which is 85% lower than the error of the original annotations in Assembly101. AssemblyHands provides 3.0M annotated images, including 490K egocentric images, making it the largest existing benchmark dataset for egocentric 3D hand pose estimation. Using this data, we develop a strong single-view baseline of 3D hand pose estimation from egocentric images. Furthermore, we design a novel action classification task to evaluate predicted 3D hand poses. Our study shows that having higher-quality hand poses directly improves the ability to recognize actions.

AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation最先出现在Nweon Paper。

Teams: imec, Sense and Actuate Technologies, Leuven, 3001, Belgium；KU Leuven,；Meta Reality Labs

Writers: T.C. Huang, Kris Myny, Marc Ameys, Mauricio Velazquez Lopez, Nikolaos Papadopoulos

PDF: 3.55-Watt Output Power LTPS TFT DCDC converter for actuators on Wearable devices on flexible Substrate

Abstract

In this work, a high-power LTPS DCDC converter on flexible polyimide substrate is demonstrated. The diode-based boost- converter upconverts its output voltage to 9V from a 3.3V source and can reliably supply a load range of 100mA to 400mA. The efficiency of the design is reaching 54.9%. The design contains 400.000 25μm/4.5μm TFTs, resulting in a complete footprint of 10.02cm2. The DCDC converter is suitable for driving high-power actuators on flexible wearables devices. Moreover, this work reports 3.24 ampere current through the LTPS TFT in the DCDC converter resulting in sub-Ω ON-resistance.

3.55-Watt Output Power LTPS TFT DCDC converter for actuators on Wearable devices on flexible Substrate最先出现在Nweon Paper。

Teams: Delft University of Technology

Writers: Mubariz Zaffar; Liangliang Nan; Julian Francisco Pieter Kooij

PDF: CoPR: Toward Accurate Visual Localization With Continuous Place-Descriptor Regression

Abstract

Visual place recognition (VPR) is an image-based localization method that estimates the camera location of a query image by retrieving the most similar reference image from a map of geo-tagged reference images. In this work, we look into two fundamental bottlenecks for its localization accuracy: 1) reference map sparseness and 2) viewpoint invariance. First, the reference images for VPR are only available at sparse poses in a map, which enforces an upper bound on the maximum achievable localization accuracy through VPR. We, therefore, propose Continuous Place-descriptor Regression (CoPR) to densify the map and improve localization accuracy. We study various interpolation and extrapolation models to regress additional VPR feature descriptors from only the existing references. Second, we compare different feature encoders and show that CoPR presents value for all of them. We evaluate our models on three existing public datasets and report on average around 30% improvement in VPR-based localization accuracy using CoPR, on top of the 15% increase by using a viewpoint-variant loss for the feature encoder. The complementary relation between CoPR and relative pose estimation is also discussed.

CoPR: Toward Accurate Visual Localization With Continuous Place-Descriptor Regression最先出现在Nweon Paper。

Teams: 1Niantic 2University of Edinburgh 3UCL

Writers: Jamie Watson, Mohamed Sayed, Zawar Qureshi, Gabriel J. Brostow, Sara Vicente, Oisin Mac Aodha, Michael Firman

PDF: Virtual Occlusions Through Implicit Depth

Abstract

For augmented reality (AR), it is important that virtual assets appear to `sit among’ real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer’s camera moves. Unfortunately, small mistakes in the estimated scene depth can ruin the downstream occlusion mask, and thereby the AR illusion. Especially in real-time settings, depths inferred near boundaries or across time can be inconsistent. In this paper, we challenge the need for depth-regression as an intermediate step.
We instead propose an implicit model for depth and use that to predict the occlusion mask directly. The inputs to our network are one or more color images, plus the known depths of any virtual geometry. We show how our occlusion predictions are more accurate and more temporally stable than predictions derived from traditional depth-estimation models. We obtain state-of-the-art occlusion results on the challenging ScanNetv2 dataset and superior qualitative results on real scenes.

Virtual Occlusions Through Implicit Depth最先出现在Nweon Paper。

Teams: Stanford University;Cornell University

Writers: Kyle T. Yoshida, Joel X. Kiernan, Rachel A. G. Adenekan, Steven H. Trinh, Alexis J. Lowber, Allison M. Okamura, Cara M. Nunez

PDF: Cognitive and Physical Activities Impair Perception of Smartphone Vibrations

Abstract

Vibration feedback is common in everyday devices, from virtual reality systems to smartphones. However, cognitive and physical activities may impede our ability to sense vibrations from devices. In this study, we develop and characterize a smartphone platform to investigate how a shape-memory task (cognitive activity) and walking (physical activity) impair human perception of smartphone vibrations. We measured how Apple’s Core Haptics Framework parameters can be used for haptics research, namely how hapticIntensity modulates amplitudes of 230 Hz vibrations. A 23-person user study found that physical (p<0.001) and cognitive (p=0.004) activity increase vibration perception thresholds. Cognitive activity also increases vibration response time (p<0.001). This work also introduces a smartphone platform that can be used for out-of-lab vibration perception testing. Researchers can use our smartphone platform and results to design better haptic devices for diverse, unique populations.

Cognitive and Physical Activities Impair Perception of Smartphone Vibrations最先出现在Nweon Paper。

Teams: University of Alberta

Writers: Bhavya Sehgal, Vaishnavi Mendu, Aparna Mendu

PDF: Realization RGBD Image Stylization

Abstract

This research paper explores the application of style transfer in computer vision using RGB images and their corresponding depth maps. We propose a novel method that incorporates the depth map and a heatmap of the RGB image to generate more realistic style transfer results. We compare our method to the traditional neural style transfer approach and find that our method outperforms it in terms of producing more realistic color and style. The proposed method can be applied to various computer vision applications, such as image editing and virtual reality, to improve the realism of generated images. Overall, our findings demonstrate the potential of incorporating depth information and heatmap of RGB images in style transfer for more realistic results.

Realization RGBD Image Stylization最先出现在Nweon Paper。

Teams: Tampere University

Writers: Erfan Momeni Yazdi, Markku Mäkitalo, Julius Ikkala, Pekka Jääskeläinen

PDF: TauBench 1.1: A Dynamic Benchmark for Graphics Rendering

Abstract

Many graphics rendering algorithms used in both real-time games and virtual reality applications can get performance boosts by temporally reusing previous computations. However, algorithms based on temporal reuse are typically measured using trivial benchmarks with very limited dynamic features. To this end, in [1] we presented TauBench 1.0, a benchmark designed to stress temporal reuse algorithms. Now, we release TauBench version 1.1, which improves the usability of the original benchmark. In particular, these improvements reduce the size of the dataset significantly, resulting in faster loading and rendering times, and in better compatibility with 3D software that impose strict size limits for the scenes.

TauBench 1.1: A Dynamic Benchmark for Graphics Rendering最先出现在Nweon Paper。

Teams: VNU University of Engineering and Technology

Writers: Hoang-Anh Phan, Phuc Vinh Nguyen, Thu Hang Thi Khuat, Hieu Dang Van, Dong Huu Quoc Tran, Bao Lam Dang, Tung Thanh Bui, Van Nguyen Thi Thanh, Trinh Chu Duc

PDF: A sensor fusion approach for improving implementation speed and accuracy of RTAB-Map algorithm based indoor 3D mapping

Abstract

In recent years, 3D mapping for indoor environments has undergone considerable research and improvement because of its effective applications in various fields, including robotics, autonomous navigation, and virtual reality. Building an accurate 3D map for indoor environment is challenging due to the complex nature of the indoor space, the problem of real-time embedding and positioning errors of the robot system. This study proposes a method to improve the accuracy, speed, and quality of 3D indoor mapping by fusing data from the Inertial Measurement System (IMU) of the Intel Realsense D435i camera, the Ultrasonic-based Indoor Positioning System (IPS), and the encoder of the robot’s wheel using the extended Kalman filter (EKF) algorithm. The merged data is processed using a Real-time Image Based Mapping algorithm (RTAB-Map), with the processing frequency updated in synch with the position frequency of the IPS device. The results suggest that fusing IMU and IPS data significantly improves the accuracy, mapping time, and quality of 3D maps. Our study highlights the proposed method’s potential to improve indoor mapping in various fields, indicating that the fusion of multiple data sources can be a valuable tool in creating high-quality 3D indoor maps.

A sensor fusion approach for improving implementation speed and accuracy of RTAB-Map algorithm based indoor 3D mapping最先出现在Nweon Paper。

Teams: Heewoo Jun, Alex Nichol

Writers: Heewoo Jun, Alex Nichol

PDF: Shap-E: Generating Conditional 3D Implicit Functions

Abstract

We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at this https URL.

Shap-E: Generating Conditional 3D Implicit Functions最先出现在Nweon Paper。