Teams: New York University

Writers: Sonia Castelo, Joao Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, Jing Qian, Kyunghyun Cho, He He, Qi Sun, Huy Vo, Juan Bello, Michael Krone, Claudio Silva

PDF: ARGUS: Visualization of AI-Assisted Task Guidance in AR

Abstract

The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine tune the components of the AR assistant.

ARGUS: Visualization of AI-Assisted Task Guidance in AR最先出现在Nweon Paper。

Teams: Rice University

Writers: Barclay Jumet 1, Zane A. Zook 1, Anas Yousaf 1, Anoop Rajappan 1, Doris Xu 1, Te Faye Yap 1, Nathaniel Fino 1, Zhen Liu 1, Marcia K. O’Malley 1 2 3, Daniel J. Preston

PDF: Fluidically programmed wearable haptic textiles

Abstract

Haptic feedback offers a useful mode of communication in visually or auditorily noisy environments. The adoption of haptic devices in our everyday lives, however, remains limited, motivating research on haptic wearables constructed from materials that enable comfortable and lightweight form factors. Textiles, a material class fitting these needs and already ubiquitous in clothing, have begun to be used in haptics, but reliance on arrays of electromechanical controllers detracts from the benefits that textiles offer. Here, we mitigate the requirement for bulky hardware by developing a class of wearable haptic textiles capable of delivering high-resolution information on the basis of embedded fluidic programming. The designs of these haptic textiles enable tailorable amplitudinal, spatial, and temporal control. Combining these capabilities, we demonstrate wearables that deliver spatiotemporal cues in four directions with an average user accuracy of 87%. Subsequent demonstrations of washability, repairability, and utility for navigational tasks exemplify the capabilities of our approach.

Fluidically programmed wearable haptic textiles最先出现在Nweon Paper。

Teams: University of Central Florida;Goertek Electronics

Writers: Zhenyi Luo, Yannanqi Li, John Semmen, Yi Rao & Shin-Tson Wu

PDF: Achromatic diffractive liquid-crystal optics for virtual reality displays

Abstract

Diffractive liquid-crystal optics is a promising optical element for virtual reality (VR) and mixed reality as it provides an ultrathin formfactor and lightweight for human factors and ergonomics. However, its severe chromatic aberrations impose a big challenge for full-color display applications. In this study, we demonstrate an achromatic diffractive liquid-crystal device to overcome this longstanding chromatic aberration issue. The proposed device consists of three stacked diffractive liquid crystal optical elements with specifically designed spectral response and polarization selectivity. The concept is validated by both simulations and experiments. Our experimental results show a significant improvement in imaging performance with two types of light engines: a laser projector and an organic light-emitting diode display panel. In addition, our simulation results indicate that the lateral color shift is reduced by ~100 times in comparison with conventional broadband diffractive liquid-crystal lens. Potential applications for VR-enabled metaverse, spatial computing, and digital twins that have found widespread applications in smart tourism, smart education, smart healthcare, smart manufacturing, and smart construction are foreseeable.

Achromatic diffractive liquid-crystal optics for virtual reality displays最先出现在Nweon Paper。

Teams: University of Colorado Boulder；University of Wyoming

Writers: Rishi Vanukuru, Suibi Che-Chuan Weng, Krithik Ranjan, Torin Hopkins, Amy Banic, Mark D. Gross, Ellen Yi-Luen Do

PDF: DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented Reality

Abstract

In-person human interaction relies on our spatial perception of each other and our surroundings. Current remote communication tools partially address each of these aspects. Video calls convey real user representations but without spatial interactions. Augmented and Virtual Reality (AR/VR) experiences are immersive and spatial but often use virtual environments and characters instead of real-life representations. Bridging these gaps, we introduce DualStream, a system for synchronous mobile AR remote communication that captures, streams, and displays spatial representations of users and their surroundings. DualStream supports transitions between user and environment representations with different levels of visuospatial fidelity, as well as the creation of persistent shared spaces using environment snapshots. We demonstrate how DualStream can enable spatial communication in real-world contexts, and support the creation of blended spaces for collaboration. A formative evaluation of DualStream revealed that users valued the ability to interact spatially and move between representations, and could see DualStream fitting into their own remote communication practices in the near future. Drawing from these findings, we discuss new opportunities for designing more widely accessible spatial communication tools, centered around the mobile phone.

DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented Reality最先出现在Nweon Paper。

Teams: Dipesh Gyawali

Writers: Dipesh Gyawali

PDF: Mixed Reality: The Interface of the Future

Abstract

The world is slowly moving towards everything being simulated digitally and virtually. Mixed Reality (MR) is the amalgam of the real world with virtual stimuli. It has great prospects in the future in terms of various applications additionally with some challenges. This paper focuses on how Mixed Reality could be used in the future along with the challenges that could arise. Several application areas along with the potential benefits are studied in this research. Three research questions are proposed, analyzed, and concluded through the experiments. While the availability of MR devices could introduce a lot of potential, specific challenges need to be scrutinized by the developers and manufacturers. Overall, MR technology has a chance to enhance personalized, supportive, and interactive experiences for human lives.

Mixed Reality: The Interface of the Future最先出现在Nweon Paper。

Teams: Queen’s University；University of Calgary

Writers: Ahmad M. Nagib, Hatem Abou-Zeid, Hossam S. Hassanein

PDF: How Does Forecasting Affect the Convergence of DRL Techniques in O-RAN Slicing?

Abstract

The success of immersive applications such as virtual reality (VR) gaming and metaverse services depends on low latency and reliable connectivity. To provide seamless user experiences, the open radio access network (O-RAN) architecture and 6G networks are expected to play a crucial role. RAN slicing, a critical component of the O-RAN paradigm, enables network resources to be allocated based on the needs of immersive services, creating multiple virtual networks on a single physical infrastructure. In the O-RAN literature, deep reinforcement learning (DRL) algorithms are commonly used to optimize resource allocation. However, the practical adoption of DRL in live deployments has been sluggish. This is primarily due to the slow convergence and performance instabilities suffered by the DRL agents both upon initial deployment and when there are significant changes in network conditions. In this paper, we investigate the impact of time series forecasting of traffic demands on the convergence of the DRL-based slicing agents. For that, we conduct an exhaustive experiment that supports multiple services including real VR gaming traffic. We then propose a novel forecasting-aided DRL approach and its respective O-RAN practical deployment workflow to enhance DRL convergence. Our approach shows up to 22.8%, 86.3%, and 300% improvements in the average initial reward value, convergence rate, and number of converged scenarios respectively, enhancing the generalizability of the DRL agents compared with the implemented baselines. The results also indicate that our approach is robust against forecasting errors and that forecasting models do not have to be ideal.

How Does Forecasting Affect the Convergence of DRL Techniques in O-RAN Slicing?最先出现在Nweon Paper。

Teams: Australian National University

Writers: Xingyu Fu, Mingze Xi

PDF: Typing on Any Surface: A Deep Learning-based Method for Real-Time Keystroke Detection in Augmented Reality

Abstract

Frustrating text entry interface has been a major obstacle in participating in social activities in augmented reality (AR). Popular options, such as mid-air keyboard interface, wireless keyboards or voice input, either suffer from poor ergonomic design, limited accuracy, or are simply embarrassing to use in public. This paper proposes and validates a deep-learning based approach, that enables AR applications to accurately predict keystrokes from the user perspective RGB video stream that can be captured by any AR headset. This enables a user to perform typing activities on any flat surface and eliminates the need of a physical or virtual keyboard. A two-stage model, combing an off-the-shelf hand landmark extractor and a novel adaptive Convolutional Recurrent Neural Network (C-RNN), was trained using our newly built dataset. The final model was capable of adaptive processing user-perspective video streams at ~32 FPS. This base model achieved an overall accuracy of 91.05% when typing 40 Words per Minute (wpm), which is how fast an average person types with two hands on a physical keyboard. The Normalised Levenshtein Distance also further confirmed the real-world applicability of that our approach. The promising results highlight the viability of our approach and the potential for our method to be integrated into various applications. We also discussed the limitations and future research required to bring such technique into a production system.

Typing on Any Surface: A Deep Learning-based Method for Real-Time Keystroke Detection in Augmented Reality最先出现在Nweon Paper。

Teams: Duke University

Writers: Tim Scargill, Ying Chen, Tianyi Hu, Maria Gorlatova

PDF: SiTAR: Situated Trajectory Analysis for In-the-Wild Pose Error Estimation

Abstract

Virtual content instability caused by device pose tracking error remains a prevalent issue in markerless augmented reality (AR), especially on smartphones and tablets. However, when examining environments which will host AR experiences, it is challenging to determine where those instability artifacts will occur; we rarely have access to ground truth pose to measure pose error, and even if pose error is available, traditional visualizations do not connect that data with the real environment, limiting their usefulness. To address these issues we present SiTAR (Situated Trajectory Analysis for Augmented Reality), the first situated trajectory analysis system for AR that incorporates estimates of pose tracking error. We start by developing the first uncertainty-based pose error estimation method for visual-inertial simultaneous localization and mapping (VI-SLAM), which allows us to obtain pose error estimates without ground truth; we achieve an average accuracy of up to 96.1% and an average F1 score of up to 0.77 in our evaluations on four VI-SLAM datasets. Next we present our SiTAR system, implemented for ARCore devices, combining a backend that supplies uncertainty-based pose error estimates with a frontend that generates situated trajectory visualizations. Finally, we evaluate the efficacy of SiTAR in realistic conditions by testing three visualization techniques in an in-the-wild study with 15 users and 13 diverse environments; this study reveals the impact both environment scale and the properties of surfaces present can have on user experience and task performance.

SiTAR: Situated Trajectory Analysis for In-the-Wild Pose Error Estimation最先出现在Nweon Paper。

Teams: Stanford University

Writers: Tom Van Wouwe, Seunghwan Lee, Antoine Falisse, Scott Delp, C. Karen Liu

PDF: Diffusion Inertial Poser: Human Motion Reconstruction from Arbitrary Sparse IMU Configurations

Abstract

Motion capture from a limited number of inertial measurement units (IMUs) has important applications in health, human performance, and virtual reality. Real-world limitations and application-specific goals dictate different IMU configurations (i.e., number of IMUs and chosen attachment body segments), trading off accuracy and practicality. Although recent works were successful in accurately reconstructing whole-body motion from six IMUs, these systems only work with a specific IMU configuration. Here we propose a single diffusion generative model, Diffusion Inertial Poser (DiffIP), which reconstructs human motion in real-time from arbitrary IMU configurations. We show that DiffIP has the benefit of flexibility with respect to the IMU configuration while being as accurate as the state-of-the-art for the commonly used six IMU configuration. Our system enables selecting an optimal configuration for different applications without retraining the model. For example, when only four IMUs are available, DiffIP found that the configuration that minimizes errors in joint kinematics instruments the thighs and forearms. However, global translation reconstruction is better when instrumenting the feet instead of the thighs. Although our approach is agnostic to the underlying model, we built DiffIP based on physiologically realistic musculoskeletal models to enable use in biomedical research and health applications.

Diffusion Inertial Poser: Human Motion Reconstruction from Arbitrary Sparse IMU Configurations最先出现在Nweon Paper。

Teams: Innolux Corp;Coretronic Corp

Writers: Yung-Hsun Wu, Chuan-Chung Chang, Yu-Shih Tsou, Yi-Chien Lo, Chia-Hao Tsai, Chih-hung Lu, Chiu-Lien Yang, Fu-Ming Chuang

PDF: Enhancing virtual reality with high-resolution light field liquid crystal display technology

Abstract

An improved light field VR display is introduced, featuring an ultra-high-pixel density liquid crystal display with a resolution of 3.1 in. and 1411 PPI. By utilizing a 3K3K resolution display panel, this optimized display enhances the field of view and provides a more immersive experience. Furthermore, it incorporates advanced functions, such as vision correction (without the need for glasses), reduced vergence-accommodation conflict, and an enlarged eye box with eye tracking technology.

Enhancing virtual reality with high-resolution light field liquid crystal display technology最先出现在Nweon Paper。

Teams: Qurv Technologies,；ICFO at Barcelona Institute of Science and Technology

Writers: Gabriel Mercier, Emre O. Polat, Shengtai Shi, Shuchi Gupta, Gerasimos Konstantatos, Stijn Goossens, and Frank H. L. Koppens

PDF: Semitransparent Image Sensors for Eye-Tracking Applications

Abstract

Due to their ability to capture vast amounts of information, traditional image sensors play a pivotal role in today’s society. However, the opaque nature of both their pixels and stacked read-out electronics can be a limiting factor in applications such as human–computer interfaces, smart displays, and both augmented and virtual reality. In this paper, we present the development and analysis of the first semitransparent image sensor and its applicability as an eye-tracking device. Consisting of an 8 × 8 array of semitransparent photodetectors and electrodes deposited on a fully transparent substrate, the sensor’s pixels achieve an optical transparency of 85–95% and high sensitivity, with more than 90% of these pixels demonstrating a noise equivalent irradiance <10–4 W/m2 for wavelengths of 637 nm. The fabrication of such sensors represents a fundamental shift in how we think about cameras and imaging as these devices can be concealed in plain sight.

Semitransparent Image Sensors for Eye-Tracking Applications最先出现在Nweon Paper。

Teams: Fraunhofer Institute

Writers: Uwe Vogel, Philipp Wartenberg, Bernd Richter, Stephan Brenner, Judith Baumgarten, Michael Thomschke, Karsten Fehse, Olaf Hild

PDF: Near-to-eye displays with embedded eye-tracking by bi-directional OLED microdisplay

Abstract

Near-to-eye (NTE) projection is the major approach to “Smart Glasses”, which have gained lot of traction during the last few years. Micro-displays based on organic light-emitting diodes (OLEDs) achieve high optical performance with excellent contrast ratio and large dynamic range at low power consumption, making them suitable for such application. In state-of-the-art applications the micro-display typically acts as a purely unidirectional output device. With the integration of an additional image sensor, the functionality of the micro-display can be extended to a bidirectional optical input/output device, aiming for implementation of eye-tracking capabilities in see-through (ST-)NTE applications to achieve gaze-based human-display-interaction. This paper describes a new bi-directional OLED microdisplay featuring SVGA resolution for both image display and acquisition, and its implementation with see-through NTE optics.

Near-to-eye displays with embedded eye-tracking by bi-directional OLED microdisplay最先出现在Nweon Paper。

Teams: University of British Columbia ；University of Windsor

Writers: Qingxuan Feng, Peng Yang, Zhixuan Huang, Jiayin Chen, Ning Zhang

PDF: Deadline Aware Two-Timescale Resource Allocation for VR Video Streaming

Abstract

In this paper, we investigate resource allocation problem in the context of multiple virtual reality (VR) video flows sharing a certain link, considering specific deadline of each video frame and the impact of different frames on video quality. Firstly, we establish a queuing delay bound estimation model, enabling link node to proactively discard frames that will exceed the deadline. Secondly, we model the importance of different frames based on viewport feature of VR video and encoding method. Accordingly, the frames of each flow are sorted. Then we formulate a problem of minimizing long-term quality loss caused by frame dropping subject to per-flow quality guarantee and bandwidth constraints. Since the frequency of frame dropping and network fluctuation are not on the same time scale, we propose a two-timescale resource allocation scheme. On the long timescale, a queuing theory based resource allocation method is proposed to satisfy quality requirement, utilizing frame queuing delay bound to obtain minimum resource demand for each flow. On the short timescale, in order to quickly fine-tune allocation results to cope with the unstable network state, we propose a low-complexity heuristic algorithm, scheduling available resources based on the importance of frames in each flow. Extensive experimental results demonstrate that the proposed scheme can efficiently improve quality and fairness of VR video flows under various network conditions.

Deadline Aware Two-Timescale Resource Allocation for VR Video Streaming最先出现在Nweon Paper。

Teams: University Hospital Essen

Writers: Jianning Li, Antonio Pepe, Christina Gsaxner, Gijs Luijten, Yuan Jin, Narmada Ambigapathy, Enrico Nasca, Naida Solak, Gian Marco Melito, Viet Duc Vu, Afaque R. Memon, Xiaojun Chen, Jan Stefan Kirschke, Ezequiel de la Rosa, Patrick Ferdinand Christ, Hongwei Bran Li, David G. Ellis, Michele R. Aizenberg, Sergios Gatidis, Thomas Küstner, Nadya Shusharina, Nicholas Heller, Vincent Andrearczyk, Adrien Depeursinge, Mathieu Hatt, Anjany Sekuboyina, Maximilian Löffler, Hans Liebl, Reuben Dorent, Tom Vercauteren, Jonathan Shapey, Aaron Kujawa, Stefan Cornelissen, Patrick Langenhuizen, Achraf Ben-Hamadou, Ahmed Rekik, Sergi Pujades, Edmond Boyer, Federico Bolelli, Costantino Grana, Luca Lumetti, Hamidreza Salehi, Jun Ma, Yao Zhang, Ramtin Gharleghi, Susann Beier, Arcot Sowmya, Eduardo A. Garza-Villarreal, Thania Balducci, Diego Angeles-Valdez, Roberto Souza, Leticia Rittner, Richard Frayne, Yuanfeng Ji, Soumick Chatterjee, Florian Dubost, Stefanie Schreiber, Hendrik Mattern, Oliver Speck, Daniel Haehn, Christoph John, Andreas Nürnberger, João Pedrosa, Carlos Ferreira, Guilherme Aresta, António Cunha, Aurélio Campilho, Yannick Suter, Jose Garcia, Alain Lalande, Emmanuel Audenaert, Claudia Krebs, Timo Van Leeuwen, Evie Vereecke, Rainer Röhrig, Frank Hölzle, Vahid Badeli, Kathrin Krieger, Matthias Gunzer, Jianxu Chen, Amin Dada, Miriam Balzer, Jana Fragemann, Frederic Jonske, Moritz Rempe, Stanislav Malorodov, Fin H. Bahnsen, Constantin Seibold, Alexander Jaus, Ana Sofia Santos, Mariana Lindo, André Ferreira, Victor Alves, Michael Kamp, Amr Abourayya, Felix Nensa, Fabian Hörst, Alexander Brehmer, Lukas Heine, Lars E. Podleska et al. (25 additional authors not shown)

PDF: MedShapeNet – A Large-Scale Dataset of 3D Medical Shapes for Computer Vision

Abstract

We present MedShapeNet, a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D surgical instrument models. Prior to the deep learning era, the broad application of statistical shape models (SSMs) in medical image analysis is evidence that shapes have been commonly used to describe medical data. Nowadays, however, state-of-the-art (SOTA) deep learning algorithms in medical imaging are predominantly voxel-based. In computer vision, on the contrary, shapes (including, voxel occupancy grids, meshes, point clouds and implicit surface models) are preferred data representations in 3D, as seen from the numerous shape-related publications in premier vision conferences, such as the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), as well as the increasing popularity of ShapeNet (about 51,300 models) and Princeton ModelNet (127,915 models) in computer vision research. MedShapeNet is created as an alternative to these commonly used shape benchmarks to facilitate the translation of data-driven vision algorithms to medical applications, and it extends the opportunities to adapt SOTA vision algorithms to solve critical medical problems. Besides, the majority of the medical shapes in MedShapeNet are modeled directly on the imaging data of real patients, and therefore it complements well existing shape benchmarks comprising of computer-aided design (CAD) models. MedShapeNet currently includes more than 100,000 medical shapes, and provides annotations in the form of paired data. It is therefore also a freely available repository of 3D models for extended reality (virtual reality – VR, augmented reality – AR, mixed reality – MR) and medical 3D printing. This white paper describes in detail the motivations behind MedShapeNet, the shape acquisition procedures, the use cases, as well as the usage of the online shape search portal: https://medshapenet.ikim.nrw/

MedShapeNet – A Large-Scale Dataset of 3D Medical Shapes for Computer Vision最先出现在Nweon Paper。

Teams: Yonsei University

Writers: Myungjin Shin, Dohae Lee, In-Kwon Lee

PDF: Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals

Abstract

The most popular type of devices used to track a user’s posture in a virtual reality experience consists of a head-mounted display and two controllers held in both hands. However, due to the limited number of tracking sensors (three in total), faithfully recovering the user in full-body is challenging, limiting the potential for interactions among simulated user avatars within the virtual world. Therefore, recent studies have attempted to reconstruct full-body poses using neural networks that utilize previously learned human poses or accept a series of past poses over a short period. In this paper, we propose a method that utilizes information from a neural motion prior to improve the accuracy of reconstructed user’s motions. Our approach aims to reconstruct user’s full-body poses by predicting the latent representation of the user’s overall motion from limited input signals and integrating this information with tracking sensor inputs. This is based on the premise that the ultimate goal of pose reconstruction is to reconstruct the motion, which is a series of poses. Our results show that this integration enables more accurate reconstruction of the user’s full-body motion, particularly enhancing the robustness of lower body motion reconstruction from impoverished signals. Web: https://https://mjsh34.this http URL

Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from Very Sparse Signals最先出现在Nweon Paper。

Teams: Dept. of EEE,

Writers: Md Awsafur Rahman, Shaikh Anowarul Fattah

PDF: Semi-Supervised Semantic Depth Estimation using Symbiotic Transformer and NearFarMix Augmentation

Abstract

In computer vision, depth estimation is crucial for domains like robotics, autonomous vehicles, augmented reality, and virtual reality. Integrating semantics with depth enhances scene understanding through reciprocal information sharing. However, the scarcity of semantic information in datasets poses challenges. Existing convolutional approaches with limited local receptive fields hinder the full utilization of the symbiotic potential between depth and semantics. This paper introduces a dataset-invariant semi-supervised strategy to address the scarcity of semantic information. It proposes the Depth Semantics Symbiosis module, leveraging the Symbiotic Transformer for achieving comprehensive mutual awareness by information exchange within both local and global contexts. Additionally, a novel augmentation, NearFarMix is introduced to combat overfitting and compensate both depth-semantic tasks by strategically merging regions from two images, generating diverse and structurally consistent samples with enhanced control. Extensive experiments on NYU-Depth-V2 and KITTI datasets demonstrate the superiority of our proposed techniques in indoor and outdoor environments.

Semi-Supervised Semantic Depth Estimation using Symbiotic Transformer and NearFarMix Augmentation最先出现在Nweon Paper。

Teams: IIIT Hyderabad；Fujitsu Research

Writers: Siddharth Katageri, Arkadipta De, Chaitanya Devaguptapu, VSSV Prasad, Charu Sharma, Manohar Kaul

PDF: Synergizing Contrastive Learning and Optimal Transport for 3D Point Cloud Domain Adaptation

Abstract

Recently, the fundamental problem of unsupervised domain adaptation (UDA) on 3D point clouds has been motivated by a wide variety of applications in robotics, virtual reality, and scene understanding, to name a few. The point cloud data acquisition procedures manifest themselves as significant domain discrepancies and geometric variations among both similar and dissimilar classes. The standard domain adaptation methods developed for images do not directly translate to point cloud data because of their complex geometric nature. To address this challenge, we leverage the idea of multimodality and alignment between distributions. We propose a new UDA architecture for point cloud classification that benefits from multimodal contrastive learning to get better class separation in both domains individually. Further, the use of optimal transport (OT) aims at learning source and target data distributions jointly to reduce the cross-domain shift and provide a better alignment. We conduct a comprehensive empirical study on PointDA-10 and GraspNetPC-10 and show that our method achieves state-of-the-art performance on GraspNetPC-10 (with approx 4-12% margin) and best average performance on PointDA-10. Our ablation studies and decision boundary analysis also validate the significance of our contrastive learning module and OT alignment.

Synergizing Contrastive Learning and Optimal Transport for 3D Point Cloud Domain Adaptation最先出现在Nweon Paper。

Teams: Utah State University

Writers: Dongyun Han, Donghoon Kim, Kangsoo Kim, Isaac Cho

PDF: Investigating Psychological Ownership in a Shared AR Space: Effects of Human and Object Reality and Object Controllability

Abstract

Augmented reality (AR) provides users with a unique social space where virtual objects are natural parts of the real world. The users can interact with 3D virtual objects and virtual humans projected onto the physical environment. This work examines perceived ownership based on the reality of objects and partners, as well as object controllability in a shared AR setting. Our formal user study with 28 participants shows a sense of possession, control, separation, and partner presence affect their perceived ownership of a shared object. Finally, we discuss the findings and present a conclusion.

Investigating Psychological Ownership in a Shared AR Space: Effects of Human and Object Reality and Object Controllability最先出现在Nweon Paper。

Teams: VNU-HCM

Writers: Khoi-Nguyen Nguyen-Ngoc, Thanh-Tung Phan-Nguyen, Khanh-Duy Le, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le

PDF: DM-VTON: Distilled Mobile Real-time Virtual Try-On

Abstract

The fashion e-commerce industry has witnessed significant growth in recent years, prompting exploring image-based virtual try-on techniques to incorporate Augmented Reality (AR) experiences into online shopping platforms. However, existing research has primarily overlooked a crucial aspect – the runtime of the underlying machine-learning model. While existing methods prioritize enhancing output quality, they often disregard the execution time, which restricts their applications on a limited range of devices. To address this gap, we propose Distilled Mobile Real-time Virtual Try-On (DM-VTON), a novel virtual try-on framework designed to achieve simplicity and efficiency. Our approach is based on a knowledge distillation scheme that leverages a strong Teacher network as supervision to guide a Student network without relying on human parsing. Notably, we introduce an efficient Mobile Generative Module within the Student network, significantly reducing the runtime while ensuring high-quality output. Additionally, we propose Virtual Try-on-guided Pose for Data Synthesis to address the limited pose variation observed in training images. Experimental results show that the proposed method can achieve 40 frames per second on a single Nvidia Tesla T4 GPU and only take up 37 MB of memory while producing almost the same output quality as other state-of-the-art methods. DM-VTON stands poised to facilitate the advancement of real-time AR applications, in addition to the generation of lifelike attired human figures tailored for diverse specialized training tasks. this https URL

DM-VTON: Distilled Mobile Real-time Virtual Try-On最先出现在Nweon Paper。

Teams: Virginia Tech；Georgia Tech

Writers: Gunnar Reiske, Sungwon In, Yalong Yang

PDF: Multi-Focus Querying of the Human Genome Information on Desktop and in Virtual Reality: an Evaluation

Abstract

The human genome is incredibly information-rich, consisting of approximately 25,000 protein-coding genes spread out over 3.2 billion nucleotide base pairs contained within 24 unique chromosomes. The genome is important in maintaining spatial context, which assists in understanding gene interactions and relationships. However, existing methods of genome visualization that utilize spatial awareness are inefficient and prone to limitations in presenting gene information and spatial context. This study proposed an innovative approach to genome visualization and exploration utilizing virtual reality. To determine the optimal placement of gene information and evaluate its essentiality in a VR environment, we implemented and conducted a user study with three different interaction methods. Two interaction methods were developed in virtual reality to determine if gene information is better suited to be embedded within the chromosome ideogram or separate from the ideogram. The final ideogram interaction method was performed on a desktop and served as a benchmark to evaluate the potential benefits associated with the use of VR. Our study findings reveal a preference for VR, despite longer task completion times. In addition, the placement of gene information within the visualization had a notable impact on the ability of a user to complete tasks. Specifically, gene information embedded within the chromosome ideogram was better suited for single target identification and summarization tasks, while separating gene information from the ideogram better supported region comparison tasks.

Multi-Focus Querying of the Human Genome Information on Desktop and in Virtual Reality: an Evaluation最先出现在Nweon Paper。

Teams: Waseda University；Durham University；Waseda Research Institute for Science and Engineering

Writers: Qi Feng, Hubert P. H. Shum, Shigeo Morishima

PDF: Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation

Abstract

Pre-captured immersive environments using omnidirectional cameras provide a wide range of virtual reality applications. Previous research has shown that manipulating the eye height in egocentric virtual environments can significantly affect distance perception and immersion. However, the influence of eye height in pre-captured real environments has received less attention due to the difficulty of altering the perspective after finishing the capture process. To explore this influence, we first propose a pilot study that captures real environments with multiple eye heights and asks participants to judge the egocentric distances and immersion. If a significant influence is confirmed, an effective image-based approach to adapt pre-captured real-world environments to the user’s eye height would be desirable. Motivated by the study, we propose a learning-based approach for synthesizing novel views for omnidirectional images with altered eye heights. This approach employs a multitask architecture that learns depth and semantic segmentation in two formats, and generates high-quality depth and semantic segmentation to facilitate the inpainting stage. With the improved omnidirectional-aware layered depth image, our approach synthesizes natural and realistic visuals for eye height adaptation. Quantitative and qualitative evaluation shows favorable results against state-of-the-art methods, and an extensive user study verifies improved perception and immersion for pre-captured real-world environments.

Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation最先出现在Nweon Paper。

Teams: ByteDance In

Writers: Xiaozheng Zheng, Zhuo Su, Chao Wen, Zhou Xue, Xiaojie Jin

PDF: Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling

Project: Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling

Abstract

To bridge the physical and virtual worlds for rapidly developed VR/AR applications, the ability to realistically drive 3D full-body avatars is of great significance. Although real-time body tracking with only the head-mounted displays (HMDs) and hand controllers is heavily under-constrained, a carefully designed end-to-end neural network is of great potential to solve the problem by learning from large-scale motion data. To this end, we propose a two-stage framework that can obtain accurate and smooth full-body motions with the three tracking signals of head and hands only. Our framework explicitly models the joint-level features in the first stage and utilizes them as spatiotemporal tokens for alternating spatial and temporal transformer blocks to capture joint-level correlations in the second stage. Furthermore, we design a set of loss terms to constrain the task of a high degree of freedom, such that we can exploit the potential of our joint-level modeling. With extensive experiments on the AMASS motion dataset and real-captured data, we validate the effectiveness of our designs and show our proposed method can achieve more accurate and smooth motion compared to existing approaches.

Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling最先出现在Nweon Paper。

Teams: Southeast University；PICO

Writers: Binghui Zuo, Zimeng Zhao, Wenqian Sun, Wei Xie, Zhou Xue, Yangang Wang

PDF: Reconstructing Interacting Hands with Interaction Prior from Monocular Images

Project: Reconstructing Interacting Hands with Interaction Prior from Monocular Images

Abstract

Reconstructing interacting hands from monocular images is indispensable in AR/VR applications. Most existing solutions rely on the accurate localization of each skeleton joint. However, these methods tend to be unreliable due to the severe occlusion and confusing similarity among adjacent hand parts. This also defies human perception because humans can quickly imitate an interaction pattern without localizing all joints. Our key idea is to first construct a two-hand interaction prior and recast the interaction reconstruction task as the conditional sampling from the prior. To expand more interaction states, a large-scale multimodal dataset with physical plausibility is proposed. Then a VAE is trained to further condense these interaction patterns as latent codes in a prior distribution. When looking for image cues that contribute to interaction prior sampling, we propose the interaction adjacency heatmap (IAH). Compared with a joint-wise heatmap for localization, IAH assigns denser visible features to those invisible joints. Compared with an all-in-one visible heatmap, it provides more fine-grained local interaction information in each interaction region. Finally, the correlations between the extracted features and corresponding interaction codes are linked by the ViT module. Comprehensive evaluations on benchmark datasets have verified the effectiveness of this framework. The code and dataset are publicly available at this https URL

Reconstructing Interacting Hands with Interaction Prior from Monocular Images最先出现在Nweon Paper。

Teams: Beijing University of Posts and Telecommunications；PICO

Writers: Pengfei Ren, Chao Wen, Xiaozheng Zheng, Zhou Xue, Haifeng Sun, Qi Qi, Jingyu Wang, Jianxin Liao

PDF: Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Project: Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image

Abstract

Reconstructing interacting hands from a single RGB image is a very challenging task. On the one hand, severe mutual occlusion and similar local appearance between two hands confuse the extraction of visual features, resulting in the misalignment of estimated hand meshes and the image. On the other hand, there are complex spatial relationship between interacting hands, which significantly increases the solution space of hand poses and increases the difficulty of network learning. In this paper, we propose a decoupled iterative refinement framework to achieve pixel-alignment hand reconstruction while efficiently modeling the spatial relationship between hands. Specifically, we define two feature spaces with different characteristics, namely 2D visual feature space and 3D joint feature space. First, we obtain joint-wise features from the visual feature map and utilize a graph convolution network and a transformer to perform intra- and inter-hand information interaction in the 3D joint feature space, respectively. Then, we project the joint features with global information back into the 2D visual feature space in an obfuscation-free manner and utilize the 2D convolution for pixel-wise enhancement. By performing multiple alternate enhancements in the two feature spaces, our method can achieve an accurate and robust reconstruction of interacting hands. Our method outperforms all existing two-hand reconstruction methods by a large margin on the InterHand2.6M dataset.

Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image最先出现在Nweon Paper。

Teams: State Key Laboratory of Networking and Switching Technology, BUPT

Writers: Xiaozheng Zheng, Chao Wen, Zhou Xue, Pengfei Ren, Jingyu Wang

PDF: HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning

Project: HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning

Abstract

Recent advancements in 3D hand pose estimation have shown promising results, but its effectiveness has primarily relied on the availability of large-scale annotated datasets, the creation of which is a laborious and costly process. To alleviate the label-hungry limitation, we propose a self-supervised learning framework, HaMuCo, that learns a single-view hand pose estimator from multi-view pseudo 2D labels. However, one of the main challenges of self-supervised learning is the presence of noisy labels and the “groupthink” effect from multiple views. To overcome these issues, we introduce a cross-view interaction network that distills the single-view estimator by utilizing the cross-view correlated features and enforcing multi-view consistency to achieve collaborative learning. Both the single-view estimator and the cross-view interaction network are trained jointly in an end-to-end manner. Extensive experiments show that our method can achieve state-of-the-art performance on multi-view self-supervised hand pose estimation. Furthermore, the proposed cross-view interaction network can also be applied to hand pose estimation from multi-view input and outperforms previous methods under the same settings.

HaMuCo: Hand Pose Estimation via Multiview Collaborative Self-Supervised Learning最先出现在Nweon Paper。

Teams: KAIST

Writers: Dooyoung Kim, Woontack Woo

PDF: Edge-Centric Space Rescaling with Redirected Walking for Dissimilar Physical-Virtual Space Registration

Abstract

We propose a novel space-rescaling technique for registering dissimilar physical-virtual spaces by utilizing the effects of adjusting physical space with redirected walking. Achieving a seamless immersive Virtual Reality (VR) experience requires overcoming the spatial heterogeneities between the physical and virtual spaces and accurately aligning the VR environment with the user’s tracked physical space. However, existing space-matching algorithms that rely on one-to-one scale mapping are inadequate when dealing with highly dissimilar physical and virtual spaces, and redirected walking controllers could not utilize basic geometric information from physical space in the virtual space due to coordinate distortion. To address these issues, we apply relative translation gains to partitioned space grids based on the main interactable object’s edge, which enables space-adaptive modification effects of physical space without coordinate distortion. Our evaluation results demonstrate the effectiveness of our algorithm in aligning the main object’s edge, surface, and wall, as well as securing the largest registered area compared to alternative methods under all conditions. These findings can be used to create an immersive play area for VR content where users can receive passive feedback from the plane and edge in their physical environment.

Edge-Centric Space Rescaling with Redirected Walking for Dissimilar Physical-Virtual Space Registration最先出现在Nweon Paper。

Teams: Xi’an Jiaotong-Liverpool University

Writers: Rongkai Shi, Yushi Wei, Yue Li, Lingyun Yu, Hai-Ning Liang

PDF: Expanding Targets in Virtual Reality Environments: A Fitts’ Law Study

Abstract

Target pointing selection is a fundamental task. According to Fitts’ law, users need more time to select targets with smaller sizes. Expanding the target to a larger size is a practical approach that can facilitate pointing selection. It has been well-examined and -deployed in 2D user interfaces. However, limited research has investigated target expansion methods using an immersive virtual reality (VR) head-mounted display (HMD). In this work, we aimed to fill this gap by conducting a user study using ISO 9241-411 multi-directional pointing task to examine the effect of target expansion on target selection performance in VR HMD. Based on our results, we found that compared to not expanding the target, expanding the target width by 1.5 and 2.5 times during the movement can significantly reduce the selection time. We hope that the design and results derived from the study can help frame future work.

Expanding Targets in Virtual Reality Environments: A Fitts’ Law Study最先出现在Nweon Paper。

Teams: Utah State University

Writers: DongHoon Kim, Dongyun Han, Isaac Cho

PDF: Comparative Analysis of Change Blindness in Virtual Reality and Augmented Reality Environments

Abstract

Change blindness is a phenomenon where an individual fails to notice alterations in a visual scene when a change occurs during a brief interruption or distraction. Understanding this phenomenon is specifically important for the technique that uses a visual stimulus, such as Virtual Reality (VR) or Augmented Reality (AR). Previous research had primarily focused on 2D environments or conducted limited controlled experiments in 3D immersive environments. In this paper, we design and conduct two formal user experiments to investigate the effects of different visual attention-disrupting conditions (Flickering and Head-Turning) and object alternative conditions (Removal, Color Alteration, and Size Alteration) on change blindness detection in VR and AR environments. Our results reveal that participants detected changes more quickly and had a higher detection rate with Flickering compared to Head-Turning. Furthermore, they spent less time detecting changes when an object disappeared compared to changes in color or size. Additionally, we provide a comparison of the results between VR and AR environments.

Comparative Analysis of Change Blindness in Virtual Reality and Augmented Reality Environments最先出现在Nweon Paper。

Teams: ETH Zurich

Writers: Thomas Ruegg, Pietro Bonazzi, Andrea Ronco

PDF: Gaze Estimation on Spresense

Abstract

Gaze estimation is a valuable technology with numerous applications in fields such as human-computer interaction, virtual reality, and medicine. This report presents the implementation of a gaze estimation system using the Sony Spresense microcontroller board and explores its performance in latency, MAC/cycle, and power consumption. The report also provides insights into the system’s architecture, including the gaze estimation model used. Additionally, a demonstration of the system is presented, showcasing its functionality and performance. Our lightweight model TinyTrackerS is a mere 169Kb in size, using 85.8k parameters and runs on the Spresense platform at 3 FPS.

Gaze Estimation on Spresense最先出现在Nweon Paper。

Teams: The University of Melbourne;United Arab Emirates University

Writers: Leila Ismail, Rajkumar Buyya

PDF: Metaverse: A Vision, Architectural Elements, and Future Directions for Scalable and Realtime Virtual Worlds

Abstract

With the emergence of Cloud computing, Internet of Things-enabled Human-Computer Interfaces, Generative Artificial Intelligence, and high-accurate Machine and Deep-learning recognition and predictive models, along with the Post Covid-19 proliferation of social networking, and remote communications, the Metaverse gained a lot of popularity. Metaverse has the prospective to extend the physical world using virtual and augmented reality so the users can interact seamlessly with the real and virtual worlds using avatars and holograms. It has the potential to impact people in the way they interact on social media, collaborate in their work, perform marketing and business, teach, learn, and even access personalized healthcare. Several works in the literature examine Metaverse in terms of hardware wearable devices, and virtual reality gaming applications. However, the requirements of realizing the Metaverse in realtime and at a large-scale need yet to be examined for the technology to be usable. To address this limitation, this paper presents the temporal evolution of Metaverse definitions and captures its evolving requirements. Consequently, we provide insights into Metaverse requirements. In addition to enabling technologies, we lay out architectural elements for scalable, reliable, and efficient Metaverse systems, and a classification of existing Metaverse applications along with proposing required future research directions.

Metaverse: A Vision, Architectural Elements, and Future Directions for Scalable and Realtime Virtual Worlds最先出现在Nweon Paper。