跳至内容
  • 首页
  • 资讯
  • 行业方案
  • 付费阅读
  • Job招聘
  • Paper论文
  • Patent专利
  • 映维会员
  • 导航收录
  • 合作
  • 关于
  • 微信群
  • All
  • XR
  • CV
  • CG
  • HCI
  • Video
  • Optics
  • Perception
  • Reconstruction

LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

小编 广东客   |   分类:CV   |   2025年2月27日

Note: We don't have the ability to review paper

PubDate: Feb 2025

Teams:Beihang University, Beijing Academy of Blockchain and Edge Computing, Psyche AI,The University of Alabama at Birmingham

Writers:Jian Yang, Xukun Wang, Wentao Wang, Guoming Li, Qihang Fang, Ruihong Yuan, Tianyang Wang, Jason Zhaoxin Fan

PDF:LaDTalk: Latent Denoising for Synthesizing Talking Head Videos with High Frequency Details

Abstract

Audio-driven talking head generation is a pivotal area within film-making and Virtual Reality. Although existing methods have made significant strides following the end-to-end paradigm, they still encounter challenges in producing videos with high-frequency details due to their limited expressivity in this domain. This limitation has prompted us to explore an effective post-processing approach to synthesize photo-realistic talking head videos. Specifically, we employ a pretrained Wav2Lip model as our foundation model, leveraging its robust audio-lip alignment capabilities. Drawing on the theory of Lipschitz Continuity, we have theoretically established the noise robustness of Vector Quantised Auto Encoders (VQAEs). Our experiments further demonstrate that the high-frequency texture deficiency of the foundation model can be temporally consistently recovered by the Space-Optimised Vector Quantised Auto Encoder (SOVQAE) we introduced, thereby facilitating the creation of realistic talking head videos. We conduct experiments on both the conventional dataset and the High-Frequency TalKing head (HFTK) dataset that we curated. The results indicate that our method, LaDTalk, achieves new state-of-the-art video quality and out-of-domain lip synchronization performance.

本文链接:https://paper.nweon.com/16220

您可能还喜欢...

  • Fast and Robust Registration of multiple Depth-Sensors and Virtual Worlds

    2022年04月26日 映维

  • FMKit - An In-Air-Handwriting Analysis Library and Data Repository

    2020年06月22日 映维

  • DynIBaR: Neural Dynamic Image-Based Rendering

    2023年06月28日 映维

关注:

RSS 最新AR/VR行业分享

  • XR日报:Meta AR智能眼镜或叫Meta Celeste,Steam市占Quest 3 暴跌6.38% 2025年7月4日
  • 索尼专利提出基于眼动追踪的智能阅读控制模式 2025年7月4日
  • 索尼专利提出多模态手指追踪技术扩展游戏控制器功能 2025年7月4日
  • 纽约大学与Meta合作开发注视点实例分割框架FovealSeg 2025年7月4日
  • 范德比尔特大学研发柔性热反馈装置提升AR沉浸感 2025年7月4日

RSS 最新AR/VR专利

  • Meta Patent | Reconfigurable headset that transitions between virtual reality, augmented reality, and actual reality 2025年7月3日
  • Samsung Patent | Deposition equipment 2025年6月26日
  • Snap Patent | Determining gaze direction to generate augmented reality content 2025年6月26日
  • Apple Patent | Eye characteristic determination 2025年6月26日
  • Snap Patent | Eyewear device charging case 2025年6月26日

RSS 最新AR/VR行业招聘

  • Microsoft AR/VR Job | High Performance Compute, Director 2025年6月5日
  • Microsoft AR/VR Job | Data Center Technician/ Technicien de Centre de Données 2025年6月3日
  • Microsoft AR/VR Job | Senior Product Designer 2025年5月16日
  • Apple AR/VR Job | AirPlay Audio Engineer 2025年3月27日
  • Apple AR/VR Job | iOS Perception Engineer 2025年3月27日

联系微信:ovalics

版权所有:广州映维网络有限公司 © 2025

备案许可:粤ICP备17113731号-2

粤公网安备 44011302004835号

友情链接: AR/VR行业导航

读者QQ群:251118691

Quest QQ群:526200310

开发者QQ群:688769630

Paper