Comic-Guided Speech Synthesis

PubDate: Nov 2019

Teams: Beijing Institute of Technology 2Inception Institute of Artificial Intelligence 3George Mason University

Writers: Yujia Wang1 Wenguan Wang1,2 Wei Liang1 Lap-Fai Yu

PDF: Comic-Guided Speech Synthesis

Project: Comic-Guided Speech Synthesis

Abstract

We introduce a novel approach for synthesizing realistic speeches for comics. Using a comic page as input, our approach synthesizes speeches for each comic character following the reading flow. It adopts a cascading strategy to synthesize speeches in two stages: Comic Visual Analysis and Comic Speech Synthesis. In the first stage, the input comic page is analyzed to identify the gender and age of the characters, as well as texts each character speaks and corresponding emotion. Guided by this analysis, in the second stage, our approach synthesizes realistic speeches for each character, which are consistent with the visual observations. Our experiments show that the proposed approach can synthesize realistic and lively speeches for different types of comics. Perceptual studies performed on the synthesis results of multiple sample comics validate the efficacy of our approach.