Exploring the Effect of Transfer Learning on Facial Expression Recognition using Photo-Reflective Sensors embedded into a Head-Mounted Display

As one of the techniques to recognize head-mounted display (HMD) user’s facial expressions, the photo-reflective sensor (PRS) has been employed. Since the classification performance of PRS-based method is affected by rewearing an HMD and difference in facial geometry for each user, the user have to perform dataset collection for each wearing of an HMD to build a facial expression classifier. To tackle this issue, we investigate how transfer learning improve within-user and cross-user accuracy and reduce training data in the PRS-based facial expression recognition. We collected a dataset of five facial expressions (Neutral, Smile, Angry, Surprised, Sad) when participants wore the PRS-embedded HMD five times. Using the dataset, we evaluated facial expression classification accuracy using a neural network with/without fine tuning. Our result showed fine tuning improved the within-user and cross-user facial expression classification accuracy compared with non-fine-tuned classifier. Also, applying fine tuning to the classifier trained with the other participant dataset achieved higher classification accuracy than the non-fine-tuned classifier.