BAUFER: A Baseline-Enabled Facial Expression Recognition Pipeline Trained With Limited Annotations

Book chapter, 2023

Social science theories suggest that facial expressions serve as a valuable indicator of one’s emotions, well-being, and overall functioning. Recent research has found that the facial expressions of participants in clinical trials can be linked to their self-reported quality of life. Since manual facial expression annotation and interpretation is time and cost intensive, automated facial expression recognition (FER) tools have the potential to make it quicker and more consistent to study an individual’s emotional responses. This paper introduces BAUFER, Baseline-enabled Action Unit identification for Facial Expression Recognition, with the following features: (1) a personalized baseline component to calibrate for the neutral expression of a participant; (2) predictions for anatomically-based facial muscle movement labels (Action Units), which have been reliably linked to emotional experiences in prior research, to enhance interpretability; and (3) a multi-stage training approach with several types of annotations from different datasets to overcome the known challenge of insufficient labeled data. While developed with non-clinical data, an intended future application of BAUFER is in the clinical domain to enhance our understanding of the patient experience.