Classifying Facial Expressions of Students Being Tutored in a Gateway College Math Course

When it comes to tutoring, computers have not quite been able to achieve the success that humans have in helping students improve learning outcomes. This research sought to address one aspect of what makes human tutors more effective, the ability to identify and to interpret facial expressions. When a student is feeling anxious, confused, distracted or frustrated, or when a student has an ‘aha’ moment, human tutors can identify the student’s facial expressions and adjust their tutoring approach as necessary. This study sought to determine if, in the context of a gateway college math course, these particular learning-centered affects could be accurately identified by a facial expression recognition deep learning convolutional neural network. Two models were tested. The first model was trained on ground truth images collected during tutoring sessions and annotated by the tutors themselves. This model achieved an accuracy of .25, failing to classify most facial expressions. The second model attempted to predict facial expressions using valence/ arousal matrix scores. This model was trained on Affectnet and achieved RMSE scores consistent with other models trained on the same database. A closer look at the data revealed that although RMSE scores were comparable, there were large areas of facial expressions where the model misidentified the valence in particular. This result suggests that while the valence/ arousal matrix was a better fit in terms of identifying learning-centered affects during tutoring, using the large emotion recognition image databases will not always produce the desired interpretation. This indicates the need for additional exploration into training valence/arousal models using images specific to tutoring scenarios.