Kavach Dheer

Title: Exploring the Integration of Emotional, Cognitive and BioPhysical Sensing into Recommender Systems for Digital Entertainment

Supervision Team: Josephine Griffith, UoG / Robert Ross, TU Dublin / Peter Corcoran, UoG and Joe Lemley, Xperi

Description: Passive sensing of a user’s emotional state is challenging without measuring biophysical signals, although there has been progress in determining emotional states from video-based facial analysis, human-speech analysis and combined approaches. Cognition and stress-assessment are niche areas of research but are important recently in driver monitoring systems. This research will explore new approaches to combine SotA hybrid speech/imaging techniques to perform a real-time emotional/cognitive state assessment (ECSA) of a user interacting with a recommender system. In parallel the recommender model will be adapted to respond to the emotional/cognitive inputs, employing these to dynamically adapt the outputs provided to the user, based on their assessed emotional/cognitive states. As an indicative example, an in-car entertainment system might take decisions between suggesting video entertainment for occupants, or music for the driver, based on ECSAs of both driver and occupants, using data from an in-car camera and microphone.