Robust and Sequential Recognition of Emotional States Using Hidden Markov Models and Integrated Speech, Video, and Thermal Features

This proposal defines unique and promising solutions for robust and sequential emotional and stress state recognition in military operational environments. The solutions are non-invasive, multi-modality approaches, including speech, video, and thermal images. The feature sets are selected automatically based on the operational environments. For example, when the environment is too noisy, the system focuses on image features; when the lighting condition is too bad, it focuses on speech features; when both acoustic and lighting conditions are good, the system uses both speech and image features. This will provide the military with a degree of system versatility and allow broad applications to many operational scenarios. Multi-layer hidden Markov models are defined as the statistical models to characterize the emotional states. The recently developed sequential detection algorithm is proposed to detect the changes from one emotional state to another. Furthermore, a multi-modality database will be collected to study the feasibility of detecting at least the following emotional states in this project: anger, drowsiness, anxiety, fear, confusion, disorientation, and frustration. The new feature extraction and recognition algorithms will be developed and implemented through this project to discriminatively recognize those states. The developed system will be portable and can be operated in military environments automatically and continuously.