Multimodal Assessment of Pilots’ Affective States using Psycho Physiological Sensor Signals and Facial Recognition Analysis
(Room S320A)
28 Nov 17
3:00 PM
-
3:30 PM
Human error in aviation can lead to catastrophic results. Since 1959, the majority of fatal airplane accidents worldwide occurred during the takeoff and landing phases of flight and upwards of 60% are attributed to pilot error. Psychologists Robert Yerkes and John Dodson proposed an empirical relationship (Yerkes-Dodson law) that performance increases with physiological or mental arousal, but only up to a point, after which it decreases. Understanding a pilot’s arousal during flight and its relationship with performance can ultimately contribute to the design of improved flight safety and flight training systems.
The objective of this research is to provide guidelines for the multimodal assessment of pilots’ arousal level and affective states using noninvasive biosensors to answer the following three questions: i) Which data processing approach must be adopted to track pilots’ arousal; ii) What affective/cognitive model can be used to interpret these measurements; and iii) How can these measurements be used as training assessment criteria?
This paper reports on an experimental study we conducted in a full-flight simulator to explore the answers to the above questions. Pilots’ heart rate and galvanic skin response (GSR) were non-invasively and continuously recorded using a wristwatch biosensors during a 45 min flight of varying levels of complexity. Their facial expressions were recorded using a cockpit webcam to enable facial recognition analysis. Results revealed that physiological patterns may be related to the complexity level of the flight phase and to the pilots’ performance and experience. Specifically, the mean amplitude sum of the GSR phasic component revealed the pilot arousal level while the valence of the pilot’s affective state was captured through facial emotion recognition. These results can be mapped to the Circumplex Model of Affect as a framework to assess both individual and group performance.