Improving Assessments using Intelligent Agents with Transient Emotional States
(Room 320F)
04 Dec 19
2:30 PM
-
3:00 PM
Tracks:
Full Schedule, Wednesday Schedule
The assessment of interpersonal leadership skills has historically been impeded by the lack of assessment techniques free of response bias and/or resource prohibitions. Advances in psychology and computer science (e.g., Reactive, Open-Response Assessments; RORAs) have provided novel alternatives to traditional assessment methods (Brou, Stallings, Normand, Stearns, & Ledford, 2018). RORAs assess interpersonal skills via scenarios in which users interact with virtual agents; however, these agents currently lack sophisticated emotional response capabilities. The present research begins to address this shortcoming by developing virtual agents with the capacity to parse emotional inputs and generate appropriate emotional responses. Such techniques applied in other domains using intelligent agents have been shown to increase the effectiveness of information delivery and retrieval (Janarthanam, 2017). Four intelligent agents were developed using the DialogFlow Platform. Agents’ affective states and user inputs were classified using Russell's Circumplex Model of Affect (Russell, 1980). One-hundred and twenty-five participants provided conversational inputs intended to modify the agents’ affective states in particular ways during interactions. Affective models correctly classified 78% of user inputs, leading to the appropriate shifts in agent affective states. Next, deep learning techniques (e.g., sequence to sequence) were used to generate novel agent utterances based on agent affective states. Novel agent utterances were evaluated using the BLEU metric (Papineni, Roukos, Ward, & Zhu, 2002). Utterances achieved a BLEU rating of 0.2, indicating performance consistent with similar natural language generation systems in the literature (Gkatzia & Mahamood, 2015). These results demonstrate the potential for improving effective dialog between a human user and an intelligent agent during interpersonal skill assessment.