Developing a Scaled Performance Evaluation Measurement System to Measure Performance
(Room 320E)
05 Dec 19
9:00 AM
-
9:30 AM
Tracks:
Full Schedule, Thursday Schedule
Background: Training capability developers lack consistent tools for analyzing the benefits of synthetic training environment integration – especially related to measuring the transfer-of-training for live, virtual, and constructive systems (Government Accountability Office Report 13-698). The purpose of this study is to define, develop, and test a scaled performance evaluation measurement system (SPEMS) that can be used across a wide variety of training tasks. We used the buddy rush task as a test case for the utility of SPEMS. Trainees are evaluated on a nine-step performance evaluation checklist (PECL) during the buddy rush task. Currently, trainees only receive a Go/No Go evaluation for each step. Method: We defined, developed, and tested SPEMS by Step 1. Convening focus groups to establish 5 level behaviorally anchored rating scales (BARS) to be implemented into SPEMS; Step 2. Convening subject matter expert (SME) focus groups to confirm SPEMS inter-rater reliability utilizing a virtual video analysis of the buddy rush task; Step 3. Empirically testing the accuracy and predicative capability of SPEMS in an operational environment. We will conduct an experiment by observing two groups of evaluators (control group and SPEMS group) who will be evaluating trainees’ performance of INF-MAN-3001, the buddy rush task, using either the current PECL or SPEMS evaluation systems. We expect to collect approximately 72 evaluations from each group. Additionally, we will determine if SPEMS predicts objective measures of performance in the buddy rush task. Preliminary results: Suitable inter-rater reliability was found for both the BARS (Step 1: 87% inter-rater agreement) and SPEMS (Step 2: Cronbach’s Alpha 0.93 to 0.98) The experiment will be conducted in March and data analysis will be completed by June. Conclusion: Validating the accuracy and reliability of SPEMS will provide training capability developers a method for determining the training transfer that results from employing alternative training solutions.