DEVELOPING A SCALED PERFORMANCE EVALUATION MEASUREMENT SYSTEM TO EVALUATE MARINE PERFORMANCE

Abstract

Training developers lack methods for determining the benefits of integrating live, virtual, and constructive training. This study defined and tested a scaled performance evaluation measurement system (SPEMS) to be used across tasks. We used the buddy rush task to test SPEMS and compare it to the current “Go/No Go” performance evaluation checklist (PECL). We developed SPEMS in three steps: we convened focus groups to establish 5-level behaviorally anchored rating scales (BARS); confirmed SPEMS reliability using subject-matter expert (SME) virtual video analysis; and empirically tested SPEMS’ predictive capability in an operational environment. Suitable inter-rater reliability was found for BARS (87% agreement) and SPEMS (Cronbach’s Alpha 0.93 to 0.98). Percent exposure was selected by SMEs as the objective measure of buddy rush performance. Fifty-two trainees (26 pairs) were evaluated using a PECL and SPEMS at three time points. The results showed that SPEMS has a moderate, negative, linear relationship with percent exposure at an R2 = 0.41/0.40. Conversely, PECL has a weak, slightly negative linear relationship with percent exposure at an R2 = 0.03/0.2. We reject the null hypotheses and conclude that SPEMS scores are significantly related to percent exposure and have more predictive strength than PECL scores. These findings demonstrate a verifiable, repeatable, and reliable potential solution to the problem of measuring military task performance across training solutions.