DEVELOPING A SCALED PERFORMANCE EVALUATION MEASUREMENT SYSTEM TO EVALUATE MARINE PERFORMANCE
Loading...
Authors
Loeffelman, Garrett A.
Subjects
evaluation
human performance
LVC
metrics
operational environment
proficiency
readiness
return on investment
training
human performance
LVC
metrics
operational environment
proficiency
readiness
return on investment
training
Advisors
Hodges, Glenn A.
Kennedy, Meghan Q.
Date of Issue
2019-06
Date
Publisher
Monterey, CA; Naval Postgraduate School
Language
Abstract
Training developers lack methods for determining the benefits of integrating live, virtual, and constructive training. This study defined and tested a scaled performance evaluation measurement system (SPEMS) to be used across tasks. We used the buddy rush task to test SPEMS and compare it to the current “Go/No Go” performance evaluation checklist (PECL). We developed SPEMS in three steps: we convened focus groups to establish 5-level behaviorally anchored rating scales (BARS); confirmed SPEMS reliability using subject-matter expert (SME) virtual video analysis; and empirically tested SPEMS’ predictive capability in an operational environment. Suitable inter-rater reliability was found for BARS (87% agreement) and SPEMS (Cronbach’s Alpha 0.93 to 0.98). Percent exposure was selected by SMEs as the objective measure of buddy rush performance. Fifty-two trainees (26 pairs) were evaluated using a PECL and SPEMS at three time points. The results showed that SPEMS has a moderate, negative, linear relationship with percent exposure at an R2 = 0.41/0.40. Conversely, PECL has a weak, slightly negative linear relationship with percent exposure at an R2 = 0.03/0.2. We reject the null hypotheses and conclude that SPEMS scores are significantly related to percent exposure and have more predictive strength than PECL scores. These findings demonstrate a verifiable, repeatable, and reliable potential solution to the problem of measuring military task performance across training solutions.
Type
Thesis
Description
Series/Report No
Department
Computer Science (CS)
Organization
Identifiers
NPS Report Number
Sponsors
Funder
Format
Citation
Distribution Statement
Approved for public release; distribution is unlimited.
Rights
This publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.