ASSESSING DOD CONFIDENCE AND BIAS IN AI/LLM AUTHORED EVALUATION FACTORS
Loading...
Authors
Hedgepeth, Steven C.
Tagatac, Ryan Mark D.
Subjects
large language model
artificial intelligence
contract
source selection
evaluation factors
trade-off
FAR
DFARS
ChatGPT
2x2 factorial design
bias
aversion
automation
algorithm
DOD
PWS
confidence
artificial intelligence
contract
source selection
evaluation factors
trade-off
FAR
DFARS
ChatGPT
2x2 factorial design
bias
aversion
automation
algorithm
DOD
PWS
confidence
Advisors
Finkenstadt, Daniel J.
Porchia, Jamie M.
Josephson, Brett, George Mason University
Date of Issue
2023-12
Date
Publisher
Monterey, CA; Naval Postgraduate School
Language
Abstract
Artificial intelligence (AI)/Large Language Models (LLMs) have shown promise in various tasks, but their use in authoring source selection evaluation factors in the Department of Defense (DOD) is not well studied. Understanding the effectiveness of AI-authored evaluation factors is crucial for reliable decision-making. The integration of LLM technology in the DOD aligns with the rise of AI. This exploratory analysis investigated DOD acquisition professionals’ confidence in and bias toward AI-authored evaluation factors. Surveys at George Mason University (GMU) and Naval Postgraduate School presented professionals with requirements documentation and human or AI-generated evaluation factors. Due to statistically significant differences between the surveys, only the GMU data was relied on. Statistical and qualitative analyses evaluated variations in confidence ratings across different participant groupings and authorship disclosure. Results reveal reduced confidence and slight algorithm aversion to AI-authored factors versus human-authored, especially among older professionals. Despite limitations including sampling constraints, notable discrepancies emerge in perceptions of AI versus human outputs. Recommendations include the development of an AI guide to aid responsible use of AI in acquisitions. Further research with larger, varied samples and various AI tools is needed. This initial work advances AI integration policy discussions and public trust in defense acquisitions.
Type
Thesis
Description
Series/Report No
Department
Organization
Identifiers
NPS Report Number
Sponsors
Funder
Format
Citation
Distribution Statement
Approved for public release. Distribution is unlimited.
Rights
This publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.