ASSESSING DOD CONFIDENCE AND BIAS IN AI/LLM AUTHORED EVALUATION FACTORS

Simple item page
dc.contributor.advisorFinkenstadt, Daniel J.
dc.contributor.advisorPorchia, Jamie M.
dc.contributor.advisorJosephson, Brett, George Mason University
dc.contributor.authorHedgepeth, Steven C.
dc.contributor.authorTagatac, Ryan Mark D.
dc.contributor.departmentDepartment of Defense Management (DDM)
dc.date.accessioned2024-02-19T22:56:48Z
dc.date.available2024-02-19T22:56:48Z
dc.date.issued2023-12
dc.description.abstractArtificial intelligence (AI)/Large Language Models (LLMs) have shown promise in various tasks, but their use in authoring source selection evaluation factors in the Department of Defense (DOD) is not well studied. Understanding the effectiveness of AI-authored evaluation factors is crucial for reliable decision-making. The integration of LLM technology in the DOD aligns with the rise of AI. This exploratory analysis investigated DOD acquisition professionals’ confidence in and bias toward AI-authored evaluation factors. Surveys at George Mason University (GMU) and Naval Postgraduate School presented professionals with requirements documentation and human or AI-generated evaluation factors. Due to statistically significant differences between the surveys, only the GMU data was relied on. Statistical and qualitative analyses evaluated variations in confidence ratings across different participant groupings and authorship disclosure. Results reveal reduced confidence and slight algorithm aversion to AI-authored factors versus human-authored, especially among older professionals. Despite limitations including sampling constraints, notable discrepancies emerge in perceptions of AI versus human outputs. Recommendations include the development of an AI guide to aid responsible use of AI in acquisitions. Further research with larger, varied samples and various AI tools is needed. This initial work advances AI integration policy discussions and public trust in defense acquisitions.en_US
dc.description.distributionstatementApproved for public release. Distribution is unlimited.en_US
dc.description.serviceCaptain, United States Air Forceen_US
dc.description.serviceCaptain, United States Air Forceen_US
dc.identifier.curriculumcode815, Defense Contract Management
dc.identifier.curriculumcode815, Defense Contract Management
dc.identifier.thesisid39830
dc.identifier.urihttps://hdl.handle.net/10945/72537
dc.publisherMonterey, CA; Naval Postgraduate Schoolen_US
dc.relation.ispartofseriesMaster of Business Administration (MBA) Professional Reports
dc.rightsThis publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.en_US
dc.subject.authorlarge language modelen_US
dc.subject.authorartificial intelligenceen_US
dc.subject.authorcontracten_US
dc.subject.authorsource selectionen_US
dc.subject.authorevaluation factorsen_US
dc.subject.authortrade-offen_US
dc.subject.authorFARen_US
dc.subject.authorDFARSen_US
dc.subject.authorChatGPTen_US
dc.subject.author2x2 factorial designen_US
dc.subject.authorbiasen_US
dc.subject.authoraversionen_US
dc.subject.authorautomationen_US
dc.subject.authoralgorithmen_US
dc.subject.authorDODen_US
dc.subject.authorPWSen_US
dc.subject.authorconfidenceen_US
dc.titleASSESSING DOD CONFIDENCE AND BIAS IN AI/LLM AUTHORED EVALUATION FACTORSen_US
dc.typeThesisen_US
dspace.entity.typePublication
etd.thesisdegree.disciplineMaster of Business Administrationen_US
etd.thesisdegree.disciplineMaster of Business Administrationen_US
etd.thesisdegree.grantorNaval Postgraduate Schoolen_US
etd.thesisdegree.levelMastersen_US
etd.thesisdegree.levelMastersen_US
etd.thesisdegree.nameMaster of Business Administrationen_US
etd.thesisdegree.nameMaster of Business Administrationen_US
relation.isDepartmentOfPublicationb8b0094c-45b5-4a6a-8813-f61a5dec4f2a
relation.isDepartmentOfPublication.latestForDiscoveryb8b0094c-45b5-4a6a-8813-f61a5dec4f2a
relation.isOrgUnitOfPublicationb8b0094c-45b5-4a6a-8813-f61a5dec4f2a
relation.isSeriesOfPublicatione0c365f0-ed04-4d27-bded-11759168d050
relation.isSeriesOfPublication.latestForDiscoverye0c365f0-ed04-4d27-bded-11759168d050
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
23Dec_Hedgepeth_Tagatac.pdf
Size:
38.21 MB
Format:
Adobe Portable Document Format
Download
Collections
Theses