EVALUATING MANAGERIAL IMPLICATIONS IN RESEARCH PAPERS WITH GENERATIVE PRE-TRAINED TRANSFORMERS

Abstract

Academic management research has been criticized for prioritizing theoretical contributions rather than practical relevance for managers, creating a theory–practice gap. This exploratory study examined the strengths and weaknesses of utilizing artificial intelligence (AI) systems to evaluate managerial relevance. Ten papers published between 2015 and 2025 in leading supply chain and management journals were evaluated by six AI systems (ChatGPT-4o, ChatGPT-4o Deep Research, Grok-Fast, Grok-Expert, Opus 4.5, and Opus 4.5 Extended Thinking) across six criteria: (1) actionability, (2) novelty, (3) feasibility (problem-solving), (4) feasibility (resources), (5) impact, and (6) accessibility. Results showed that mean total scores varied widely across AI systems, ranging from 23.0 to 38.0, a difference of 15 points. Although all systems used the same evaluation rubric, they showed differences in scoring patterns, suggesting system-level biases related to model architecture. Papers from supply chain journals generally received higher scores, likely due to better alignment with the evaluation rubric. Overall, the findings suggest that current AI systems still struggle to apply complex, subjective criteria when assessing managerial relevance. As a result, hybrid approaches combining AI with expert human judgment are recommended for future applications in research evaluation.