Improving Defense Acquisition Processes with Evidence-based Analysis: An Illustrative Case Using the DoD SBIR Program

Abstract

This paper proposes and demonstrates that experimental and quasi-experimental program evaluation methods can be applied to some parts of the defense acquisition system to provide evidence of program effectiveness. The specific example presented is a quasi-experimental evaluation of the Department of Defense Small Business Innovation Research (DoD SBIR) program. Quasi-experimental methods are a set of program evaluation techniques that allow researchers to approximate the results of an experimental study, such as a randomized controlled trial, without performing the experiment. The paper performs a quasi-experimental evaluation of the DoD SBIR program, which provides evidence that the program is effective at transitioning SBIR-funded technologies into other DoD programs. This demonstration that quasi-experimental methods can be used to evaluate certain aspects of the DoD acquisition system provides policy analysts with new tools to meet Congressional requirements for acquisition system evaluation. The paper recommends that more quasi-experimental studies be conducted and actual experimental studies be executed. These methods can help the DoD overcome the well-documented deficiency in evaluating the effectiveness of its acquisition systems. The Office of Management and Budget, the Government Accountability Office and the House Armed Services Committee unanimously agree that the DoD does not objectively measure the performance of its acquisition system.