The Correct Use of Subject Matter Experts in Cost Risk Analysis
Braxton, Peter J.
Druker, Eric R.
Coleman, Richard L.
MetadataShow full item record
Subject Matter Experts (SMEs) are commonly used in cost risk analysis (and in other fields as well) for values that either are not available in historical data or for which no appropriate analogy can be found. Problems commonly arise in two areas in particular: (1) when multiple experts give opinions on a single effect or entity and the inputs are not identical in distribution (which is almost inevitable), and (2) when a single expert provides distributional information that is intractable or suspiciously unlikely in its form (which is common).This paper will put forward correct solutions in case (1), in which the authors'' experience shows that practitioners (and even experts) use incorrect solutions. It is important to note that the commonly exercised incorrect solution underestimates the dispersion, and thus the 80th percentile, in some cases by a large margin. The authors believe that their solution is rare and, further, are unaware of any use of the solution, and will recommend tenets to guide the practitioner. In preparation for the solutions laid out above, the authors will first describe the method of expert-based risk analysis, with the erroneous method for combining SME testimony, and then show the correction. An analytical treatment will quantify the impacts of the erroneous approach. The paper will also explain why the new method of conflating expert assessments is to be preferred to the common Delphi technique, which may fall prey to both anchoring and domination by a vocal minority. The paper will also briefly address case (2) by presenting common examples of problematic formulations and proposed resolutions. These include intractable specification of a triangular distribution, specification of a discrete categorical distribution when triangular was intended, and specification of a triangular with low and high values that are not the true extremes as well as errors committed by the risk analyst. In any situation, correct treatment of risk is important. In the current era, with 80th percentiles required for all weapon systems cost estimates by the Weapon Systems Acquisition Reform Act of 2009, and budgeting to the 80th percentile as the default practice, the correct determination of the distribution is more important than ever before.
Proceedings Paper (for Acquisition Research Program)
RightsThis publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.
NPS Report NumberNPS-AM-10-031
Showing items related by title, author, creator and subject.
Evans, Donald Vincent (Monterey, California. Naval Postgraduate School, 1982-12);This thesis presents a method for decomposing a specification of a problem into a sequence of subproblem specifications. The method uses the specification to build a tree-like structure called a semantic net. The net is ...
Homeland Security Affairs Journal, Supplement - 2012: IEEE 2011 Conference on Technology for Homeland Security: Best Papers Naval Postgraduate School Center for Homeland Defense and Security (CHDS) (Monterey, California. Naval Postgraduate SchoolCenter for Homeland Defense and Security, 2012);IEEE Supplement 2012. Supplement: IEEE 2011 Conference on Technology for Homeland Security: Best Papers. As the field of homeland defense and security expands and matures, the contributions from various disciplines become ...
Ayik, Nejdet (Monterey, California. Naval Postgraduate School, 1989);This thesis discusses the formal specification techniques for communication protocols and the ANSI/IEEE Standard 802.5 Token Ring Access Method and Physical Layer Specification. Background information on formal protocol ...