Mathematical Modeling for Optimal System Testing under Fixed-cost Constraint
Pfeiffer, Karl D.
Kanevsky, Valery A.
Housel, Thomas J.
MetadataShow full item record
Testing of complex systems is a fundamentally difficult task, whether locating faults (diagnostic testing) or implementing upgrades (regression testing). Branch paths through the system increase as a function of the number of components and interconnections, leading to exponential growth in the number of test cases for exhaustive examination. In practice, the typical cost for testing in schedule or in budget means that only a small fraction of these paths are investigated. Given some fixed cost, then, which tests should we execute to guarantee the greatest information returned for the effort? In this work, we develop an approach to system testing using an abstract model flexible enough to be applied to both diagnostic and regression testing, grounded in a mathematical model suitable for rigorous analysis and Monte Carlo simulation. Early results indicate that in many cases of interest, a good, though not optimal, solution to the fixed-constraint problem (how many tests for budget x?) can be approached as a simple best-next strategy (which test returns the highest information per unit cost?). The goal of this modeling work is to construct a decision-support tool for the Navy Program Executive Office Integrated Warfare Systems (PEO IWS) offering quantitative information about cost versus diagnostic certainty in system testing.
Proceedings Paper (for Acquisition Research Program)Approved for public release; distribution unlimited.
NPS Report NumberNPS-AM-09-023
Showing items related by title, author, creator and subject.
Mathematical modeling for risk-based system testing / by Karl D. Pfeiffer, Valery A. Kanevsky, and Thomas J. Housel. Pfeiffer, Karl D.; Kanevsky, Valery A.; Housel, Thomas J. (Monterey, California. Naval Postgraduate School, 2009); NPS-GSBPP-09-028Testing of complex systems is a fundamentally difficult task whether locating faults (diagnostic testing) or implementing upgrades (regression testing). Branch paths through the system increase as a function of the number ...
Stanford, David Leland (Monterey, California: U.S. Naval Postgraduate School, 1966);The first section of this paper is concerned with determining the required lower confidence limit that must be met by testing after a missile system becomes operational. Some of the costs of making decisions about the ...
Reducing the cost of risk-based testing: management of testing options to manage risk in test and evaluation / by Karl D. Pfeiffer, Valery A. Kanevsky, Thomas J. Housel. Pfeiffer, Karl D.; Kanevsky, Valery A.; Housel, Thomas J. (Monterey, California. Naval Postgraduate School, 2009); NPS-GSBPP-09-027In the acquisition or management of complex systems, testing is the means by which we trade budget or schedule for information about the likelihood our system will work correctly under operational load. Branch paths in ...