Finite-sample performance of absolute precision stopping rules
Singham, Dashi I.
Schruben, Lee W.
MetadataShow full item record
Absolute precision stopping rules are often used to determine the length of sequential experiments to estimate confidence intervals for simulated performance measures. Much is known about the asymptotic behavior of such procedures. In this paper, we introduce coverage contours to quantify the trade-offs in interval coverage, stopping times, and precision for finite-sample experiments using absolute precision rules. We use these contours to evaluate the coverage of a basic absolute precision stopping rule, and we show that this rule will lead to a bias in coverage even if all of the assumptions supporting the procedure are true. We define optimal stopping rules that deliver nominal coverage with the smallest expected number of observations. Contrary to previous asymptotic results that suggest decreasing the precision of the rule to approach nominal coverage in the limit, we find that it is optimal to increase the confidence coefficient used in the stopping rule, thus obtaining nominal coverage in a finite-sample experiment. If the simulation data are independent and identically normally distributed, we can calculate coverage contours analytically and find a stopping rule that is insensitive to the variance of the data while delivering at least nominal coverage for any precision value.
The article of record as published may be found at http://dx.doi.org/10.1287/ijoc.1110.0471
RightsThis publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.
Showing items related by title, author, creator and subject.
Singham, Dashi I. (ACM, 2014);The sample size decision is crucial to the success of any sampling experiment. More samples imply better confidence and precision in the results, but require higher costs in terms of time, computing power, and money. ...
Singham, Dashi I.; Schruben, Lee W. (2009);Sequential stopping rules applied to confidence interval procedures (CIPs) may lead to coverage that is less than nominal. This paper introduces a method for estimating coverage functions analytically in order to evaluate ...
Seymour, Christian N. (Monterey, California: Naval Postgraduate School, 2014-09);The Synthetic Theater Operations Research Model (STORM) is the primary campaign analysis tool used by the Office of the Chief of Naval Operations, Assessment Division (OPNAV N81) and other Department of Defense organizations ...