Finite-sample performance of absolute precision stopping rules
Abstract
Absolute precision stopping rules are often used to determine the length of sequential experiments to estimate confidence intervals for simulated performance measures. Much is known about the asymptotic behavior of such procedures. In this paper, we introduce coverage contours to quantify the trade-offs in
interval coverage, stopping times, and precision for finite-sample experiments using absolute precision rules. We use these contours to evaluate the coverage of a basic absolute precision stopping rule, and we show that this rule will lead to a bias in coverage even if all of the assumptions supporting the procedure are true. We define optimal stopping rules that deliver nominal coverage with the smallest expected number of observations. Contrary to previous asymptotic results that suggest decreasing the precision of the rule to approach nominal coverage in the limit, we find that it is optimal to increase the confidence coefficient used in the stopping rule, thus obtaining nominal coverage in a finite-sample experiment. If the simulation data are
independent and identically normally distributed, we can calculate coverage contours analytically and find a stopping rule that is insensitive to the variance of the data while delivering at least nominal coverage for any precision value.
Description
The article of record as published may be found at http://dx.doi.org/10.1287/ijoc.1110.0471
Rights
This publication is a work of the U.S. Government as defined in Title 17, United States Code, Section 101. Copyright protection is not available for this work in the United States.Collections
Related items
Showing items related by title, author, creator and subject.
-
Selecting stopping rules for confidence interval procedures
Singham, Dashi I. (ACM, 2014);The sample size decision is crucial to the success of any sampling experiment. More samples imply better confidence and precision in the results, but require higher costs in terms of time, computing power, and money. ... -
Analysis of sequential stopping rules
Singham, Dashi I.; Schruben, Lee W. (2009);Sequential stopping rules applied to confidence interval procedures (CIPs) may lead to coverage that is less than nominal. This paper introduces a method for estimating coverage functions analytically in order to evaluate ... -
Capturing the full potential of the Synthetic Theater Operations Research Model (STORM)
Seymour, Christian N. (Monterey, California: Naval Postgraduate School, 2014-09);The Synthetic Theater Operations Research Model (STORM) is the primary campaign analysis tool used by the Office of the Chief of Naval Operations, Assessment Division (OPNAV N81) and other Department of Defense organizations ...