Sequential multiple comparison testing for budget-limited applications

Abstract

Computer simulations which forecast the performance of complicated systems are used as decision aids in many applications. For example, a ship's defensive system may use simulation to support an automated real-time response to a perceived threat, such as an incoming missile. The system uses cumulative simulation data to evaluate a set of options in order to choose the best countermeasure. Since everything happens in "real time", the system has limited time to run the simulation. Normally, a system would run the simulation an equal number of times for each option before coming to a decision. But this may cause the system to waste time on options which can be deemed non-optimal after only a few simulation runs. This time can be better used to help adjudicate between the better options. We evaluate the performance of sequential multiple comparisons algorithms to eliminate inferior options as quickly as possible, in order to have more time to dedicate to the exploration of better options, so that better decisions may be made. These algorithms allow inferior options to be dropped quickly depending on how well separated they are from others, but the algorithms differ in how well they achieve this objective.