Resilient real-time network anomaly detection using novel non-parametric statistical tests

Abstract

This work describes a novel application of robust estimation to the detection of volumetric anomalies in computer network traffic. The proposed tests are based on sample location and dispersion and derived from relatively unknown Zero Order Statistics. The proposed tests are non-parametric and suitable for a range of applications to heavy-tailed data analysis outside of network traffic. The performance of these tests is examined using two different real-world denial-of-service attacks contained in actual high-volume backbone traffic. The proposed tests outperform traditional metrics such as mean and variance due to the presence of heavy tails in the network traffic, a frequent characteristic of traffic in actual networks. Monte Carlo analysis is used to quantify the performance gains and show an improvement in accuracy between 7 and 11% at very low false alarm rates. The proposed tests also demonstrate equivalent or superior performance to the median, a common robust statistic. Constructive timing of key system processes is used to demonstrate near real-time perfor- mance. Three- and six- second data windows containing between 750 and 1200 elements can be processed in less than one second using commodity hardware running unoptimized code. These timing results imply scalability to a variety of networks and commercial ap- plications. Scalability prospects are further enhanced by demonstrating resilient detection performance at attack volumes between 25 and 100 percent of baseline rates in both real and generated traffic.