Detecting anomalies from big network traffic data using an adaptive detection approach

Zhang, Ji and Li, Hongzhou and Gao, Qigang and Wang, Hai and Luo, Yonglong (2015) Detecting anomalies from big network traffic data using an adaptive detection approach. Information Sciences, 318. pp. 91-110. ISSN 0020-0255

Abstract

The unprecedented explosion of real-life big data sets have sparked a lot of research interests in data mining in recent years. Many of these big data sets are generated in network environment and are characterized by a dauntingly large size and high dimensionality which pose great challenges for detecting useful knowledge and patterns, such as network traffic anomalies, from them. In this paper, we study the problem of anomaly detection in big network connection data sets and propose an outlier detection technique, called Adaptive Stream Projected Outlier deTector (A-SPOT), to detect anomalies from large data sets using a novel adaptive subspace analysis approach. A case study of A-SPOT is conducted in this paper by deploying it to the 1999 KDD CUP anomaly detection application. Innovative approaches for training data generation, anomaly classification and false positive reduction are proposed in this paper as well to better tailor A-SPOT to deal with the case study. Experimental results demonstrate that A-SPOT is effective and efficient in detecting anomalies from network data sets and outperforms existing detection methods.


Statistics for USQ ePrint 26761
Statistics for this ePrint Item
Item Type: Article (Commonwealth Reporting Category C)
Refereed: Yes
Item Status: Live Archive
Additional Information: Permanent restricted access to Published version due to publisher copyright policy.
Faculty / Department / School: Current - Faculty of Health, Engineering and Sciences - School of Agricultural, Computational and Environmental Sciences
Date Deposited: 24 Feb 2015 06:01
Last Modified: 10 Jul 2015 04:51
Uncontrolled Keywords: anomaly detection; big data; outlier detection
Fields of Research : 08 Information and Computing Sciences > 0801 Artificial Intelligence and Image Processing > 080109 Pattern Recognition and Data Mining
09 Engineering > 0906 Electrical and Electronic Engineering > 090609 Signal Processing
08 Information and Computing Sciences > 0804 Data Format > 080401 Coding and Information Theory
Socio-Economic Objective: B Economic Development > 89 Information and Communication Services > 8901 Communication Networks and Services > 890103 Mobile Data Networks and Services
Identification Number or DOI: 10.1016/j.ins.2014.07.044
URI: http://eprints.usq.edu.au/id/eprint/26761

Actions (login required)

View Item Archive Repository Staff Only