With the advent of the big data era, anomaly detection becomes increasingly crucial for ensuring the security and reliability of systems. This paper investigates large-scale anomaly detection based on the Isolation Forest algorithm, enhancing the algorithm’s performance in the context of big data by introducing the method of adaptive feature selection. The proposed approach is a fusion of the Isolation Forest and adaptive feature selection, dynamically adjusting feature weights to adapt more flexibly to the contributions of different features. Experimental results on large-scale datasets demonstrate that adaptive feature selection significantly improves the anomaly detection performance of the Isolation Forest algorithm. This method provides a new perspective for enhancing anomaly detection techniques and addressing the challenges posed by large-scale, high-dimensional data. Its practical implications are crucial for real-world applications.
SamariyaDThakkarA. A comprehensive survey of anomaly detection algorithms. Annals of Data Science2023; 10(3): 829–850.
2.
JainMKaurGSaxenaV. A K-Means clustering and SVM based hybrid concept drift detection technique for network anomaly detection. Expert Syst Appl2022; 193: 116510.
3.
NassifABTalibMANasirQ, et al.Machine learning for anomaly detection: a systematic review. IEEE Access2021; 9: 78658–78700.
4.
Ur RehmanABelhaouariSB. Unsupervised outlier detection in multidimensional data. J Big Data2021; 8(1): 80.
5.
GadalSMokhtarRAbdelhaqM, et al.Machine learning-based anomaly detection using K-mean array and sequential minimal optimization. Electronics2022; 11(14): 2158.
6.
KwonDKimHKimJ, et al.A survey of deep learning-based network anomaly detection. Cluster Computing, 2019, vol 22, pp. 949–961.
7.
LesoupleJBaudoinCSpigaiM, et al.Generalized isolation forest for anomaly detection. Pattern Recognit Lett2021; 149: 109–119.
8.
DomańskiPD. Study on statistical outlier detection and labelling. Int J Autom Comput2020; 17(6): 788–811.
9.
RipanRCSarkerIHHossainSMM, et al.A data-driven heart disease prediction model through K-means clustering-based anomaly detection. SN Computer Science2021; 2(2): 112.
10.
MaoJWangHSpencerJBF. Toward data anomaly detection for automated structural health monitoring: exploiting generative adversarial nets and autoencoders. Struct Health Monit2021; 20(4): 1609–1626.
11.
KopčanJŠkvarekOKlimoM. Anomaly detection using autoencoders and deep convolution generative adversarial networks. Transp Res Procedia2021; 55: 1296–1303.
12.
AlalharethMHongSC. An improved mutual information feature selection technique for intrusion detection systems in the Internet of Medical Things. Sensors2023; 23(10): 4971.
13.
MohammadiMRashidTAKarimSHT, et al.A comprehensive survey and taxonomy of the SVM-based intrusion detection systems. J Netw Comput Appl2021; 178: 102983.
14.
NakashimaMSimAKimY, et al.Automated feature selection for anomaly detection in network traffic data. ACM Trans Manag Inf Syst2021; 12(3): 1–28.
15.
MasenoEMWangZ. Hybrid wrapper feature selection method based on genetic algorithm and extreme learning machine for intrusion detection. J Big Data2024; 11(1): 24.
16.
AwadMFraihatS. Recursive feature elimination with cross-validation with decision tree: feature selection method for machine learning-based intrusion detection systems. J Sens Actuator Netw2023; 12(5): 67.
17.
SahaSPriyotiATSharmaA, et al.Towards an optimized ensemble feature selection for DDoS detection using both supervised and unsupervised method. Sensors2022; 22(23): 9144.
18.
MaldonadoJRiffMCNeveuB. A review of recent approaches on wrapper feature selection for intrusion detection. Expert Syst Appl2022; 198: 116822.
19.
SadafKSultanaJ. Intrusion detection based on autoencoder and isolation forest in fog computing. IEEE Access2020; 8: 167059–167068.
20.
ChabchoubYTogbeMUBolyA, et al.An in-depth study and improvement of isolation forest. IEEE Access2022; 10: 10219–10237.
21.
LaskarMTRHuangJXSmetanaV, et al.Extending isolation forest for anomaly detection in big data via K-means. ACM Trans Cyber-Phys Syst2021; 5(4): 1–26.
22.
LinWZhaoXWangY, et al.AdaFS: adaptive feature selection in deep recommender system. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. New York: Association for Computing Machinery, 2022, pp. 3309–3317.
23.
LiuXWangSLuS, et al.Adapting feature selection algorithms for the classification of Chinese texts. Systems2023; 11(9): 483.
24.
SureshkumarSPrasannaGKDSanthoshR. Adaptive butterfly optimization algorithm (ABOA) based feature selection and deep neural network (DNN) for detection of distributed denial-of-service (DDoS) attacks in cloud. Comput Syst Sci Eng2023; 47(1): 1109–1123.
25.
ShiDZhuLLiJ, et al.Unsupervised adaptive feature selection with binary hashing. IEEE Trans Image Process2023; 32: 838–853.
26.
TavallaeeMBagheriELuW, et al.A detailed analysis of the KDD CUP 99 data set. In: 2009 IEEE symposium on computational intelligence for security and defense applications. Piscataway: IEEE, 2009, pp. 1–6.
27.
HanSZhuKZhouM, et al.Competition-driven multimodal multiobjective optimization and its application to feature selection for credit card fraud detection. IEEE Trans Syst Man Cybern Syst2022; 52(12): 7845–7857.
28.
CohenGAfsharSTapsonJ, et al.EMNIST: extending MNIST to handwritten letters. In: 2017 international joint conference on neural networks (IJCNN). Piscataway: IIEEE, 2017, pp. 2921–2926.
29.
BalaRNagpalR. A review on kdd cup99 and nsl nsl-kdd dataset. Int J Adv Res Comput Sci2019; 10(2): 64–67.
30.
ChengZZouCDongJ. Outlier detection using isolation forest and local outlier factor. Proceedings of the conference on research in adaptive and convergent systems. New York: Association for Computing Machinery, 2019, pp. 161–168.
31.
PangJPuXLiC. A hybrid algorithm incorporating vector quantization and one-class support vector machine for industrial anomaly detection. IEEE Trans Industr Inform2022; 18(12): 8786–8796.
