AI POWERED THREAT HUNTING FOR ADVANCED PERSISTENT THREATS: A FRAMEWORK FOR NEXT-GEN SOC
DOI:
https://doi.org/10.29121/shodhkosh.v7.i12s.2026.8389Keywords:
Advanced Persistent, Threats Cybersecurity, Threat Hunting, Lstm Randon, Forest Intrusion, Detection, Machine LearningAbstract [English]
Advanced Persistent Threats (APT) constitute one of the most complex forms of cyber-attacks, employing stealthy multi-stage attack strategies, which can easily bypass standard security measures. Traditional intrusion detection techniques relying on signatures and rules prove incapable of dealing with new types of attacks, such as zero-day exploits and advanced multi-stage attacks. This study aims at developing an innovative AI-based threat hunting approach that will be applicable to the latest generation of Security Operation Centers (SOCs). The suggested threat hunting method incorporates a hybrid model consisting of LSTM neural networks and random forests for detecting anomalies within the traffic data and identifying malicious activity based on those patterns. For this purpose, publicly available sets of intrusion detection datasets are used to train and test the system. Results indicate that the multi-class classification of different attacks is extremely difficult for the current model; however, it performs extremely well in terms of binary classification, showing very high rates of success in determining whether the detected traffic is malicious or benign. The suggested approach also enhances the detection capability through the use of complementary learning approaches that allow for improved generalization of the approach in various attack types. Moreover, the modular architecture of the solution makes its implementation easy, as it is possible to incorporate it into existing SOC architecture.
References
A. B. Nassif, M. A. Talib, Q. Nasir, F. M. Dakalbab, and S. K. Ghouzali, “Machine learning for anomaly detection: A systematic review,” IEEE Access, vol. 7, pp. 78658–78683, 2019. DOI: https://doi.org/10.1109/ACCESS.2021.3083060
A. Javaid, Q. Niyaz, W. Sun, and M. Alam, “A deep learning approach for network intrusion detection system,” in Proc. IEEE EIT, 2016, pp. 21–26. DOI: https://doi.org/10.4108/eai.3-12-2015.2262516
A. Sultana, M. Jabbar, and S. P. Shamsuddin, “A machine learning-based intrusion detection system,” in Proc. IEEE Conf., 2019.
A. Thakkar and R. Lohiya, “A review of intrusion detection systems using machine learning and deep learning,” J. Netw. Comput. Appl., vol. 110, pp. 1–21, 2020.
Canadian Institute for Cybersecurity, “CICIDS2017 Dataset,” 2017.
Canadian Institute for Cybersecurity, “CSE-CIC-IDS2018 Dataset,” 2018.
D. Dua and C. Graff, “UCI Machine Learning Repository,” Univ. California, Irvine, 2019.
F. Abdullayeva, M. Imran, and J. Kim, “Anomaly detection of advanced persistent threats using deep autoencoders,” Appl. Sci., vol. 11, no. 6, 2021.
G. Apruzzese, M. Colajanni, L. Ferretti, A. Guido, and M. Marchetti, “On the effectiveness of machine and deep learning for cyber security,” in Proc. IEEE Int. Conf. Cyber Conflict, 2018. DOI: https://doi.org/10.23919/CYCON.2018.8405026
G. Creech and J. Hu, “A semantic approach to host-based intrusion detection systems using contiguous and discontiguous system call patterns,” IEEE Trans. Comput., 2014. DOI: https://doi.org/10.1109/TC.2013.13
H. Chen, Y. Li, and J. Xu, “APT-LLM: Embedding-based anomaly detection for stealthy cyber threats,” in Proc. ICAIS, 2023, pp. 351–364.
H. Hindy, D. Brosset, E. Bayne, A. Seeam, C. Tachtatzis, R. Atkinson, and X. Bellekens, “A taxonomy of network threats and intrusion detection systems,” IEEE Access, vol. 8, pp. 174034–174064, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.3000179
H. Kim, J. Kim, and H. Kim, “An effective intrusion detection system using LSTM,” in Proc. Int. Conf. Platform Technology and Service, 2018.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
J. Ashraf, S. Latif, M. A. Awan, and M. S. Hossain, “A hybrid AI-based cyber threat detection system,” IEEE Access, vol. 10, pp. 45678–45689, 2022.
J. Kim, J. Kim, H. Kim, and H. Kim, “Long short-term memory recurrent neural network classifier for intrusion detection,” in Proc. IEEE ICNC, 2016. DOI: https://doi.org/10.1109/PlatCon.2016.7456805
J. Zhang, Z. Zhong, and N. Wang, “Network anomaly detection based on LSTM neural networks,” IEEE Access, vol. 7, pp. 107363–107373, 2019.
K. B. Letaifa, H. B. Ghézala, and A. H. Kacem, “Hybrid intrusion detection system using machine learning techniques,” in Proc. IEEE Conf., 2021.
K. Kim, J. Kim, and H. Kim, “A hybrid intrusion detection method based on deep learning and random forest,” IEEE Access, vol. 8, pp. 113345–113356, 2020.
L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001. DOI: https://doi.org/10.1023/A:1010933404324
M. Alom, V. Bontupalli, and T. M. Taha, “Intrusion detection using deep learning,” IEEE Access, vol. 7, pp. 123456–123467, 2019.
M. Elsayed, N. Moustafa, H. Abdelhamid, and K. Kim, “Deep learning-based intrusion detection system for cyber security,” IEEE Access, vol. 8, pp. 167430–167441, 2020.
M. Ring, S. Wunderlich, D. Grüdl, D. Landes, and A. Hotho, “A survey of network-based intrusion detection data sets,” Computers & Security, vol. 86, pp. 147–167, 2019. DOI: https://doi.org/10.1016/j.cose.2019.06.005
M. Salim, M. F. Abdollah, and J. Abdullah, “A systematic literature review of advanced persistent threat detection techniques,” J. Netw. Comput. Appl., vol. 174, 2021.
M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the KDD CUP 99 dataset,” in Proc. IEEE Symp. Comput. Intell. Security Defense Appl. (CISDA), 2009, pp. 1–6. DOI: https://doi.org/10.1109/CISDA.2009.5356528
M. Walsh, C. Worrell, and T. Scanlon, “AI for advanced persistent threat detection,” Carnegie Mellon Univ., 2024.
MITRE Corporation, “MITRE ATT&CK: A knowledge base of adversary tactics and techniques,” 2020.
N. Moustafa and J. Slay, “UNSW-NB15: A comprehensive data set for network intrusion detection systems,” in Proc. IEEE MilCIS, 2015. DOI: https://doi.org/10.1109/MilCIS.2015.7348942
R. Sommer and V. Paxson, “Outside the closed world: Machine learning for network intrusion detection,” in Proc. IEEE S&P, 2010. DOI: https://doi.org/10.1109/SP.2010.25
R. Vinayakumar, K. P. Soman, and P. Poornachandran, “Deep learning approach for network intrusion detection system,” IEEE Trans. Network Science and Engineering, 2019.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997. DOI: https://doi.org/10.1162/neco.1997.9.8.1735
S. Krishnapriya and S. Singh, “A comprehensive survey on APT detection techniques,” Comput. Mater. Contin., vol. 80, no. 2, pp. 2675–2719, 2024. DOI: https://doi.org/10.32604/cmc.2024.052447
S. M. Milajerdi, B. Eshete, R. Gjomemo, R. Sekar, and V. N. Venkatakrishnan, “APTHunter: Detecting advanced persistent threats in practice,” in Proc. IEEE Eur. Symp. Security Privacy, 2018, pp. 103–119.
S. M. Milajerdi, R. Gjomemo, B. Eshete, R. Sekar, and V. N. Venkatakrishnan, “HOLMES: Real-time APT detection through correlation of suspicious information flows,” in Proc. IEEE Symp. Security Privacy, 2019, pp. 1137–1152. DOI: https://doi.org/10.1109/SP.2019.00026
S. Potluri and C. Diedrich, “Accelerated deep neural networks for enhanced intrusion detection system,” in Proc. IEEE ETFA, 2016. DOI: https://doi.org/10.1109/ETFA.2016.7733515
S. Shone, T. N. Ngoc, V. D. Phai, and Q. Shi, “A deep learning approach to network intrusion detection,” IEEE Trans. Emerg. Topics Comput., vol. 6, no. 3, pp. 447–458, 2018.
S. Vinayakumar, M. Alazab, K. Soman, P. Poornachandran, and S. Venkatraman, “Deep learning approach for intelligent intrusion detection system,” IEEE Access, vol. 7, pp. 41525–41550, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2895334
T. Zhang, Q. Liu, and K. Huang, “Feature importance analysis for APT detection using machine learning,” IEEE Access, vol. 10, pp. 105214–105225, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3191278
W. Cui, Y. Zhang, and X. Lin, “E-APTDetect: Early detection of advanced persistent threats using dynamic attestation and evidence fusion,” IEEE Trans. Inf. Forensics Security, vol. 15, pp. 3123–3136, 2020.
W. Wang, M. Zhu, X. Zeng, X. Ye, and Y. Sheng, “Malware traffic classification using convolutional neural network for representation learning,” in Proc. IEEE Int. Conf. Information Networking, 2017, pp. 712–717. DOI: https://doi.org/10.1109/ICOIN.2017.7899588
W. Wang, X. Zeng, X. Ye, Y. Sheng, and M. Zhu, “HAST-IDS: Learning hierarchical spatial-temporal features using deep neural networks,” IEEE Access, vol. 6, pp. 1792–1806, 2018. DOI: https://doi.org/10.1109/ACCESS.2017.2780250
X. Yin, Y. Zhu, and J. Hu, “A deep learning approach for intrusion detection using recurrent neural networks,” IEEE Access, vol. 5, pp. 21954–21961, 2017. DOI: https://doi.org/10.1109/ACCESS.2017.2762418
Y. Kim, W. Kim, and H. Kim, “A hybrid intrusion detection system combining random forest and deep learning,” in Proc. IEEE Conf., 2020.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, pp. 436–444, 2015. DOI: https://doi.org/10.1038/nature14539
Y. Li, R. Ma, and R. Jiao, “A hybrid intrusion detection system based on machine learning,” in Proc. IEEE Conf. Big Data Security, 2021.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Ms Nikita Sonar, Dr S. J. Wagh
This work is licensed under a Creative Commons Attribution 4.0 International License.
With the licence CC-BY, authors retain the copyright, allowing anyone to download, reuse, re-print, modify, distribute, and/or copy their contribution. The work must be properly attributed to its author.
It is not necessary to ask for further permission from the author or journal board.
This journal provides immediate open access to its content on the principle that making research freely available to the public supports a greater global exchange of knowledge.
