基于強化學習的工控系統惡意軟件行為檢測方法

高洋; 王禮偉; 任望; 謝豐; 莫曉鋒; 羅熊; 王衛蘋; 楊璽

doi:10.13374/j.issn2095-9389.2019.09.16.005

基于強化學習的工控系統惡意軟件行為檢測方法

doi: 10.13374/j.issn2095-9389.2019.09.16.005

高洋¹,
王禮偉^{2, 3, 4},
任望¹,
謝豐¹,
莫曉鋒^{2, 3, 4},
羅熊^{2, 3, 4, ,},
王衛蘋^{2, 3, 4},
楊璽⁵

1.
中國信息安全測評中心，北京 100085
2.
北京科技大學計算機與通信工程學院，北京 100083
3.
北京科技大學人工智能研究院，北京 100083
4.
材料領域知識工程北京市重點實驗室，北京 100083
5.
北京市智能物流系統協同創新中心，北京 101149

基金項目: 國家自然科學基金資助項目(U1736117，U1836106)；北京市自然科學基金資助項目(19L2029，9204028)；北京市智能物流系統協同創新中心開放課題資助項目(BILSCIC-2019KF-08)；北京科技大學順德研究生院科技創新專項資金資助項目(BK19BF006)；材料領域知識工程北京市重點實驗室基本業務費資助項目(FRF-BD-19-012A)

詳細信息

通訊作者:
E-mail：xluo@ustb.edu.cn

中圖分類號: TP273
計量
- 文章訪問數: 2807
- HTML全文瀏覽量: 1529
- PDF下載量: 132
- 被引次數: 0
出版歷程
- 收稿日期: 2019-09-15
- 刊出日期: 2020-04-01

Reinforcement learning-based detection method for malware behavior in industrial control systems

GAO Yang¹,
WANG Li-wei^{2, 3, 4},
REN Wang¹,
XIE Feng¹,
MO Xiao-feng^{2, 3, 4},
LUO Xiong^{2, 3, 4
, ,},
WANG Wei-ping^{2, 3, 4},
YANG Xi⁵

1.
China Information Technology Security Evaluation Center, Beijing 100085, China
2.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
3.
Institute of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China
4.
Beijing Key Laboratory of Knowledge Engineering for Materials Science, Beijing 100083, China
5.
Beijing Intelligent Logistics System Collaborative Innovation Center, Beijing 101149, China

More Information

Corresponding author: E-mail: xluo@ustb.edu.cn

摘要

摘要: 網絡環境下的惡意軟件嚴重威脅著工控系統的安全，隨著目前惡意軟件變種的逐漸增多，給工控系統惡意軟件的檢測和安全防護帶來了巨大的挑戰。現有的檢測方法存在著自適應檢測識別的智能化程度不高等局限性。針對此問題，圍繞威脅工控系統網絡安全的惡意軟件對象，本文通過結合利用強化學習這一高級的機器學習算法，設計了一個檢測應用方法框架。在實現過程中，根據惡意軟件行為檢測的實際需求，充分結合強化學習的序列決策和動態反饋學習等智能特征，詳細討論并設計了其中的特征提取網絡、策略網絡和分類網絡等關鍵應用模塊。基于惡意軟件實際測試數據集進行的應用實驗驗證了本文方法的有效性，可為一般惡意軟件行為檢測提供一種智能化的決策輔助手段。
- 惡意軟件 /
- 檢測方法 /
- 強化學習 /
- 特征提取 /
- 策略網絡
Abstract: Due to the popularity of intelligent mobile devices, malwares in the internet have seriously threatened the security of industrial control systems. Increasing number of malware attacks has become a major concern in the information security community. Currently, with the increase of malware variants in a wide range of application fields, some technical challenges must be addressed to detect malwares and achieve security protection in industrial control systems. Although many traditional solutions have been developed to provide effective ways of detecting malwares, some current approaches have their limitations in intelligently detecting and recognizing malwares, as more complex malwares exist. Given the success of machine learning methods and techniques in data analysis applications, some advanced algorithms can also be applied in the detection and analysis of complex malwares. To detect malwares and consider the advantages of machine learning algorithms, we developed a detection framework for malwares that threatens the network security of industrial control systems through the combination of an advanced machine learning algorithm, i.e., reinforcement learning. During the implementation process, according to the actual needs of malware behavior detection, key modules including feature extraction, policy, and classification networks were designed on the basis of the intelligent features of reinforcement learning algorithms in relation to sequence decision and dynamic feedback learning. Moreover, the training algorithms for the above key modules were presented while providing the detailed functional analysis and implementation framework. In the application experiments, after preprocessing the actual dataset of malwares, the developed method was tested and the satisfactory classification performance for malware was achieved that verified the efficiency and effectiveness of the reinforcement learning-based method. This method can provide an intelligent decision aid for general malware behavior detection.
- malware /
- detection method /
- reinforcement learning /
- feature extraction /
- policy network

HTML全文

圖 1 總體結構

Figure 1. Framework

下載: 全尺寸圖片幻燈片

圖 2 測試集上的準確率

Figure 2. Accuracy in the test dataset

下載: 全尺寸圖片幻燈片

圖 3 測試集上查準率和查全率隨迭代次數的變化

Figure 3. Precision and recall in the test dataset

下載: 全尺寸圖片幻燈片

表 1 分類結果的混淆矩陣

Table 1. Confusion matrix

Confusion matrix	Prediction : malicious	Prediction : benign
Truth : malicious	257 (TP)	43 (FN)
Truth : benign	4 (FP)	296 (TN)

下載: 導出CSV

表 2 刪除比例最高和最低的各5個API函數

Table 2. Five API functions with the highest and lowest deletion rates

API Functions	Number of deleting operation	Number of retaining operation	Rate of deleting operation
VirtualAllocEx	174	209	0.454308
IsDBCSLeadByte	89	135	0.397321
GetSystemDirectoryA	101	206	0.328990
CreateThread	38	106	0.263889
GetDC	82	229	0.263666
GetProcAddress	0	2883	0
CloseHandle	0	2853	0
LocalFree	0	1939	0
GetModuleFileNameW	0	1485	0
lstrlenW	0	1460	0

下載: 導出CSV

259luxu-164

參考文獻(17)

[1]	Shi Y J. Research on the Key Security Issues of Mobile and Open Industrial Control System[Dissertation]. Beijing: Beijing University of Posts and Telecommunications, 2016 時憶杰. 移動互聯環境下工業控制系統安全問題研究[學位論文]. 北京: 北京郵電大學, 2016
[2]	Demontis A, Melis M, Biggio B, et al. Yes, machine learning can be more secure! A case study on android malware detection. IEEE Trans Dependable Secure Comput, 2019, 16(4): 711 doi: 10.1109/TDSC.2017.2700270
[3]	Sharif M, Lanzi A, Giffin J, et al. Impeding malware analysis using conditional code obfuscation // Proceedings of the Network and Distributed System Security Symposium. San Diego, 2008: 1939
[4]	Xiao X, Wang Z, Li Q, et al. Back-propagation neural network on Markov chains from system call sequences: a new approach for detecting Android malware with system call sequences. IET Inf Secur, 2016, 11(1): 8
[5]	Su X, Zhang D F, Li W J, et al. A deep learning approach to android malware feature learning and detection // 2016 IEEE Trustcom/BigDataSE/ISPA. Tianjin, 2016: 244
[6]	Li G L, Gomez R, Nakamura K, et al. Human-centered reinforcement learning: a survey. IEEE Trans Human Mach Syst, 2019, 49(4): 337 doi: 10.1109/THMS.2019.2912447
[7]	Wu C S, Shi J Y, Yang Y X, et al. Enhancing machine learning based malware detection model by reinforcement learning // Proceedings of the 8th International Conference on Communication and Network Security. Qingdao, 2018: 74
[8]	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518(7540): 529 doi: 10.1038/nature14236
[9]	Schultz M, Eskin E, Zadok F, et al. Data mining methods for detection of new malicious executables // Proceedings of the IEEE Symposium on Security and Privacy. Oakland, 2001: 38
[10]	Santos I, Brezo F, Ugarte-Pedrero X, et al. Opcode sequences as representation of executables for data-mining-based unknown malware detection. Inf Sci, 2013, 231: 64 doi: 10.1016/j.ins.2011.08.020
[11]	Zhang J X, Qin Z, Yin H, et al. IRMD: Malware variant detection using opcode image recognition // Proceedings of the IEEE 22nd International Conference on Parallel and Distributed Systems. Wuhan, 2016: 1175
[12]	Tandon G, Chan P. Learning rules from system call arguments and sequences for anomaly detection // Proceedings of the International Workshop on Data Mining for Computer Security. Melbourne, 2003: 20
[13]	Willems C, Holz T, Freiling F. Toward automated dynamic malware analysis using CWSandbox. IEEE Secur Privacy, 2007, 5(2): 32 doi: 10.1109/MSP.2007.45
[14]	Rieck K, Trinius P, Willems C, et al. Automatic analysis of malware behavior using machine learning. J Comput Secur, 2011, 19(4): 639 doi: 10.3233/JCS-2010-0410
[15]	Ki Y, Kim E, Kim H K. A novel approach to detect malware based on API call sequence analysis. Int J Distrib Sens Netw, 2015, 11(6): 659101 doi: 10.1155/2015/659101
[16]	Busoniu L, Babu?ka R, De Schutter B. A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C Appl Rev, 2008, 38(2): 156 doi: 10.1109/TSMCC.2007.913919
[17]	Zhang T Y, Huang M L, Zhao L, et al. Learning structured representation for text classification via reinforcement learning // Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. New Orleans, 2018: 6053