5G環境下基于深度學習的云化PLC物料識別與定位系統

付美霞; 王健全; 王曲; 孫雷; 馬彰超; 張超一; 管婉青; 李衛

doi:10.13374/j.issn2095-9389.2022.12.18.001

5G環境下基于深度學習的云化PLC物料識別與定位系統

doi: 10.13374/j.issn2095-9389.2022.12.18.001

付美霞^{1, 2},
王健全^{1, 2, ,},
王曲^{1, 2},
孫雷^{1, 2},
馬彰超^{1, 2},
張超一^{1, 2},
管婉青³,
李衛^{2, 3}

1.
北京科技大學自動化學院，北京 100083
2.
北京科技大學工業互聯網研究院，北京 100083
3.
北京科技大學計算機與通信工程學院，北京 100083

基金項目: 國家重點研發計劃資助項目（2020YFB1708800）；廣東省重點研究與開發計劃資助項目（2020B010113007）；中央高校基本科研業務費專項資金資助項目（FRF-IDRY-21-005）；廣東省基礎與應用基礎研究基金聯合基金資助項目（2021A1515110577）；中央高校基礎研究基金資助項目（FRF-MP-20-37）；中國博士后科學基金資助項目（2021M700385）

詳細信息

通訊作者:
E-mail: wangjianquan@ustb.edu.cn

中圖分類號: TG142.71
計量
- 文章訪問數: 280
- HTML全文瀏覽量: 48
- PDF下載量: 41
- 被引次數: 0
出版歷程
- 收稿日期: 2022-12-18
- 網絡出版日期: 2023-02-27
- 刊出日期: 2023-10-25

Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment

FU Meixia^{1, 2},
WANG Jianquan^{1, 2
, ,},
WANG Qu^{1, 2},
SUN Lei^{1, 2},
MA Zhangchao^{1, 2},
ZHANG Chaoyi^{1, 2},
GUAN Wanqing³,
LI Wei^{2, 3}

1.
School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
Institute of Industrial Internet, University of Science and Technology Beijing, Beijing 100083, China
3.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China

More Information

Corresponding author: E-mail: wangjianquan@ustb.edu.cn

摘要

摘要: 為了解決智能制造領域中云化控制與視覺分選應用相結合的問題，提出了基于深度學習的云化可編程邏輯控制器（Programmable logic controller，PLC）物料識別與定位系統，并在端到端5G與時間敏感網絡（Time sensitive networking，TSN）傳輸網絡環境下，實現了對云化PLC架構和控制功能有效性的驗證。首先，將傳統PLC系統控制功能容器虛擬化，實現PLC的本地和云端自由部署；其次，在云端設計人工智能學習平臺，采用基于You only look once v5 （YOLOv5）的目標檢測算法實現物料的定位和分類，獲取目標的像素坐標和類別信息；然后，利用相機標定方法把像素坐標轉換成物理世界坐標，并將目標類別、坐標、時間戳信息傳輸到云化PLC；最后，在5G和TSN融合網絡環境下，實現云化PLC對天車設備的實時控制與復雜計算功能整合。結果表明，該系統能夠有效的對多天車進行協同控制，物料定位均值平均精度（Mean average precision，mAP）達到99.65%，分選準確率達到96.67%，平均消耗時間225.99 s，滿足工業低時延、高精度的視覺分選需求。
- 智能制造 /
- 視覺分選 /
- 深度學習 /
- 云化可編程邏輯控制器 /
- 5G /
- 時間敏感網絡
Abstract: Intellectualization and unmanned manufacturing have been an inevitable trend in industrial development. The landing of intelligent applications is one of the current challenges in the industry. Due to the hierarchical architecture of the industrial automation pyramid, traditional programmable logic controllers (PLCs) that are usually employed in the field cannot cooperate with artificial intelligence (AI) algorithms that require massive data and computing resources. Therefore, it is necessary to research the virtualization of traditional PLCs as dockers, which can be deployed in the cloud, edge, or field. Cloud PLCs can be easily integrated with AI, big data, and cloud computing to achieve intelligent decision-making and control and break down data islands. The visual sorting system has attracted increasing attention for its ability to accurately detect the position of objects. Many deep learning–based methods have achieved remarkable performance in computer vision. Additionally, the requirement of a network is fundamental for guaranteeing data transmission with low latency and high reliability. The combination of 5G and time-sensitive networking (TSN) can achieve the deterministic transmission of several industrial applications. According to the above challenges, joint control between cloud PLCs of low-level devices and visual sorting systems in a reliable network is critical and has industry potential. In this study, we propose a deep learning–based material recognition and location system with a cloud PLC, which is demonstrated in a 5G-TSN network. First, traditional PLC is virtualized to realize flexible PLC function deployment in the field and cloud. Second, we establish a cloud-based AI platform and design a You only look once v5 (YOLOv5)-based object detection algorithm to locate the position and recognize the types of materials to obtain pixel coordinates. Third, the camera calibration method is used to transform pixel and world coordinates, and the material information consists of the world coordinates, types, and timestamps that are sent to cloud PLC. Finally, the commands are transmitted by the 5G-TSN environment from cloud PLC to the low-level devices for real-time control of the multi-crane cooperative. We establish an experimental system to demonstrate the significance and effectiveness of the proposed scheme, which synergistically controls multi-crane operation. The mean average precision (mAP) of material location is up to 99.65%, sorting accuracy reaches 96.67%, and the average consuming time is 25.99 s, which meets the requirements of low latency and high precision in industrial applications.
- intelligent manufacturing /
- cloud PLC /
- visual sorting system /
- deep learning /
- 5G /
- TSN

HTML全文

圖 1 云化PLC系統架構

Figure 1. Architecture of cloud based-PLC system architecture

下載: 全尺寸圖片幻燈片

圖 2 基于YOLOv5的物料識別與檢測網絡

Figure 2. Material recognition and detection network based on YOLOv5

下載: 全尺寸圖片幻燈片

圖 3 相機校準示意圖

Figure 3. Mathematical model of camera calibration

下載: 全尺寸圖片幻燈片

圖 4 試驗設備

Figure 4. Main experimental devices

下載: 全尺寸圖片幻燈片

圖 5 象棋棋子檢測效果展示

Figure 5. Experimental results of material detection based on YOLOv5

下載: 全尺寸圖片幻燈片

圖 6 多天車物料分揀效果展示

Figure 6. Performance of the multi-crane visual sorting system

下載: 全尺寸圖片幻燈片

表 1 基于物料檢測的實驗結果

Table 1. Experimental results of material detection

Model	Dataset	Number	Precision/%	Recall rate/%	mAP/%	Time/ms
Cascade-RCNN^[23]	Training set	141	100	100	100	32
	Testing set	30	100	100	100
	Validating set	30	100	100	100
SSD^[22]	Training set	141	100	100	100	24
	Testing set	30	100	100	100
	Validating set	30	100	100	100
YOLOv5^[24]	Training set	141	99.99	100	99.60	12
	Testing set	30	100	100	99.59
	Validating set	30	100	100	99.65

下載: 導出CSV

表 2 多天車視覺分選系統的消耗時間和準確率

Table 2. Consuming time and accuracy of the multi-crane visual sorting system

No.	Conveyor belt speed/(cm·s^–1)	Consuming time/s	Missed detection of red chesses pieces	Missed detection of black chesses pieces	All accuracy/%
1	2.23	27.07	5/0	5/0	100
2		28.55	5/0	5/0	100
3		26.79	5/0	5/0	100
4	2.61	25.68	5/0	5/0	100
5		25.76	5/0	5/0	100
6		25.40	5/0	5/0	100
7	3.01	23.38	5/0	5/0	100
8		30.33	5/1	5/0	90
9		27.92	5/0	5/1	90
10	3.39	23.60	5/2	5/0	80
11		24.16	5/0	5/0	100
12		23.35	5/0	5/0	100
Average		25.99	60/3	60/1	96.67

下載: 導出CSV

259luxu-164

參考文獻(30)

[1]	Zhou J. Intelligent manufacturing—Main direction of “made in China 2025”. China Mech Eng, 2015, 26(17): 2273 周濟. 智能制造——“中國制造2025”的主攻方向. 中國機械工程, 2015, 26(17):2273
[2]	Zhong R Y, Xu X, Klotz E, et al. Intelligent manufacturing in the context of industry 4.0: A review. Engineering, 2017, 3(5): 616
[3]	Xue Y, Zang J Y, Kong D J, et al. Evolution and innovative implementation of industrial model for intelligent manufacturing. J Mech Eng, 2022, 58(18): 303 doi: 10.3901/JME.2022.18.303 薛塬, 臧冀原, 孔德婧, 等. 面向智能制造的產業模式演變與創新應用. 機械工程學報, 2022, 58(18):303 doi: 10.3901/JME.2022.18.303
[4]	Wang J Q, Li W, Ma Z C, et al. 5G industrial internet empowers smart steel. Iron Steel, 2021, 56(9): 56 王健全, 李衛, 馬彰超, 等. 5G工業互聯網賦能智慧鋼鐵. 鋼鐵, 2021, 56(9):56
[5]	Krupa P, Limon D, Alamo T. Implementation of model predictive control in programmable logic controllers. IEEE Trans Control Syst Technol, 2021, 29(3): 1117 doi: 10.1109/TCST.2020.2992959
[6]	Vazquez-Gonzalez J L, Barrios-Aviles J, Rosado-Mu?oz A, et al. An industrial automation course: Common infrastructure for physical, virtual and remote laboratories for PLC programming. Int J Online Eng, 2018, 14(8): 4 doi: 10.3991/ijoe.v14i08.8758
[7]	Llano A, Angulo I, de la Vega D, et al. Virtual PLC lab enabled physical layer improvement proposals for PRIME and G3-PLC standards. Appl Sci, 2020, 10(5): 1777 doi: 10.3390/app10051777
[8]	Zhang C M, Lu Y. Study on artificial intelligence: The state of the art and future prospects. J Ind Inf Integr, 2021, 23(1): 100224
[9]	Wang J, Yang Y Q, Wang T, et al. Big data service architecture: a survey. J Internet Technol, 2020, 21(2): 393
[10]	Sadeeq M M, Abdulkareem N M, Zeebaree S R M, et al. IoT and cloud computing issues, challenges and opportunities: A review. Qubahan Academic J, 2021, 1(2): 1 doi: 10.48161/qaj.v1n2a36
[11]	Jin Y, Dai H, Li S Q. Design of PLC intelligent control system based on visual inspection. Mach Tool Hydraul, 2021, 49(23): 113 金燕, 代皇, 李書齊. 基于視覺檢測的PLC智能控制系統設計. 機床與液壓, 2021, 49(23):113
[12]	Steger C, Ulrich M, Wiedemann C. Machine Vision Algorithms and Applications. New Jersey: Wiley-VCH, 2018
[13]	Yang L, Xie Y C. Design of automatic sorting system for workpieces based on PLC and machine machine vision. Ind Instrum Autom, 2022(1): 48 楊利, 謝永超. 基于PLC和機器視覺的工件自動分揀系統設計. 工業儀表與自動化裝置, 2022(1):48
[14]	Azari M M, Solanki S, Chatzinotas S, et al. Evolution of non-terrestrial networks from 5G to 6G: A survey. IEEE Commun Surv Tutor, 2022, 24(4): 2633 doi: 10.1109/COMST.2022.3199901
[15]	Zhao L X, Pop P, Steinhorst S. Quantitative performance comparison of various traffic shapers in time-sensitive networking. IEEE Trans Netw Serv Manag, 2022, 19(3): 2899 doi: 10.1109/TNSM.2022.3180160
[16]	Larrañaga A, Lucas-Estañ M C, Martinez I, et al. Analysis of 5G-TSN integration to support industry 4.0 // 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Vienna, 2020: 1111
[17]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436 doi: 10.1038/nature14539
[18]	Zaidi S S A, Ansari M S, Aslam A, et al. A survey of modern deep learning based object detection models. Digit Signal Process, 2022, 126: 103514 doi: 10.1016/j.dsp.2022.103514
[19]	Zhao Z Q, Zheng P, Xu S T, et al. Object detection with deep learning: A review. IEEE Trans Neural Netw Learn Syst, 2019, 30(11): 3212 doi: 10.1109/TNNLS.2018.2876865
[20]	Liu L, Ouyang W L, Wang X G, et al. Deep learning for generic object detection: A survey. Int J Comput Vis, 2020, 128(2): 261 doi: 10.1007/s11263-019-01247-4
[21]	Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 779
[22]	Liu W, Anguelov D, Erhan D, et al. SSD: Single shot multibox detector // European Conference on Computer Vision. Netherlands, 2016: 21
[23]	Cai Z, Vasconcelos N. Cascade r-cnn: Delving into high quality object detection // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 6154
[24]	Jocher G, Nishimura K, Mineeva T, et al. Yolov5 code repository [EB/OL]. Website Online (2022-11-22) [2022-12-18]. https://github.com/ultralytics/yolov5
[25]	Wang C Y, Liao H YM, Wu Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, 2020: 1571
[26]	Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, 2019: 658
[27]	Chamveha I, Promwiset T, Tongdee T, et al. Automated cardiothoracic ratio calculation and cardiomegaly detection using deep learning approach. arXiv preprint (2020-2-18) [2022-12-18]. https://arxiv.org/abs/2002.07468
[28]	Zhang Z. A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell, 2000, 22(11): 1330 doi: 10.1109/34.888718
[29]	Henderson P, Ferrari V. End-to-end training of object class detectors for mean average precision // Asian Conference on Computer Vision. Taipei, 2016: 198
[30]	Kingma D P, Ba J. Adam: A method for stochastic optimization. arXiv preprint (2014-12-22) [2022-12-18]. https://arxiv.org/abs/1412.6980