Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment
-
摘要: 為了解決智能制造領域中云化控制與視覺分選應用相結合的問題,提出了基于深度學習的云化可編程邏輯控制器(Programmable logic controller,PLC)物料識別與定位系統,并在端到端5G與時間敏感網絡(Time sensitive networking,TSN)傳輸網絡環境下,實現了對云化PLC架構和控制功能有效性的驗證。首先,將傳統PLC系統控制功能容器虛擬化,實現PLC的本地和云端自由部署;其次,在云端設計人工智能學習平臺,采用基于You only look once v5 (YOLOv5)的目標檢測算法實現物料的定位和分類,獲取目標的像素坐標和類別信息;然后,利用相機標定方法把像素坐標轉換成物理世界坐標,并將目標類別、坐標、時間戳信息傳輸到云化PLC;最后,在5G和TSN融合網絡環境下,實現云化PLC對天車設備的實時控制與復雜計算功能整合。結果表明,該系統能夠有效的對多天車進行協同控制,物料定位均值平均精度(Mean average precision,mAP)達到99.65%,分選準確率達到96.67%,平均消耗時間225.99 s,滿足工業低時延、高精度的視覺分選需求。Abstract: Intellectualization and unmanned manufacturing have been an inevitable trend in industrial development. The landing of intelligent applications is one of the current challenges in the industry. Due to the hierarchical architecture of the industrial automation pyramid, traditional programmable logic controllers (PLCs) that are usually employed in the field cannot cooperate with artificial intelligence (AI) algorithms that require massive data and computing resources. Therefore, it is necessary to research the virtualization of traditional PLCs as dockers, which can be deployed in the cloud, edge, or field. Cloud PLCs can be easily integrated with AI, big data, and cloud computing to achieve intelligent decision-making and control and break down data islands. The visual sorting system has attracted increasing attention for its ability to accurately detect the position of objects. Many deep learning–based methods have achieved remarkable performance in computer vision. Additionally, the requirement of a network is fundamental for guaranteeing data transmission with low latency and high reliability. The combination of 5G and time-sensitive networking (TSN) can achieve the deterministic transmission of several industrial applications. According to the above challenges, joint control between cloud PLCs of low-level devices and visual sorting systems in a reliable network is critical and has industry potential. In this study, we propose a deep learning–based material recognition and location system with a cloud PLC, which is demonstrated in a 5G-TSN network. First, traditional PLC is virtualized to realize flexible PLC function deployment in the field and cloud. Second, we establish a cloud-based AI platform and design a You only look once v5 (YOLOv5)-based object detection algorithm to locate the position and recognize the types of materials to obtain pixel coordinates. Third, the camera calibration method is used to transform pixel and world coordinates, and the material information consists of the world coordinates, types, and timestamps that are sent to cloud PLC. Finally, the commands are transmitted by the 5G-TSN environment from cloud PLC to the low-level devices for real-time control of the multi-crane cooperative. We establish an experimental system to demonstrate the significance and effectiveness of the proposed scheme, which synergistically controls multi-crane operation. The mean average precision (mAP) of material location is up to 99.65%, sorting accuracy reaches 96.67%, and the average consuming time is 25.99 s, which meets the requirements of low latency and high precision in industrial applications.
-
Key words:
- intelligent manufacturing /
- cloud PLC /
- visual sorting system /
- deep learning /
- 5G /
- TSN
-
表 1 基于物料檢測的實驗結果
Table 1. Experimental results of material detection
Model Dataset Number Precision/% Recall rate/% mAP/% Time/ms Cascade-RCNN[23] Training set 141 100 100 100 32 Testing set 30 100 100 100 Validating set 30 100 100 100 SSD[22] Training set 141 100 100 100 24 Testing set 30 100 100 100 Validating set 30 100 100 100 YOLOv5[24] Training set 141 99.99 100 99.60 12 Testing set 30 100 100 99.59 Validating set 30 100 100 99.65 表 2 多天車視覺分選系統的消耗時間和準確率
Table 2. Consuming time and accuracy of the multi-crane visual sorting system
No. Conveyor belt
speed/(cm·s–1)Consuming
time/sMissed detection of red
chesses piecesMissed detection of
black chesses piecesAll accuracy/% 1 2.23 27.07 5/0 5/0 100 2 28.55 5/0 5/0 100 3 26.79 5/0 5/0 100 4 2.61 25.68 5/0 5/0 100 5 25.76 5/0 5/0 100 6 25.40 5/0 5/0 100 7 3.01 23.38 5/0 5/0 100 8 30.33 5/1 5/0 90 9 27.92 5/0 5/1 90 10 3.39 23.60 5/2 5/0 80 11 24.16 5/0 5/0 100 12 23.35 5/0 5/0 100 Average 25.99 60/3 60/1 96.67 259luxu-164 -
參考文獻
[1] Zhou J. Intelligent manufacturing—Main direction of “made in China 2025”. China Mech Eng, 2015, 26(17): 2273周濟. 智能制造——“中國制造2025”的主攻方向. 中國機械工程, 2015, 26(17):2273 [2] Zhong R Y, Xu X, Klotz E, et al. Intelligent manufacturing in the context of industry 4.0: A review. Engineering, 2017, 3(5): 616 [3] Xue Y, Zang J Y, Kong D J, et al. Evolution and innovative implementation of industrial model for intelligent manufacturing. J Mech Eng, 2022, 58(18): 303 doi: 10.3901/JME.2022.18.303薛塬, 臧冀原, 孔德婧, 等. 面向智能制造的產業模式演變與創新應用. 機械工程學報, 2022, 58(18):303 doi: 10.3901/JME.2022.18.303 [4] Wang J Q, Li W, Ma Z C, et al. 5G industrial internet empowers smart steel. Iron Steel, 2021, 56(9): 56王健全, 李衛, 馬彰超, 等. 5G工業互聯網賦能智慧鋼鐵. 鋼鐵, 2021, 56(9):56 [5] Krupa P, Limon D, Alamo T. Implementation of model predictive control in programmable logic controllers. IEEE Trans Control Syst Technol, 2021, 29(3): 1117 doi: 10.1109/TCST.2020.2992959 [6] Vazquez-Gonzalez J L, Barrios-Aviles J, Rosado-Mu?oz A, et al. An industrial automation course: Common infrastructure for physical, virtual and remote laboratories for PLC programming. Int J Online Eng, 2018, 14(8): 4 doi: 10.3991/ijoe.v14i08.8758 [7] Llano A, Angulo I, de la Vega D, et al. Virtual PLC lab enabled physical layer improvement proposals for PRIME and G3-PLC standards. Appl Sci, 2020, 10(5): 1777 doi: 10.3390/app10051777 [8] Zhang C M, Lu Y. Study on artificial intelligence: The state of the art and future prospects. J Ind Inf Integr, 2021, 23(1): 100224 [9] Wang J, Yang Y Q, Wang T, et al. Big data service architecture: a survey. J Internet Technol, 2020, 21(2): 393 [10] Sadeeq M M, Abdulkareem N M, Zeebaree S R M, et al. IoT and cloud computing issues, challenges and opportunities: A review. Qubahan Academic J, 2021, 1(2): 1 doi: 10.48161/qaj.v1n2a36 [11] Jin Y, Dai H, Li S Q. Design of PLC intelligent control system based on visual inspection. Mach Tool Hydraul, 2021, 49(23): 113金燕, 代皇, 李書齊. 基于視覺檢測的PLC智能控制系統設計. 機床與液壓, 2021, 49(23):113 [12] Steger C, Ulrich M, Wiedemann C. Machine Vision Algorithms and Applications. New Jersey: Wiley-VCH, 2018 [13] Yang L, Xie Y C. Design of automatic sorting system for workpieces based on PLC and machine machine vision. Ind Instrum Autom, 2022(1): 48楊利, 謝永超. 基于PLC和機器視覺的工件自動分揀系統設計. 工業儀表與自動化裝置, 2022(1):48 [14] Azari M M, Solanki S, Chatzinotas S, et al. Evolution of non-terrestrial networks from 5G to 6G: A survey. IEEE Commun Surv Tutor, 2022, 24(4): 2633 doi: 10.1109/COMST.2022.3199901 [15] Zhao L X, Pop P, Steinhorst S. Quantitative performance comparison of various traffic shapers in time-sensitive networking. IEEE Trans Netw Serv Manag, 2022, 19(3): 2899 doi: 10.1109/TNSM.2022.3180160 [16] Larrañaga A, Lucas-Estañ M C, Martinez I, et al. Analysis of 5G-TSN integration to support industry 4.0 // 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Vienna, 2020: 1111 [17] LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436 doi: 10.1038/nature14539 [18] Zaidi S S A, Ansari M S, Aslam A, et al. A survey of modern deep learning based object detection models. Digit Signal Process, 2022, 126: 103514 doi: 10.1016/j.dsp.2022.103514 [19] Zhao Z Q, Zheng P, Xu S T, et al. Object detection with deep learning: A review. IEEE Trans Neural Netw Learn Syst, 2019, 30(11): 3212 doi: 10.1109/TNNLS.2018.2876865 [20] Liu L, Ouyang W L, Wang X G, et al. Deep learning for generic object detection: A survey. Int J Comput Vis, 2020, 128(2): 261 doi: 10.1007/s11263-019-01247-4 [21] Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 779 [22] Liu W, Anguelov D, Erhan D, et al. SSD: Single shot multibox detector // European Conference on Computer Vision. Netherlands, 2016: 21 [23] Cai Z, Vasconcelos N. Cascade r-cnn: Delving into high quality object detection // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 6154 [24] Jocher G, Nishimura K, Mineeva T, et al. Yolov5 code repository [EB/OL]. Website Online (2022-11-22) [2022-12-18]. https://github.com/ultralytics/yolov5 [25] Wang C Y, Liao H YM, Wu Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, 2020: 1571 [26] Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, 2019: 658 [27] Chamveha I, Promwiset T, Tongdee T, et al. Automated cardiothoracic ratio calculation and cardiomegaly detection using deep learning approach. arXiv preprint (2020-2-18) [2022-12-18]. https://arxiv.org/abs/2002.07468 [28] Zhang Z. A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell, 2000, 22(11): 1330 doi: 10.1109/34.888718 [29] Henderson P, Ferrari V. End-to-end training of object class detectors for mean average precision // Asian Conference on Computer Vision. Taipei, 2016: 198 [30] Kingma D P, Ba J. Adam: A method for stochastic optimization. arXiv preprint (2014-12-22) [2022-12-18]. https://arxiv.org/abs/1412.6980 -