Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment

FU Meixia; WANG Jianquan; WANG Qu; SUN Lei; MA Zhangchao; ZHANG Chaoyi; GUAN Wanqing; LI Wei

doi:10.13374/j.issn2095-9389.2022.12.18.001

Volume 45 Issue 10

Oct. 2023

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2023 > 45(10): 1666-1673

FU Meixia, WANG Jianquan, WANG Qu, SUN Lei, MA Zhangchao, ZHANG Chaoyi, GUAN Wanqing, LI Wei. Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment[J]. Chinese Journal of Engineering, 2023, 45(10): 1666-1673. doi: 10.13374/j.issn2095-9389.2022.12.18.001

Citation:

FU Meixia, WANG Jianquan, WANG Qu, SUN Lei, MA Zhangchao, ZHANG Chaoyi, GUAN Wanqing, LI Wei. Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment[J]. Chinese Journal of Engineering, 2023, 45(10): 1666-1673. doi: 10.13374/j.issn2095-9389.2022.12.18.001

Citation:

FU Meixia, WANG Jianquan, WANG Qu, SUN Lei, MA Zhangchao, ZHANG Chaoyi, GUAN Wanqing, LI Wei. Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment[J]. Chinese Journal of Engineering, 2023, 45(10): 1666-1673. doi: 10.13374/j.issn2095-9389.2022.12.18.001

PDF( 2685 KB)

Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment

doi: 10.13374/j.issn2095-9389.2022.12.18.001

FU Meixia^{1, 2},
WANG Jianquan^{1, 2
,
,},
WANG Qu^{1, 2},
SUN Lei^{1, 2},
MA Zhangchao^{1, 2},
ZHANG Chaoyi^{1, 2},
GUAN Wanqing³,
LI Wei^{2, 3}

1.
School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China
2.
Institute of Industrial Internet, University of Science and Technology Beijing, Beijing 100083, China
3.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China

More Information

Corresponding author: E-mail: wangjianquan@ustb.edu.cn
Received Date: 2022-12-18
Available Online: 2023-02-27
Publish Date: 2023-10-25

Abstract

Abstract

Intellectualization and unmanned manufacturing have been an inevitable trend in industrial development. The landing of intelligent applications is one of the current challenges in the industry. Due to the hierarchical architecture of the industrial automation pyramid, traditional programmable logic controllers (PLCs) that are usually employed in the field cannot cooperate with artificial intelligence (AI) algorithms that require massive data and computing resources. Therefore, it is necessary to research the virtualization of traditional PLCs as dockers, which can be deployed in the cloud, edge, or field. Cloud PLCs can be easily integrated with AI, big data, and cloud computing to achieve intelligent decision-making and control and break down data islands. The visual sorting system has attracted increasing attention for its ability to accurately detect the position of objects. Many deep learning–based methods have achieved remarkable performance in computer vision. Additionally, the requirement of a network is fundamental for guaranteeing data transmission with low latency and high reliability. The combination of 5G and time-sensitive networking (TSN) can achieve the deterministic transmission of several industrial applications. According to the above challenges, joint control between cloud PLCs of low-level devices and visual sorting systems in a reliable network is critical and has industry potential. In this study, we propose a deep learning–based material recognition and location system with a cloud PLC, which is demonstrated in a 5G-TSN network. First, traditional PLC is virtualized to realize flexible PLC function deployment in the field and cloud. Second, we establish a cloud-based AI platform and design a You only look once v5 (YOLOv5)-based object detection algorithm to locate the position and recognize the types of materials to obtain pixel coordinates. Third, the camera calibration method is used to transform pixel and world coordinates, and the material information consists of the world coordinates, types, and timestamps that are sent to cloud PLC. Finally, the commands are transmitted by the 5G-TSN environment from cloud PLC to the low-level devices for real-time control of the multi-crane cooperative. We establish an experimental system to demonstrate the significance and effectiveness of the proposed scheme, which synergistically controls multi-crane operation. The mean average precision (mAP) of material location is up to 99.65%, sorting accuracy reaches 96.67%, and the average consuming time is 25.99 s, which meets the requirements of low latency and high precision in industrial applications.
- intelligent manufacturing,
- cloud PLC,
- visual sorting system,
- deep learning,
- 5G,
- TSN

FullText(HTML)

References(30)

References

[1]	周濟. 智能制造——“中國制造2025”的主攻方向. 中國機械工程, 2015, 26(17):2273 Zhou J. Intelligent manufacturing—Main direction of “made in China 2025”. China Mech Eng, 2015, 26(17): 2273
[2]	Zhong R Y, Xu X, Klotz E, et al. Intelligent manufacturing in the context of industry 4.0: A review. Engineering, 2017, 3(5): 616
[3]	薛塬, 臧冀原, 孔德婧, 等. 面向智能制造的產業模式演變與創新應用. 機械工程學報, 2022, 58(18):303 doi: 10.3901/JME.2022.18.303 Xue Y, Zang J Y, Kong D J, et al. Evolution and innovative implementation of industrial model for intelligent manufacturing. J Mech Eng, 2022, 58(18): 303 doi: 10.3901/JME.2022.18.303
[4]	王健全, 李衛, 馬彰超, 等. 5G工業互聯網賦能智慧鋼鐵. 鋼鐵, 2021, 56(9):56 Wang J Q, Li W, Ma Z C, et al. 5G industrial internet empowers smart steel. Iron Steel, 2021, 56(9): 56
[5]	Krupa P, Limon D, Alamo T. Implementation of model predictive control in programmable logic controllers. IEEE Trans Control Syst Technol, 2021, 29(3): 1117 doi: 10.1109/TCST.2020.2992959
[6]	Vazquez-Gonzalez J L, Barrios-Aviles J, Rosado-Mu?oz A, et al. An industrial automation course: Common infrastructure for physical, virtual and remote laboratories for PLC programming. Int J Online Eng, 2018, 14(8): 4 doi: 10.3991/ijoe.v14i08.8758
[7]	Llano A, Angulo I, de la Vega D, et al. Virtual PLC lab enabled physical layer improvement proposals for PRIME and G3-PLC standards. Appl Sci, 2020, 10(5): 1777 doi: 10.3390/app10051777
[8]	Zhang C M, Lu Y. Study on artificial intelligence: The state of the art and future prospects. J Ind Inf Integr, 2021, 23(1): 100224
[9]	Wang J, Yang Y Q, Wang T, et al. Big data service architecture: a survey. J Internet Technol, 2020, 21(2): 393
[10]	Sadeeq M M, Abdulkareem N M, Zeebaree S R M, et al. IoT and cloud computing issues, challenges and opportunities: A review. Qubahan Academic J, 2021, 1(2): 1 doi: 10.48161/qaj.v1n2a36
[11]	金燕, 代皇, 李書齊. 基于視覺檢測的PLC智能控制系統設計. 機床與液壓, 2021, 49(23):113 Jin Y, Dai H, Li S Q. Design of PLC intelligent control system based on visual inspection. Mach Tool Hydraul, 2021, 49(23): 113
[12]	Steger C, Ulrich M, Wiedemann C. Machine Vision Algorithms and Applications. New Jersey: Wiley-VCH, 2018
[13]	楊利, 謝永超. 基于PLC和機器視覺的工件自動分揀系統設計. 工業儀表與自動化裝置, 2022(1):48 Yang L, Xie Y C. Design of automatic sorting system for workpieces based on PLC and machine machine vision. Ind Instrum Autom, 2022(1): 48
[14]	Azari M M, Solanki S, Chatzinotas S, et al. Evolution of non-terrestrial networks from 5G to 6G: A survey. IEEE Commun Surv Tutor, 2022, 24(4): 2633 doi: 10.1109/COMST.2022.3199901
[15]	Zhao L X, Pop P, Steinhorst S. Quantitative performance comparison of various traffic shapers in time-sensitive networking. IEEE Trans Netw Serv Manag, 2022, 19(3): 2899 doi: 10.1109/TNSM.2022.3180160
[16]	Larrañaga A, Lucas-Estañ M C, Martinez I, et al. Analysis of 5G-TSN integration to support industry 4.0 // 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Vienna, 2020: 1111
[17]	LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436 doi: 10.1038/nature14539
[18]	Zaidi S S A, Ansari M S, Aslam A, et al. A survey of modern deep learning based object detection models. Digit Signal Process, 2022, 126: 103514 doi: 10.1016/j.dsp.2022.103514
[19]	Zhao Z Q, Zheng P, Xu S T, et al. Object detection with deep learning: A review. IEEE Trans Neural Netw Learn Syst, 2019, 30(11): 3212 doi: 10.1109/TNNLS.2018.2876865
[20]	Liu L, Ouyang W L, Wang X G, et al. Deep learning for generic object detection: A survey. Int J Comput Vis, 2020, 128(2): 261 doi: 10.1007/s11263-019-01247-4
[21]	Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 779
[22]	Liu W, Anguelov D, Erhan D, et al. SSD: Single shot multibox detector // European Conference on Computer Vision. Netherlands, 2016: 21
[23]	Cai Z, Vasconcelos N. Cascade r-cnn: Delving into high quality object detection // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 6154
[24]	Jocher G, Nishimura K, Mineeva T, et al. Yolov5 code repository [EB/OL]. Website Online (2022-11-22) [2022-12-18]. https://github.com/ultralytics/yolov5
[25]	Wang C Y, Liao H YM, Wu Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, 2020: 1571
[26]	Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, 2019: 658
[27]	Chamveha I, Promwiset T, Tongdee T, et al. Automated cardiothoracic ratio calculation and cardiomegaly detection using deep learning approach. arXiv preprint (2020-2-18) [2022-12-18]. https://arxiv.org/abs/2002.07468
[28]	Zhang Z. A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell, 2000, 22(11): 1330 doi: 10.1109/34.888718
[29]	Henderson P, Ferrari V. End-to-end training of object class detectors for mean average precision // Asian Conference on Computer Vision. Taipei, 2016: 198
[30]	Kingma D P, Ba J. Adam: A method for stochastic optimization. arXiv preprint (2014-12-22) [2022-12-18]. https://arxiv.org/abs/1412.6980