Self-attention guided multi-sequence fusion model for differentiation of hepatocellular carcinoma

JIA Xi-bin; SUN Zheng; YANG Da-wei; YANG Zheng-han

doi:10.13374/j.issn2095-9389.2021.01.13.003

Volume 43 Issue 9

Sep. 2021

Turn off MathJax

Article Contents

Article Navigation > Chinese Journal of Engineering > 2021 > 43(9): 1149-1156

JIA Xi-bin, SUN Zheng, YANG Da-wei, YANG Zheng-han. Self-attention guided multi-sequence fusion model for differentiation of hepatocellular carcinoma[J]. Chinese Journal of Engineering, 2021, 43(9): 1149-1156. doi: 10.13374/j.issn2095-9389.2021.01.13.003

Citation:

JIA Xi-bin, SUN Zheng, YANG Da-wei, YANG Zheng-han. Self-attention guided multi-sequence fusion model for differentiation of hepatocellular carcinoma[J]. Chinese Journal of Engineering, 2021, 43(9): 1149-1156. doi: 10.13374/j.issn2095-9389.2021.01.13.003

Citation:

PDF( 828 KB)

Self-attention guided multi-sequence fusion model for differentiation of hepatocellular carcinoma

doi: 10.13374/j.issn2095-9389.2021.01.13.003

JIA Xi-bin^{1, 2},
SUN Zheng^{1, 2},
YANG Da-wei³,
YANG Zheng-han^{3
,
,}

1.
Faculty of Information, Beijing University of Technology, Beijing 100124, China
2.
Beijing Municipal Key Laboratory of Multimedia and Intelligent Software Technology, Beijing University of Technology, Beijing 100124, China
3.
Department of Radiology, Beijing Friendship Hospital, Capital Medical University, Beijing 100050, China

More Information

Corresponding author: E-mail: yangzhenghan@vip.163.com
Received Date: 2021-01-13
Available Online: 2021-03-20
Publish Date: 2021-09-18

Abstract

Abstract

Hepatocellular carcinoma (HCC) is a type of primary malignant tumor and an urgent problem to be solved, particularly in China, one of the countries with the highest prevalence of HCC. In the choice of treatment methods for patients with hepatocellular carcinoma, accurate histological grading of the lesion area undoubtedly plays a vital role that helps the management and therapy of HCC patients. However, the current pathological detection as the gold standard has defects, such as invasiveness and a large sampling error. Therefore, it is an important direction of intelligent medical treatment to provide noninvasive and accurate lesion grading using imaging technology combined with artificial intelligence technology. On the basis of the radiologists' experience in reading clinical images, this paper proposed a self-attentional guidance-based histological differentiation discrimination model combined with multi-modality fusion and an attention weight calculation scheme for dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) sequences of hepatocellular carcinoma. The model combined the spatiotemporal information contained in the enhancement sequence and learned the importance of each sequence and the slice in the sequence for the classification task. It effectively used the feature information contained in the enhancement sequence in the temporal and spatial dimensions to improve the classification performance. During the experiment, the model was trained and tested on the clinical data set of the top three hospitals in China. The experimental results show that the self-attention-guided model proposed in this paper achieves higher classification performance than several benchmark and mainstream models. Comprehensive experiments were performed on the clinical dataset with labels annotated by professional radiologists. The results show that our proposed self-attention model can achieve acceptable quantitative measuring of HCC histologic grading based on the MRI sequences. In the WHO histological classification task, the classification accuracy of the proposed model reaches 80%, the sensitivity is 82%, and the precision is 82%.
- grading of hepatocellular carcinoma,
- dynamic contrast-enhanced magnetic resonance imaging,
- self-attention mechanism,
- multimodal fusion,
- auxiliary diagnosis

FullText(HTML)

References(25)

References

[1]	Yang J D, Hainaut P, Gores G J, et al. A global view of hepatocellular carcinoma: trends, risk, prevention and management. Nat Rev Gastroenterol Hepatol, 2019, 16(10): 589 doi: 10.1038/s41575-019-0186-y
[2]	Jiang Y, Sun A H, Zhao Y, et al. Proteomics identifies new therapeutic targets of early-stage hepatocellular carcinoma. Nature, 2019, 567(7747): 257 doi: 10.1038/s41586-019-0987-8
[3]	Lin H X, Wei C, Wang G X, et al. Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning. J Biophoto, 2019, 12(7): e201800435
[4]	Jimenez H, Wang M H, Zimmerman J W, et al. Tumour-specific amplitude-modulated radiofrequency electromagnetic fields induce differentiation of hepatocellular carcinoma via targeting Ca_v3.2 T-type voltage-gated calcium channels and Ca²⁺ influx. EBioMedicine, 2019, 44: 209 doi: 10.1016/j.ebiom.2019.05.034
[5]	Shioga T, Kondo R, Ogasawara S, et al. Usefulness of tumor tissue biopsy for predicting the biological behavior of hepatocellular carcinoma. Anticancer Res, 2020, 40(7): 4105 doi: 10.21873/anticanres.14409
[6]	Parr R L, Mills J, Harbottle A, et al. Mitochondria, prostate cancer, and biopsy sampling error. Discov Med, 2013, 25;15(83): 213
[7]	Henken K, Van Gerwen D, Dankelman J, et al. Accuracy of needle position measurements using fiber Bragg gratings. Minim Invasive Ther Allied Technol, 2012, 21(6): 408 doi: 10.3109/13645706.2012.666251
[8]	Li J Z, Xue F, Xu X H, et al. Dynamic contrast enhanced MRI differentiates hepatocellular carcinoma from hepatic metastasis of rectal cancer by extracting pharmacokinetic parameters and radiomic features. Exp Ther Med, 2020, 20(4): 3643
[9]	Kaissis G A, Loh?fer F K, H?rl M, et al. Combined DCE-MRI- and FDG-PET enable histopathological grading prediction in a rat model of hepatocellular carcinoma. Eur J Radiol, 2020, 124: 108848 doi: 10.1016/j.ejrad.2020.108848
[10]	Khalifa F, Soliman A, El-Baz A, et al. Models and methods for analyzing DCE-MRI: A review. Med Phys, 2014, 41(12): 124301 doi: 10.1118/1.4898202
[11]	Yang D W, Jia X B, Xiao Y J, et al. Noninvasive evaluation of the pathologic grade of hepatocellular carcinoma using MCF-3DCNN: A pilot study. Biomed Res Int, 2019: 9783106
[12]	Chernyak V, Fowler K J, Kamaya A, et al. Liver imaging reporting and data system (LI-RADS) version 2018: Imaging of hepatocellular carcinoma in at-risk patients. Radiology, 2018, 289(3): 816 doi: 10.1148/radiol.2018181494
[13]	Suk H I, Lee S W, Shen D G. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage, 2014, 101: 569 doi: 10.1016/j.neuroimage.2014.06.077
[14]	Wang Q Y, Que D S. Staging of hepatocellular carcinoma using deep feature in contrast-enhanced MR images//2nd International Conference on Computer Engineering, Information Science & Application Technology (ICCIA 2017). Wuhan, 2016: 186
[15]	Jia X B, Xiao Y J, Yang D W, et al. Temporal-spatial feature learning of dynamic contrast enhanced-MR images via 3D convolutional neural networks//Chinese Conference on Image and Graphics Technologies. Singapore, 2018: 380
[16]	Jia X B, Xiao Y J, Yang D W, et al. Multi-parametric MRIs based assessment of Hepatocellular Carcinoma Differentiation with Multi-scale ResNet. TIIS, 2019, 13(10): 5179
[17]	Antropova N, Huynh B Q, Giger M L. A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Med Phys, 2017, 44(10): 5162 doi: 10.1002/mp.12453
[18]	Hu Q Y, Whitney H M, Giger M L. A deep learning methodology for improved breast cancer diagnosis using multiparametric MRI. Sci Rep, 2020, 10: 10536 doi: 10.1038/s41598-020-67441-4
[19]	張桃紅, 范素麗, 郭徐徐, 等. 基于數據融合的智能醫療輔助診斷方法. 工程科學學報, doi: 10.13374/j.issn2095-9389.2021.01.12.003 Zhang T H, Fan S L, Guo X X, et al. Intelligent medical assistant diagnosis method based on data fusion. Chin J Eng, doi: 10.13374/j.issn2095-9389.2021.01.12.003
[20]	Ye H, Chen Q J, Wu H M, et al. Classification of liver cancer images based on deep learning//International conference on Data Science, Medicine and Bioinformatics. Singapore, 2020: 184
[21]	Zhou L, Rui J G, Zhou W X, et al. Edmondson-Steiner grade: A crucial predictor of recurrence and survival in hepatocellular carcinoma without microvascular invasio. Pathol Res Pract, 2017, 213(7): 824 doi: 10.1016/j.prp.2017.03.002
[22]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770
[23]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, 2018: 7132
[24]	Zhou Q, Zhou Z Y, Chen C M, et al. Grading of hepatocellular carcinoma using 3D SE-DenseNet in dynamic enhanced MR images. Comput Biol Med, 2019, 107: 47 doi: 10.1016/j.compbiomed.2019.01.026
[25]	Yoshinobu Y, Iwamoto Y, Xianhua H A N, et al. Deep learning method for content-based retrieval of focal liver lesions using multiphase contrast-enhanced computer tomography images//2020 IEEE International Conference on Consumer Electronics (ICCE). Las Vegas, 2020: 1