阅读说明：本技术 一种内窥镜超声影像的处理方法及系统 (Method and system for processing ultrasonic image of endoscope ) 是由 戴文睿 李锦� 李成林 邹君妮 熊红凯 于 2020-12-25 设计创作，主要内容包括：本发明提供一种内窥镜超声影像的处理方法及系统,所述方法包括：获取CP-EBUS成像技术中三个模式的超声影像,其中,三个模式是指弹性模式E、灰阶模式B和多普勒血流模式F；采用多模态神经网络对三个模式的所述超声影像进行分类,得到识别淋巴结的处理结果。本发明使用CP-EBUS成像技术中三个模式的图像数据,通过多模态神经网络对这些图像进行处理,能够更为准确的识别超声影像的淋巴结,从而有助于提升淋巴结诊断的准确率。(The invention provides a method and a system for processing an endoscope ultrasonic image, wherein the method comprises the following steps: acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F; and classifying the ultrasonic images in the three modes by adopting a multi-modal neural network to obtain a processing result for identifying the lymph nodes. The invention uses the image data of three modes in the CP-EBUS imaging technology, processes the images through the multi-modal neural network, can more accurately identify the lymph nodes of the ultrasonic images, and is favorable for improving the accuracy of lymph node diagnosis.)

1. An endoscopic ultrasound image processing method, comprising:

acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;

and classifying the ultrasonic images in the three modes by adopting a multi-modal neural network to obtain a processing result for identifying the lymph nodes.

2. The method for processing endoscopic ultrasound images according to claim 1, wherein said classifying the ultrasound images of three modes using a multi-modal neural network comprises:

extracting the image characteristics of the ultrasonic image of each mode;

performing feature fusion on the image features of each mode to obtain image fusion features of the three modes;

and classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.

3. The method for processing endoscopic ultrasound images according to claim 1, wherein said extracting image features of ultrasound images of each mode comprises:

inputting the ultrasonic image of each mode into a neural network;

the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.

4. The method for processing endoscopic ultrasound images according to claim 1, wherein said feature fusion of image features of each mode comprises:

and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.

5. The endoscopic ultrasound image processing method according to claim 1, wherein the feature fusion is implemented by using a plurality of feature fusion modules, and the fusion operation of each feature fusion module is divided into four ways;

for the extracted image characteristics of the three mode ultrasonic images, directly outputting the received original signals by using three paths respectively, and slightly outputting the original signals by using a fourth path according to the relative position of the characteristic fusion module;

for the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths; for the subsequent fusion operation of the feature fusion module, firstly, the three features are subjected to weighted connection, and then, the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion.

6. The method for processing endoscopic ultrasound images as claimed in claim 1, further comprising an image preprocessing performed before said extracting image features of ultrasound images of each mode, so that the preprocessed ultrasound images of each modality are suitable as inputs of a neural network.

7. The method for processing endoscopic ultrasound images according to claim 6, wherein said image preprocessing comprises:

and automatically positioning the scanning frames in the elastic E mode image and the Doppler blood flow F mode image, and selecting a key analysis area by using a minimum rectangular frame according to the positioning.

8. The method for processing endoscopic ultrasound images according to claim 7, wherein said automatically positioning the scan frames in the elastic E-mode and Doppler blood flow F-mode images, and selecting the key analysis area using the minimum rectangular frame according to the positioning comprises:

giving an image of elastic E-mode or Doppler flow F-modeAnd obtaining a mask matrix m set according to the corresponding modal imaging ruleBy removing extraneous information in this step, a preprocessed intermediate signal is obtained

Then, the minimum rectangle containing the detection frame is determined through a frame detection operator G,whereinIs the original signal contained by the smallest rectangle.

9. The method for processing endoscopic ultrasound images according to claim 1 or 7, wherein the image preprocessing comprises:

and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.

10. The method for processing endoscopic ultrasound images according to claim 8, wherein said evaluating the quality of the images according to the characteristics of each mode comprises:

-for the elastic E-mode image, determining colored pixels according to the variance of the three channel values at each position, if the proportion of colored pixels is less than a threshold T_eConsidering the image quality as unqualified, otherwise, continuously calculating the average intensity of the image, and if the intensity is greater than the threshold value T_IIf the image quality is qualified, otherwise, the image quality is unqualified;

for images of the Doppler flow F-mode, determining colored pixels from the variance of the three channel values at each location, if the proportion of colored pixels is less than a threshold T_fThe image quality is considered to be qualified, otherwise, the image quality is not qualified;

for images in grayscale B mode, the quality is qualified by default.

11. An endoscopic ultrasound image processing system, comprising:

the image acquisition module is used for acquiring ultrasonic images of three modes in the CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;

and the image processing module is used for classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes.

12. The endoscopic ultrasound image processing system according to claim 11, wherein said image processing module comprises:

the characteristic extraction module is used for extracting the image characteristics of the ultrasonic image of each mode;

the characteristic fusion module is used for carrying out characteristic fusion on the image characteristics of each mode to obtain the image fusion characteristics of the three modes;

and the classifier is used for classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.

13. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program being adapted to perform the method of processing endoscopic ultrasound images as claimed in any one of claims 1 to 10.

14. A computer-readable storage medium on which a computer program is stored, characterized in that the program, when being executed by a processor, is adapted to carry out the method of processing endoscopic ultrasound images as claimed in any one of claims 1 to 10.

Technical Field

The invention relates to the technical field of medical image processing, in particular to a method and a system for processing an endoscope ultrasonic image.

Background

The ultrasonic endoscope is an endoscope which fills some special indications which can not be covered by common endoscopes, body surface ultrasound, CT and the like. CP-EBUS (convex probe ultrasonic bronchoscope) is a minimally invasive intrathoracic lymph node diagnosis technique, which uses a probe equipped with a biopsy needle, an ultrasonic device and an endoscope to accurately enter the vicinity of a lymph node to be detected from a bronchus, and uses the biopsy needle to obtain the tissue components of the target lymph node, and can accurately diagnose the disease of a patient through pathological analysis of the extracted tissue. However, due to the small amount of tissue taken from a biopsy, the diagnosis of a biopsy is false negative by 20%. In the process of obtaining biopsy tissues, the ultrasonic device can obtain ultrasonic images of lymph nodes under different modalities, and the prior researches show that the ultrasonic images have important value for diagnosing the benign and malignant lymph nodes. According to the diagnosis result of the ultrasonic image, a doctor can be helped to select a proper lymph node for puncture in the biopsy process, and the diagnosis method can also be used as a supplement of the biopsy diagnosis to make up for the defect of the biopsy diagnosis.

Existing diagnostics for CP-EBUS images are classified as semi-quantitative or quantitative methods, where semi-quantitative methods rely primarily on physician observation of certain features in the image, e.g., the contour of lymph nodes in gray-scale B-mode images, which tend to be malignant if sharp and benign otherwise. The limitation of this type of method is that the accuracy of the diagnosis depends on the individual experience of the doctor, since the sharpness of the same B-mode image varies from person to person. The method is difficult to be applied in the areas with deficient medical resources, and simultaneously, the stability of the diagnosis accuracy rate is difficult to be ensured. Another type of quantitative method is to set a certain threshold value, usually using some statistical features of the image as an index, and exceeding or falling below the threshold value, the lymph node is considered to be malignant or benign. The limitation of this method is that the setting of the threshold is often very close to the relation of the data set, and there is also a large difference between different subdivided disease categories. In addition to the limitations described above, the diagnostic accuracy of the above methods is often limited.

Through retrieval, chinese patent CN201911310175.1 provides an image recognition method, device and storage medium based on deep learning, the method is: firstly, acquiring an ultrasonic image in real time, and intercepting an image area in the ultrasonic image as an ultrasonic image; and then, automatically identifying the ultrasonic image by adopting a deep convolutional neural network algorithm, and judging whether the ultrasonic image contains the central lymph node. The method can improve the accuracy of the central lymph node image identification. The patent is only used for judging whether the ultrasonic image contains the central lymph node, cannot identify and process benign and malignant lymph nodes, and has little significance for assisting diagnosis of doctors.

Disclosure of Invention

Aiming at the problems, the invention provides a method and a system for processing an endoscope ultrasonic image, which can more accurately identify the lymph node of the ultrasonic image, thereby being beneficial to improving the accuracy of lymph node diagnosis and being suitable for CP-EBUS imaging technology.

In a first aspect of the present invention, there is provided a method for processing an endoscopic ultrasound image, including:

acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;

and classifying the ultrasonic images in the three modes by adopting a multi-modal neural network to obtain a processing result for identifying the lymph nodes.

Optionally, the classifying the ultrasound image in three modes by using a multi-modal neural network includes:

extracting the image characteristics of the ultrasonic image of each mode;

performing feature fusion on the image features of each mode to obtain image fusion features of the three modes;

and classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.

Optionally, the extracting the image feature of the ultrasound image of each mode includes:

inputting the ultrasonic image of each mode into a neural network;

the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.

Optionally, the performing feature fusion on the image features of each mode includes:

and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.

Optionally, the method further comprises an image preprocessing performed before the extracting of the image features of the ultrasound image of each mode, so that the ultrasound image of each modality is suitable as an input of a neural network after being preprocessed.

Optionally, the image preprocessing comprises:

and automatically positioning the scanning frames in the elastic E mode image and the Doppler blood flow F mode image, and selecting a key analysis area by using a minimum rectangular frame according to the positioning.

Optionally, the image preprocessing further comprises:

and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.

In a second aspect of the present invention, there is provided an endoscopic ultrasound image processing system including:

the image acquisition module is used for acquiring ultrasonic images of three modes in the CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;

and the image processing module is used for classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes.

Optionally, the image processing module includes:

the characteristic extraction module is used for extracting the image characteristics of the ultrasonic image of each mode;

the characteristic fusion module is used for carrying out characteristic fusion on the image characteristics of each mode to obtain the image fusion characteristics of the three modes;

and the classifier is used for classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.

In a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the endoscopic ultrasound image processing method.

In a fourth aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program characterized in that the program is executed by a processor for executing the processing method of endoscopic ultrasound images.

Compared with the prior art, the embodiment of the invention has at least one of the following advantages:

according to the processing method and system for the endoscope ultrasonic image, the image data of three modes in the CP-EBUS imaging technology are used, the images are processed through the multi-modal neural network, the lymph node of the ultrasonic image can be identified more accurately, and therefore the accuracy of lymph node diagnosis is improved.

The processing method and the system of the endoscope ultrasonic image can be used for identifying the benign and malignant lymph nodes, can process multi-mode ultrasonic images, namely three modes of gray scale, blood flow and elasticity, and integrates different mode information to assist in improving the diagnosis rate.

Compared with the prior art, the processing method and the processing system for the endoscope ultrasonic image are easy to popularize, can be quickly and efficiently arranged in a place with CP-EBUS equipment, and do not need manual intervention.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:

FIG. 1 is a flowchart of a method for processing endoscopic ultrasound images according to an embodiment of the present invention;

FIG. 2 is a flow chart of the processing of a multi-modal neural network in a preferred embodiment of the present invention;

FIG. 3 is a flowchart of a process for extracting image features in a preferred embodiment of the present invention;

FIG. 4 is a flowchart of the process of image quality identification in a preferred embodiment of the present invention;

FIG. 5 is a block diagram of a system for processing endoscopic ultrasound images in accordance with an embodiment of the present invention;

FIG. 6 is a block diagram of a system for processing endoscopic ultrasound images in accordance with a preferred embodiment of the present invention.

Detailed Description

The present invention will be described in detail below with reference to specific embodiments and the accompanying drawings. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.

Fig. 1 is a flowchart of a processing method of endoscopic ultrasound images according to an embodiment of the present invention. Referring to fig. 1, the method for processing an endoscopic ultrasound image in the present embodiment includes:

s100, acquiring ultrasonic images of three modes in a CP-EBUS imaging technology;

s200, classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for recognizing the lymph nodes.

In the above CP-EBUS imaging technique, the three modes of the ultrasound image are an elastic mode E, a gray-scale mode B, and a doppler blood flow mode F. By adopting the multi-mode image based on the convex array probe ultrasonic bronchoscope (CP-EBUS), the CP-EBUS image of the target lymph node is processed by using the multi-mode neural network, manual intervention is not needed, the lymph node of the ultrasonic image can be more accurately and automatically identified, and the accuracy of lymph node diagnosis is improved.

FIG. 2 is a flow chart of the processing of the multi-modal neural network in a preferred embodiment of the present invention. Referring to fig. 2, in the preferred embodiment, the classifying the three-mode ultrasound images by using the multi-mode neural network specifically includes:

s201, extracting image characteristics of the ultrasonic image of each mode;

s202, performing feature fusion on the image features of each mode to obtain image fusion features of three modes;

and S203, classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.

According to the preferred embodiment, the image features of the ultrasonic images in the three modes are extracted and fused and are used for classification, so that the final lymph node identification of the ultrasonic images, including the identification of the malignancy and the benign of the lymph node, can be improved, the diagnosis accuracy can be improved according to the processing result of the CP-EBUS images, and a doctor can be assisted in puncturing lymph nodes.

FIG. 3 is a flowchart of a preferred embodiment of the present invention for extracting image features. In order to better extract the image features of the ultrasound image of each mode, as shown in fig. 3, in a preferred embodiment, the method may include:

s2011, inputting the ultrasonic image of each mode into a neural network; the neural network adopts a convolution neural network;

and S2012, the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.

Preferably, in a specific embodiment, the image feature of the ultrasound image of each mode is extracted according to the following detailed operations:

and (3) convolution operation: dividing the input image signal x of the ultrasonic image of each mode into three branches, wherein one branch is subjected to 1 × 1 dimensionality reduction convolution, the dimensionality reduction convolution results are respectively subjected to 1 × 1 dimensionality convolution and 3 × 3 dimensionality convolution, and the output results of the two dimensionality reduction convolution are respectively added with the other two branches of the input signal and then connected together. Expressed mathematically as:wherein c is₁,c₂,c₃Respectively 1 × 1 dimensionality reduction convolution, 1 × 1 dimensionality lifting convolution and 3 × 3 dimensionality lifting convolution, a join operation in the channel dimension.

An attention mechanism is as follows: the features extracted by the convolution operation are weighted by channel. Specifically, the input image signal x is divided into two branches, one branch of the input image signal x is subjected to adaptive average pooling in spatial dimension, then the adaptive average pooling is performed through a 1 × 1 dimensionality reduction convolution operation, then the linear rectification function (ReLU) r is performed, then the 1 × 1 dimensionality increase convolution operation is performed, then the weighting of each channel is obtained through a softmax operation sigma, then the weighting value is dot-multiplied with the other branch of the input image signal x according to the channel, and a dot-multiplied result is output. Expressed mathematically as: x is the number of_out＝σ(r(p(x)*c₁)*c₂) X, whereinWhere p is a spatial adaptive pooling function,"·" is a feature map weighting operation along the channel dimension.

In the above preferred embodiment, in order to implement feature fusion on the image features of each mode, it may be: and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.

The feature fusion is realized by adopting a plurality of feature fusion modules, and the fusion operation of each feature fusion module is divided into four paths. Feature extraction output for three mode images The received original signals are directly output by three paths of the signals respectively. And the fourth path has slightly different outputs according to the relative position of the feature fusion module.

For the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths, and the mathematical form of the fourth path is as follows:wherein w₁,w₂,w₃Are the weight parameters that can be learned and,a join operation in the channel dimension.

For fusing subsequent feature fusion modules, e.g. the l-th feature fusion module (l)>1) Firstly, the three features are weighted and connected, then the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion, and the comprehensive mathematical form is as follows:wherein w₄,w₅As are learnable parameters. In the preferred embodiment, the four-way fusion features are output for subsequent classification based on the multi-modal images.

In addition, in order to ensure that the image input by the neural network meets the requirement, the image can be preprocessed before being input. That is, the image feature of the ultrasound image of each mode is extracted before, and the ultrasound image of each mode is adapted as an input of the neural network by preprocessing.

FIG. 4 is a flow chart of the process of image quality identification in a preferred embodiment of the present invention. Referring to fig. 4, the preprocessing may specifically include scan frame positioning, where the scan frame positioning is to automatically position the scan frames in the elastic E-mode and doppler flow F-mode images, and select the key analysis area using the minimum rectangular frame according to the positioning. In order to better reduce the data processing work of the subsequent neural network, on the basis of the preprocessing, the method can further comprise the following steps of identifying the image quality: and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.

For the CP-EBUS image (elastic mode E, doppler blood flow mode F) in the above embodiment, the specific operations of positioning the preprocessed scan frames may be: giving an image of elastic E-mode or Doppler flow F-modeAnd obtaining a mask matrix m set according to the corresponding modal imaging ruleBy removing extraneous information in this step, a preprocessed intermediate signal is obtainedThen, the minimum rectangle containing the detection frame is determined through a frame detection operator G,whereinIs the original signal contained by the smallest rectangle. Of course, the operation of the preferred embodiment is not limited thereto, and other techniques may be used to achieve the above-described effects.

For the CP-EBUS image (elastic mode E, gray scale mode B, doppler blood flow mode F) in the above embodiment, the specific operations of the pre-processing image quality identification may be:

(1) for the image of the elastic E mode, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold value T_eThe image quality is considered to be unqualified, otherwise, the average intensity of the image is calculated, and if the intensity is larger than the threshold value T_IIf the image quality is qualified, otherwise, the image quality is unqualified.

(2) For the Doppler blood flow F-mode image, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold valueT_fThe image quality is considered to be qualified, otherwise, the image quality is not qualified.

(3) For the grayscale B mode image, the quality is qualified by default.

In the above embodiments, the classifier performs global pooling on the features of the four branches output by the feature fusion module, and then multiplies the feature by the classification matrix to give a final classification result.

FIG. 5 is a block diagram of a system for processing endoscopic ultrasound images in accordance with an embodiment of the present invention.

Referring to fig. 5, the system for processing an endoscopic ultrasound image in the present embodiment includes: the device comprises an image acquisition module and an image processing module, wherein the image acquisition module acquires ultrasonic images in three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F; the image processing module adopts a multi-modal neural network to classify the ultrasonic images in the three modes to obtain an image processing result for identifying the lymph nodes.

In the above embodiment, the image processing module may further include: the system comprises a feature extraction module, a feature fusion module and a classifier, wherein the feature extraction module extracts image features of the ultrasonic image of each mode; the feature fusion module performs feature fusion on the image features of each mode to obtain image fusion features of the three modes; and classifying the image fusion characteristics by a classifier to obtain a processing result for identifying the lymph nodes.

FIG. 6 is a block diagram of a system for processing endoscopic ultrasound images in accordance with a preferred embodiment of the present invention.

Referring to fig. 6, in the preferred embodiment, there is provided an endoscopic ultrasound image processing system including:

the data preprocessing module is used for respectively preprocessing the data according to the characteristics of each modal data so as to enable the data to be suitable for being used as the input of a neural network;

a feature extraction module which performs feature extraction on an input image signal by using convolution operation, attention mechanism, cross-layer connection operation and the like, and applies the extracted features to a subsequent feature fusion module;

the characteristic fusion module fuses the characteristics of different modes by utilizing learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image;

and a classifier for classifying the features output by the feature fusion module and outputting a result of the predicted diagnosis.

Specifically, the data preprocessing module is composed of a scanning frame positioning module and an image quality judging module, wherein:

a scanning frame positioning module, which automatically positions the scanning frames in the elastic E mode image and the Doppler blood flow F mode image and selects a key analysis area by using a minimum rectangular frame according to the positioning;

and the image quality judging module evaluates the quality of the image according to the characteristics of the mode, and sends out a prompt aiming at the image with unqualified quality so as to avoid useless analysis.

Furthermore, in the scan frame positioning module, an image of an elastic E mode or a Doppler blood flow F mode is givenAnd obtaining a mask matrix m set according to the corresponding modal imaging rule By removing extraneous information in this step, a preprocessed intermediate signal is obtainedThen, the minimum rectangle containing the detection frame is determined through a frame detection operator G,whereinIs the smallest rectangular bagIncluding the original signal.

Further, in the image quality identification module, for the image of the elastic E mode, the variance of three channel values at each position is firstly calculated, and if the variance at a certain position is larger than the threshold value T_cJudging that the pixel point is a colored pixel point, and if the proportion of the colored pixel point is less than the threshold value T_eThe image quality is considered to be unqualified, otherwise, the average intensity of the image is calculated, and if the intensity is larger than the threshold value T_IIf the image quality is qualified, otherwise, the image quality is unqualified. For Doppler blood flow F-mode images, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold value T_fThe image quality is considered to be qualified, otherwise, the image quality is not qualified. For the grayscale B mode image, the quality is qualified by default.

Further, the feature extraction module comprises: a convolution module and a channel attention module.

The convolution module is used for extracting the characteristics of the image signal. Specifically, in the module: the input signal x is divided into three branches, one branch is input into a 1 x 1 dimensionality reduction convolution module, the output result of the module is respectively input into a 1 x 1 dimensionality increasing convolution module and a 3 x 3 dimensionality increasing convolution module, and the output results of the two dimensionality increasing convolution modules are respectively added to the other two branches of the input signal and then connected together. Expressed mathematically as: wherein c is₁,c₂,c₃Respectively 1 × 1 dimensionality reduction convolution, 1 × 1 dimensionality lifting convolution and 3 × 3 dimensionality lifting convolution, for connections in channel dimensionsAnd (6) performing operation.

The channel attention model is used to weight the features extracted by the convolution module by channel. Specifically, in the module, an input signal x is divided into two branches, one branch of the input signal x is subjected to adaptive average pooling in a spatial dimension, then a 1 × 1 dimensionality reduction convolution operation is carried out, then a linear rectification function (ReLU) r is carried out, then a 1 × 1 dimensionality increase convolution operation is carried out, then a weight value for each channel is obtained through a softmax operation sigma, then the weight value is subjected to dot multiplication with the other branch of the input signal according to the channels, and a dot multiplication result is output. Expressed mathematically as: x is the number of_out＝σ(r(p(x)*c₁)*c₂) X, whereinWhere p is a spatial adaptive pooling function,"·" is a feature map weighting operation along the channel dimension.

Further, there are a plurality of feature fusion modules. The fusion operation of each module is divided into four ways, and the output of the feature extraction module for three mode imagesThe received original signals are directly output by three paths of the signals respectively. And the fourth path has slightly different outputs according to the relative position of the feature fusion module.

For the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths, and the mathematical form of the fourth path is as follows:wherein w₁,w₂,w₃Are the weight parameters that can be learned and,a join operation in the channel dimension.

For subsequent fusion operations of the feature fusion module, e.g. lFeature fusion module (l)>1) Firstly, the three features are weighted and connected, then the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion, and the comprehensive mathematical form is as follows:wherein w₄,w₅As are learnable parameters.

The number of the feature extraction modules and the number of the feature fusion modules can be adjusted by specific scenes.

Further, the classifier respectively performs global pooling on the four paths of features output by the feature fusion module and then multiplies the four paths of features by a classification matrix to give a final classification result.

The processing system of the endoscope ultrasonic image in the above embodiment of the present invention uses the image data of the three modes in the CP-EBUS imaging technology, and processes the images through the multi-modal neural network, so that the lymph node of the ultrasonic image can be identified more accurately, which is helpful to improve the accuracy of lymph node diagnosis.

In another embodiment of the present invention, there is also provided a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for processing an endoscopic ultrasound image in any one of the above embodiments.

In another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program for executing the processing method of endoscopic ultrasound images in any one of the above embodiments when the program is executed by a processor.

To verify the effectiveness of the above-described embodiments of the present invention, experiments were performed on authentic CP-EBUS datasets and performance comparisons were performed with existing methods used by human experts. In the experiment, the model was cross-validated five-fold on 2205 images collected from 245 lymph nodes, and the model selected from the five validations was tested on 441 images collected from 49 additional lymph nodes, and the results were recorded as the mean and confidence interval of the five model tests. And comparing the expert groups on the same test set, testing the test sets by the three experts respectively, and recording the average value and the confidence interval of the test results of the three experts as results.

The results of the experiment are shown in table 1:

as can be seen from table 1, the lymph node identification result obtained by the above embodiment of the present invention has significantly improved accuracy compared to the result of the existing method implemented by the present human experts, and the superiority and effectiveness of the present invention are well verified. In addition, the consistency analysis was performed on the lymph node identification result of the embodiment of the present invention and the expert group result, the kappa value of the result of the embodiment of the present invention was 0.7605, and the kappa value of the human expert group as a comparison was 0.5800, which shows that the lymph node identification result of the embodiment of the present invention has better diagnostic consistency.

It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may refer to the technical solution of the system to implement the step flow of the method, that is, the embodiment in the system may be understood as a preferred example for implementing the method, and details are not described herein.

Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer readable program code means, the method steps can be fully programmed to implement the same functions by implementing the system and its various devices in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices thereof provided by the present invention can be regarded as a hardware component, and the devices included in the system and various devices thereof for realizing various functions can also be regarded as structures in the hardware component; means for performing the functions may also be regarded as structures within both software modules and hardware components for performing the methods.

The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The above-described preferred features may be used in any combination without conflict with each other.