阅读说明：本技术 基于深度学习的多模态医学图像融合方法及系统 (Multi-modal medical image fusion method and system based on deep learning ) 是由 刘星宇 张逸凌 于 2021-06-07 设计创作，主要内容包括：本发明提供一种基于深度学习的多模态医学图像融合方法及系统,该方法包括：获取患者的至少两种模态的二维医学图像；将所述至少两种模态的二维医学图像分别输入至预先训练的相应的图像分割网络模型,以分别获得各个模态本体位置区域的二维医学图像的输出；基于点云配准算法,将所述各个模态本体位置区域的二维医学图像进行点云配准融合,以获得多模态相融合二维医学图像；对所述多模态相融合二维医学图像进行三维重建处理,以获得多模态相融合三维医学图像。本发明的多模态医学图像配准精度高,适用于多种复杂的图像融合情况,还可以提高术者的手术准确性以及提高手术效率。(The invention provides a method and a system for multi-modal medical image fusion based on deep learning, wherein the method comprises the following steps: acquiring two-dimensional medical images of at least two modalities of a patient; respectively inputting the two-dimensional medical images of the at least two modalities into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the modality bodies; performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modality bodies based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image; and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image. The multi-modal medical image registration precision of the invention is high, is suitable for various complicated image fusion conditions, and can also improve the operation accuracy of an operator and improve the operation efficiency.)

1. A multi-modal medical image fusion method based on deep learning is characterized by comprising the following steps:

acquiring two-dimensional medical images of at least two modalities of a patient;

respectively inputting the two-dimensional medical images of the at least two modalities into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the modality bodies;

performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modality bodies based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image.

2. The deep learning-based multi-modal medical image fusion method according to claim 1, wherein point cloud registration fusion is performed on the two-dimensional medical images of the respective modal ontology position regions based on a point cloud registration algorithm to obtain a multi-modal fused two-dimensional medical image, and the method comprises:

respectively determining a body mark point set and a body head mark point set of the two-dimensional medical image of each modality based on the two-dimensional medical image of each modality body position area, wherein the body mark point set and the body head mark point set are respectively used as point cloud sets corresponding to the two-dimensional medical images of each modality;

and performing point cloud registration fusion on the point cloud sets corresponding to the two-dimensional medical images of all the modalities based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image.

3. The deep learning based multi-modal medical image fusion method according to claim 2, wherein the two-dimensional medical images of the at least two modalities include at least two of a two-dimensional CT medical image, a two-dimensional MRI medical image, a two-dimensional ultrasound medical image, a two-dimensional PETCT medical image, the body comprises a femur, the body head comprises a femoral head;

respectively determining a body mark point set and a body head mark point set of the two-dimensional medical image based on the two-dimensional medical image of each modal body position area, wherein the body mark point set and the body head mark point set are respectively used as point cloud sets corresponding to the two-dimensional medical images of each modality, and the method comprises the following steps:

determining a femur central point set and a femoral head central point set of the femur position area as a first point cloud set corresponding to a CT mode based on a two-dimensional CT medical image of the femur position area; determining a femur central point set and a femoral head central point set of the femur position area as second point cloud sets corresponding to an MRI modality based on the two-dimensional MRI medical image of the femur position area;

based on a point cloud registration algorithm, performing point cloud registration and fusion on point cloud sets corresponding to two-dimensional medical images of various modalities to obtain a multi-modality fused two-dimensional medical image, wherein the method comprises the following steps:

and performing point cloud registration fusion on the first point cloud set and the second point cloud set based on an ICP point cloud registration algorithm to obtain a two-dimensional medical image fused by a CT mode and an MRI mode.

4. The deep learning based multi-modal medical image fusion method according to any one of claims 1 to 3, wherein the three-dimensional reconstruction processing is performed on the multi-modal fused two-dimensional medical image to obtain a multi-modal fused three-dimensional medical image, and comprises:

and inputting the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network so as to obtain the multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed.

5. The deep learning based multimodal medical image fusion method according to claim 4, wherein the three-dimensional reconstruction network comprises a coding network, a conversion network and a decoding network; inputting a multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network to obtain a multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed, comprising:

extracting two-dimensional image features of the multi-modal fused two-dimensional medical image based on the coding network;

converting two-dimensional image features of the multi-modal fused two-dimensional medical image into three-dimensional image features based on the conversion network;

and converting the three-dimensional image features into multi-modal fused three-dimensional medical images corresponding to the multi-modal fused two-dimensional medical images based on the decoding network.

6. The deep learning based multi-modal medical image fusion method according to claim 3, wherein the training process of the image segmentation network model comprises:

acquiring a two-dimensional medical image dataset of a plurality of patients, wherein the two-dimensional medical image dataset comprises a plurality of two-dimensional medical images;

marking the femoral position area in each two-dimensional medical image by adopting at least one of automatic marking and manual marking;

dividing each two-dimensional medical image after labeling into a training data set and a testing data set according to a preset proportion;

and training the image segmentation network model based on the training data set and by combining a neural network algorithm and deep learning.

7. The deep learning based multi-modal medical image fusion method according to claim 6, wherein training out the image segmentation network model based on the training data set and combining neural network algorithm and deep learning comprises:

segmenting the training data set by a first image segmentation model: performing a plurality of downsampling on the image data in the training dataset to identify deep features of each image data through processing of a convolutional layer and a pooling layer; performing up-sampling on the down-sampled image data for multiple times to reversely store the deep features into the image data through processing of an up-sampling layer and a convolution layer to obtain an image rough segmentation result;

based on the image rough segmentation result, screening feature point data with preset confidence coefficient from the deep features through a second image segmentation model, carrying out bilinear interpolation calculation on the feature point data, and identifying the category of the deep features based on the calculated feature point data to obtain a final image segmentation result;

calculating a loss function based on the final image segmentation result and the training data set;

and adjusting parameters of the image segmentation network model based on the loss function until the image segmentation network model is successfully trained.

8. The deep learning based multimodal medical image fusion method of claim 7, wherein the method further comprises:

setting an activation function after each convolution layer;

and/or after the last upsampling is finished, discarding part of neural network units in the image segmentation network model according to a preset probability through a set dropout layer.

9. A system for multi-modal medical image fusion based on deep learning, comprising:

a multi-modality image acquisition unit configured to acquire two-dimensional medical images of at least two modalities of a patient;

a two-dimensional image output unit configured to input the two-dimensional medical images of the at least two modalities to a pre-trained image segmentation network model to obtain outputs of the two-dimensional medical images of the respective modality body position regions, respectively;

a two-dimensional image fusion unit configured to perform point cloud registration fusion on the two-dimensional medical images of the respective modality body position regions based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

a three-dimensional reconstruction unit configured to perform three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain a multi-modal fused three-dimensional medical image.

10. A computer-readable storage medium storing computer instructions for causing the computer to perform all or part of the steps of the deep learning based multi-modal medical image fusion method according to any one of claims 1 to 8.

Technical Field

The invention relates to the technical field of medical image processing, in particular to a method and a system for multi-modal medical image fusion based on deep learning.

Background

In modern digital medical diagnosis, medical staff usually need to analyze the lesion of a patient by using the acquired multi-modal three-dimensional image of the patient before performing a surgery, so as to make an appropriate surgical plan. Because the highlighted image features of each image are different, in order to facilitate the observation and planning of the operation by the doctor, the advantages of the images of multiple modalities acquired before the operation need to be integrated, that is, multi-modality image registration needs to be performed, so that the images of different modalities are registered to the same angle, and the image features of the lesion part of the patient, which can be provided by each image, are fused into one image to be displayed.

In the implementation of the embodiments of the present disclosure, it is found that at least the following problems exist in the related art:

the requirements on the initial alignment condition and the similarity of each image are high, so that the complexity of image registration and fusion is high, the image registration and fusion accuracy is low, the time cost is high, and the method cannot be effectively applied to non-rigid registration.

Disclosure of Invention

The invention provides a multi-modal medical image fusion method and system based on deep learning, which are used for overcoming the defects that in the prior art, multi-modal images are low in registration fusion precision, high in complexity and high in time cost, cannot be effectively applied to the non-rigid registration condition and the like, improving the multi-modal image fusion precision, reducing the time cost, being suitable for various complex image fusion conditions, improving the operation accuracy and the operation efficiency of an operator, and being effectively applied to the non-rigid registration condition.

The invention provides a multi-modal medical image fusion method based on deep learning, which comprises the following steps:

acquiring two-dimensional medical images of at least two modalities of a patient;

respectively inputting the two-dimensional medical images of the at least two modalities into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the modality bodies;

performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modality bodies based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image.

In some embodiments, performing point cloud registration fusion on the two-dimensional medical images of the respective modality body position regions based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image, including:

respectively determining a body mark point set and a body head mark point set of the two-dimensional medical image of each modality based on the two-dimensional medical image of each modality body position area, wherein the body mark point set and the body head mark point set are respectively used as point cloud sets corresponding to the two-dimensional medical images of each modality;

and performing point cloud registration fusion on the point cloud sets corresponding to the two-dimensional medical images of all the modalities based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image.

In some embodiments, the two-dimensional medical images of the at least two modalities include at least two of a two-dimensional CT medical image, a two-dimensional MRI medical image, a two-dimensional ultrasound medical image, a two-dimensional PETCT medical image, the body comprises a femur, the body head comprises a femoral head;

respectively determining a body mark point set and a body head mark point set of the two-dimensional medical image based on the two-dimensional medical image of each modal body position area, wherein the body mark point set and the body head mark point set are respectively used as point cloud sets corresponding to the two-dimensional medical images of each modality, and the method comprises the following steps:

determining a femur central point set and a femoral head central point set of the femur position area as a first point cloud set corresponding to a CT mode based on a two-dimensional CT medical image of the femur position area; determining a femur central point set and a femoral head central point set of the femur position area as second point cloud sets corresponding to an MRI modality based on the two-dimensional MRI medical image of the femur position area;

based on a point cloud registration algorithm, performing point cloud registration and fusion on point cloud sets corresponding to two-dimensional medical images of various modalities to obtain a multi-modality fused two-dimensional medical image, wherein the method comprises the following steps:

and performing point cloud registration fusion on the first point cloud set and the second point cloud set based on an ICP point cloud registration algorithm to obtain a two-dimensional medical image fused by a CT mode and an MRI mode.

In some embodiments, the three-dimensional reconstruction processing of the multi-modal fused two-dimensional medical image to obtain a multi-modal fused three-dimensional medical image comprises:

and inputting the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network so as to obtain the multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed.

In some embodiments, the three-dimensional reconstruction network comprises an encoding network, a translation network, and a decoding network; inputting a multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network to obtain a multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed, comprising:

extracting two-dimensional image features of the multi-modal fused two-dimensional medical image based on the coding network;

converting two-dimensional image features of the multi-modal fused two-dimensional medical image into three-dimensional image features based on the conversion network;

and converting the three-dimensional image features into multi-modal fused three-dimensional medical images corresponding to the multi-modal fused two-dimensional medical images based on the decoding network.

In some embodiments, the training process of the image segmentation network model includes:

acquiring a two-dimensional medical image dataset of a plurality of patients, wherein the two-dimensional medical image dataset comprises a plurality of two-dimensional medical images;

marking the femoral position area in each two-dimensional medical image by adopting at least one of automatic marking and manual marking;

dividing each two-dimensional medical image after labeling into a training data set and a testing data set according to a preset proportion;

and training the image segmentation network model based on the training data set and by combining a neural network algorithm and deep learning.

In some embodiments, training the image segmentation network model based on the training dataset in combination with neural network algorithms and deep learning comprises:

segmenting the training data set by a first image segmentation model: performing a plurality of downsampling on the image data in the training dataset to identify deep features of each image data through processing of a convolutional layer and a pooling layer; performing up-sampling on the down-sampled image data for multiple times to reversely store the deep features into the image data through processing of an up-sampling layer and a convolution layer to obtain an image rough segmentation result;

based on the image rough segmentation result, screening feature point data with preset confidence coefficient from the deep features through a second image segmentation model, carrying out bilinear interpolation calculation on the feature point data, and identifying the category of the deep features based on the calculated feature point data to obtain a final image segmentation result;

calculating a loss function based on the final image segmentation result and the training data set;

and adjusting parameters of the image segmentation network model based on the loss function until the image segmentation network model is successfully trained.

In some embodiments, the method further comprises:

setting an activation function after each convolution layer;

and/or after the last upsampling is finished, discarding part of neural network units in the image segmentation network model according to a preset probability through a set dropout layer.

The invention also provides a multi-modal medical image fusion system based on deep learning, which comprises the following components:

a multi-modality image acquisition unit configured to acquire two-dimensional medical images of at least two modalities of a patient;

a two-dimensional image output unit configured to input the two-dimensional medical images of the at least two modalities to a pre-trained image segmentation network model to obtain outputs of the two-dimensional medical images of the respective modality body position regions, respectively;

a two-dimensional image fusion unit configured to perform point cloud registration fusion on the two-dimensional medical images of the respective modality body position regions based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

a three-dimensional reconstruction unit configured to perform three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain a multi-modal fused three-dimensional medical image.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements all or part of the steps of the deep learning based multi-modal medical image fusion method according to any one of the above items when executing the computer program.

The present invention also provides a computer readable storage medium having stored thereon computer instructions for causing the computer to execute all or part of the steps of the depth learning based multimodal medical image fusion method according to any one of the above.

The invention provides a multi-modal medical image fusion method and system based on deep learning, wherein the method comprises the steps of respectively carrying out image segmentation on two-dimensional medical images of the same part of the same patient and different modalities, carrying out accurate point cloud registration fusion on the two-dimensional medical images of position areas of each modality body after completing image segmentation to obtain a multi-modal fused two-dimensional medical image, and then three-dimensionally reconstructing the multi-modal fused two-dimensional medical image into a multi-modal fused three-dimensional medical image; the method ensures high multi-modal medical image registration precision, can reduce time cost, is suitable for various complex image fusion conditions, and can also improve the operation accuracy of an operator and improve the operation efficiency.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic flow chart of a deep learning-based multi-modal medical image fusion method provided by the present invention;

FIG. 2A is a two-dimensional CT medical image of a femoral location area provided by an embodiment of the present invention;

FIG. 2B is a two-dimensional MRI medical image of a femoral location area provided by an embodiment of the present invention;

FIG. 2C is a two-dimensional medical image of a fused CT modality and MRI modality of a femoral location region provided by an embodiment of the present invention;

FIG. 3 is a flow chart diagram of a deep learning-based multi-modal medical image fusion method provided by the present invention;

FIG. 4 is a fused three-dimensional medical image of a CT modality of a femoral position region and an MRI modality of a femoral position region provided by the present invention;

FIG. 5 is a three-dimensional medical image fused by a CT modality of a femoral position region and an MRI modality of a femoral necrosis position region provided by the present invention;

FIG. 6 is a three-dimensional CT medical image of a femur position region after image segmentation and three-dimensional reconstruction by a multi-modal medical image fusion method based on deep learning provided by the present invention;

FIG. 7 is a three-dimensional MRI medical image of a femur position region after image segmentation and three-dimensional reconstruction by a multi-modal medical image fusion method based on deep learning provided by the present invention;

FIG. 8 is a three-dimensional MRI medical image of a femoral necrosis location region after image segmentation and three-dimensional reconstruction by the deep learning based multi-modal medical image fusion method provided by the present invention;

FIG. 9 is a schematic flow chart illustrating a pre-training process of a CT image segmentation network model in the method provided by the present invention;

FIG. 10 is a flow chart illustrating a pre-training process of an MRI image segmentation network model in the method provided by the present invention;

FIG. 11 is a block diagram of a deep learning training network for the training process shown in FIGS. 9 and 10;

FIG. 12 is a schematic structural diagram of a deep learning-based multi-modal medical image fusion system provided by the present invention;

fig. 13 is a schematic structural diagram of an electronic device provided by the present invention.

Reference numerals:

1010: a multimodal image acquisition unit; 1020: a two-dimensional image output unit; 1030: a two-dimensional image fusion unit; 1040: a three-dimensional reconstruction unit;

1310: a processor; 1320: a communication interface; 1330: a memory; 1340: a communication bus.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The spatial resolution of the CT medical image is high, the rigid skeleton can be clearly positioned, but the imaging contrast of soft tissues is low, and the focus can not be clearly displayed; MRI medical images provide high contrast imaging of anatomical structures such as soft tissues, blood vessels, organs, etc., but have lower spatial resolution than CT medical images and lack rigid bone structures as a reference for locating lesions. Therefore, in clinical applications, medical images of a single modality often fail to provide comprehensive medical reference information to the relevant medical personnel.

The embodiment of the invention combines the artificial intelligent image segmentation algorithm with the multi-modal medical image fusion technology, integrates the advantages of various medical imaging technologies, extracts the complementary information of medical images of different modalities, and generates a synthetic image which contains more effective medical reference information than any single-modality image after fusion so as to help related medical care personnel to diagnose, stage and treat various diseases such as femoral head necrosis and the like.

The following describes a method, a system, a computer storage medium and an electronic device for multi-modal medical image fusion based on deep learning according to the present invention with reference to fig. 1 to 13. The invention provides a multimodal medical image fusion method based on deep learning, fig. 1 is one of the flow diagrams of the multimodal medical image fusion method based on deep learning provided by the invention, as shown in fig. 1, the method comprises the following steps:

s110, acquiring two-dimensional medical images of at least two modalities of the patient.

Two-dimensional medical images of two or more modalities of the same part of the body of the same patient can be acquired, for example, for a patient suffering from hip joint diseases, two-dimensional CT medical images, two-dimensional MRI medical images, two-dimensional ultrasound medical images, two-dimensional PETCT medical images and other two-dimensional medical images of the femoral part of the hip joint of the patient can be acquired.

And S120, respectively inputting the two-dimensional medical images of at least two modes into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the mode bodies.

Inputting the two-dimensional medical images of the plurality of modalities acquired in step S110 into the pre-trained corresponding image segmentation network model one by one, so as to obtain output of the two-dimensional medical image of each modality body position region, for example, inputting the two-dimensional CT medical image of the patient into the corresponding CT image segmentation network model for the CT image, so as to obtain the CT medical image of the femur position region; or inputting the two-dimensional MRI medical image of the patient into the corresponding MRI image segmentation network model aiming at the MRI image to obtain the MRI medical image of the femur position area; or inputting the two-dimensional ultrasonic medical image of the patient into the corresponding ultrasonic medical image segmentation network model aiming at the ultrasonic medical image so as to obtain the ultrasonic medical image of the femur position area; alternatively, the two-dimensional PETCT medical image of the patient is input into its corresponding PETCT medical image segmentation network model for the PETCT medical image to obtain the PETCT medical image of the femoral position region. Optionally, other two-dimensional medical images of the same body part of the patient may also be input into their respective corresponding image segmentation network models for processing, which is not limited in the embodiment of the present invention.

If the body part of the patient does not have any symptoms, the two-dimensional medical images of the modalities are normal images, and images about lesions or necrosis do not appear. If the body part of the patient has a certain lesion or necrosis, the two-dimensional medical image of at least one of the modality body position regions is a two-dimensional medical image capable of indicating the body necrosis position region of the patient in the modality. For example, at least one of the two-dimensional medical image of the body position region in the CT modality and the two-dimensional medical image of the body position region in the MRI modality is output separately, for example, the two-dimensional medical image of the body position region in the MRI modality includes the two-dimensional medical image of the body necrosis position region of the patient in the MRI modality, or the two-dimensional medical image of the body necrosis position region of the patient in the MRI modality may be understood as another independent two-dimensional medical image that is juxtaposed to the two-dimensional medical image of the body position region in the MRI modality, but still regarded as an integral with the two-dimensional medical image of the body position region in the same modality.

S130, performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modality bodies based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image.

Referring to fig. 2A-2C, fig. 2A is a two-dimensional CT medical image of a femoral location area provided by an embodiment of the present invention; FIG. 2B is a two-dimensional MRI medical image of a femoral location area provided by an embodiment of the present invention; fig. 2C is a fused two-dimensional medical image of a CT modality and an MRI modality of a femoral position region provided by an embodiment of the present invention.

In some embodiments, determining, based on the two-dimensional medical image of each modality ontology position region, an ontology landmark point set and an ontology header landmark point set as point cloud sets corresponding to the two-dimensional medical image of each modality, respectively, includes:

determining a femur central point set and a femoral head central point set of the femur position area as a first point cloud set corresponding to a CT mode based on a two-dimensional CT medical image of the femur position area; determining a femur central point set and a femoral head central point set as second point cloud sets corresponding to an MRI modality based on a two-dimensional MRI medical image of a femur position area;

based on a point cloud registration algorithm, performing point cloud registration and fusion on point cloud sets corresponding to two-dimensional medical images of various modalities to obtain a multi-modality fused two-dimensional medical image, wherein the method comprises the following steps:

and performing point cloud registration fusion on the first point cloud set and the second point cloud set based on an ICP point cloud registration algorithm to obtain a two-dimensional medical image fused by a CT mode and an MRI mode.

S140, performing three-dimensional reconstruction processing on the multi-modal fusion two-dimensional medical image to obtain the multi-modal fusion three-dimensional medical image.

In some embodiments, the two-dimensional medical images of the position regions of the modality bodies can be respectively subjected to three-dimensional reconstruction and then point cloud registration fusion to obtain a multi-modality fused three-dimensional medical image. Optionally, the obtained two-dimensional medical images of the position regions of the modality bodies are respectively subjected to three-dimensional reconstruction to obtain three-dimensional medical images of the position regions of the modality bodies, and then the three-dimensional medical images of the position regions of the modality bodies are subjected to point cloud registration fusion to obtain multi-modality fused three-dimensional medical images.

The invention provides a depth learning-based multi-modal medical image fusion method, which comprises the steps of completing image segmentation of two-dimensional medical images of the same part and different modalities of the same patient, then carrying out accurate point cloud registration fusion to obtain a multi-modal fused two-dimensional medical image, and then three-dimensionally reconstructing the multi-modal fused two-dimensional medical image into a multi-modal fused three-dimensional medical image; or the two-dimensional medical images of different modes are respectively subjected to image segmentation processing and then three-dimensional reconstruction, and finally the three-dimensional medical images of different modes after three-dimensional reconstruction are subjected to accurate point cloud registration fusion to obtain the multi-mode fused three-dimensional medical image.

On the basis of the embodiment shown in fig. 1, the step S130 of performing point cloud registration fusion on the two-dimensional medical images of the respective modality body position regions based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image may include:

s1311, determining a body mark point set and a body head mark point set to be respectively used as corresponding point cloud sets of the two-dimensional medical images of the various modes based on the two-dimensional medical images of the body position areas of the various modes;

and respectively determining a body mark point set and a body head mark point set based on the two-dimensional medical image of each modal body position area so as to determine a point cloud set corresponding to each modality. The body mark point and the body head mark point can be set by selecting a reference point according to actual requirements. Of course, the body center point and the body center point can be selected from the body mark points and the body head mark points, so as to determine the body center point set and the body head center point set in each mode. The center point of the body area and the center point of the body head can be better used as reference points, so that the point cloud sets corresponding to all the modes are calculated and determined on the basis of the points.

Optionally, the body may comprise a femur and the body head may comprise a femoral head.

S1312, carrying out point cloud registration fusion on the point cloud sets corresponding to the two-dimensional medical images of all the modalities based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

and (4) performing comprehensive point cloud registration and fusion on the point cloud sets corresponding to the two-dimensional medical images of each modality determined in the step (1311) based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image.

On the basis of the embodiment shown in fig. 1, the step S140 of performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image may include: and inputting the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network so as to obtain the multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed.

In some embodiments, the three-dimensional reconstruction network comprises an encoding network, a translation network, and a decoding network; inputting the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed into a pre-trained three-dimensional reconstruction network to obtain a multi-modal fused three-dimensional medical image corresponding to the multi-modal fused two-dimensional medical image to be three-dimensionally reconstructed, comprising:

extracting two-dimensional image features of the multi-mode fused two-dimensional medical image based on the coding network;

converting two-dimensional image features of the multi-modal fused two-dimensional medical image into three-dimensional image features based on a conversion network;

and converting the three-dimensional image features into multi-modal fused three-dimensional medical images corresponding to the multi-modal fused two-dimensional medical images based on a decoding network.

In some embodiments, the multi-modal fused two-dimensional medical image may be further reconstructed into a multi-modal fused three-dimensional medical image based on a three-dimensional image reconstruction method.

And performing three-dimensional reconstruction on the multi-modal fused two-dimensional medical image based on a three-dimensional image reconstruction method (using a three-dimensional image processing library) to obtain the multi-modal fused three-dimensional medical image. The three-dimensional image reconstruction method can be performed by referring to the existing techniques such as the three-dimensional image open source processing library, and the details are not described herein.

Fig. 3 is a schematic flow diagram of a multi-modal medical image fusion method based on deep learning according to the present invention, and as shown in fig. 3, when performing point cloud registration fusion after performing three-dimensional reconstruction on two-dimensional medical images of each modal body position region, respectively, to obtain a multi-modal fused three-dimensional medical image, the method may include:

s1321, reconstructing the two-dimensional medical image of each modal body position region into a three-dimensional medical image of each modal body position region respectively based on a three-dimensional image reconstruction method;

and respectively carrying out three-dimensional reconstruction on the two-dimensional medical images of the position areas of the various modality bodies based on a three-dimensional image reconstruction method (using a three-dimensional image processing library), and correspondingly obtaining the three-dimensional medical images of the position areas of the various modality bodies. The three-dimensional image reconstruction method can be performed by referring to the existing techniques such as the three-dimensional image open source processing library, and the details are not described herein.

S1322, determining a body mark point set and a body head mark point set of the three-dimensional medical image respectively based on the body position area of each mode as a point cloud set corresponding to the mode;

and determining a point cloud set corresponding to each modality according to the determined body mark point set and the body head mark point set respectively based on the three-dimensional medical image of each modality body position area reconstructed in the step S1321. The body mark point and the body head mark point can be set by selecting a reference point according to actual requirements. Of course, the body center point and the body center point can be selected from the body mark points and the body head mark points, so as to determine the body center point set and the body head center point set in each mode. The center point of the body area and the center point of the body head can be better used as reference points, so that the point cloud sets corresponding to all the modes are calculated and determined on the basis of the points.

And S1323, performing point cloud registration and fusion on the point cloud sets corresponding to the three-dimensional medical images of all the modalities based on a point cloud registration algorithm to obtain multi-modality fused three-dimensional medical images.

And finally, performing comprehensive point cloud registration fusion on the point cloud sets corresponding to the three-dimensional medical images of all the modalities determined in the step S1322 based on a point cloud registration algorithm, and finally obtaining the multi-modality fused three-dimensional medical image.

According to the multi-modal medical image fusion method based on deep learning provided by the invention, the two-dimensional medical images of at least two modalities comprise at least two of two-dimensional CT medical images, two-dimensional MRI medical images, two-dimensional ultrasound medical images and two-dimensional PETCT medical images. Two-dimensional medical images in other modalities may of course be included, as the invention is not limited in this respect.

Alternatively, when the applied patient is a patient of the type suffering from hip joint disease, a two-dimensional medical image of the hip joint part, in particular the femur part, of the patient may be acquired for diagnostic reference by medical personnel. Thus, the present embodiment provides that the body is understood to be a femur and correspondingly the head of the body is the femoral head. Therefore, the two-dimensional medical images of the respective modality body position regions output by the image segmentation network model in step S120 are, for example, two-dimensional medical images of the femur position regions in the CT modality and the MRI modality.

According to the method for fusing multi-modal medical images based on deep learning provided by the present invention, on the basis of the above embodiment, the step S120 of respectively inputting the two-dimensional medical images of at least two modalities to the corresponding image segmentation network model trained in advance to respectively obtain the outputs of the two-dimensional medical images of the location areas of the ontology of each modality further includes:

inputting the two-dimensional CT medical image into a pre-trained CT image segmentation network model to obtain a CT medical image of a femur position area; and/or inputting the two-dimensional MRI medical image into a pre-trained MRI image segmentation network model to obtain an MRI medical image of the femur position area; and/or inputting the two-dimensional ultrasonic medical image into a pre-trained ultrasonic image segmentation network model to obtain an ultrasonic medical image of the femur position area; and/or inputting the two-dimensional PETCT medical image into a pre-trained PETCT image segmentation network model to obtain the PETCT medical image of the femur position area.

When the patient femur position area has necrosis or lesion, the MRI medical image of the femur position area is also set to include the MRI medical image of the femur necrosis position area, and the two-dimensional MRI medical image with femur necrosis can also be set to be acquired separately and input into the MRI image segmentation network model trained in advance to acquire a single MRI medical image of the femur necrosis position area.

The step S120 may further include: and respectively inputting the two-dimensional CT medical image and the two-dimensional MRI medical image into the corresponding image segmentation network models trained in advance, so as to respectively output the CT medical image of the femoral position area and the MRI medical image of the femoral position area. The MRI medical image of the femur position region includes an MRI medical image of the femur necrosis position region, that is, the output MRI medical image of the femur position region includes a representation of the MRI medical image of the femur necrosis position region in the MRI modality. Alternatively, the MRI medical image of the femur necrosis location area may also be understood as another separate two-dimensional medical image that coexists with the MRI medical image of the femur location area under the MRI modality, but is still required to be logically viewed as one whole with the MRI medical image of the femur location area.

When the method of steps S1321-S1323 is executed, the MRI medical image of the femur necrosis location area is combined to include the setting of the MRI medical image of the femur necrosis location area, and the specific process is described as follows:

and S131, reconstructing the two-dimensional medical image of each modality body position region into a three-dimensional medical image of each modality body position region respectively based on a three-dimensional image reconstruction method.

That is, based on the three-dimensional image reconstruction method, it is possible to reconstruct the CT medical image of the femoral position region into the three-dimensional CT medical image of the femoral position region and reconstruct the MRI medical image of the femoral position region (including the femoral necrosis position region MRI medical image) into the three-dimensional MRI medical image of the femoral position region (including the femoral necrosis position region MRI medical image) using the three-dimensional image processing library. The three-dimensional MRI medical image of the femoral necrosis location region may be understood as another independent three-dimensional medical image coexisting with the three-dimensional MRI medical image of the femoral necrosis location region, or may be understood as being included in the three-dimensional MRI medical image of the femoral necrosis location region together with the three-dimensional MRI medical image of the femoral necrosis location region as a whole three-dimensional medical image.

Step S132, respectively based on the three-dimensional medical images of the body position areas of each modality, determining a body mark point set and a body head mark point set thereof as point cloud sets corresponding to the three-dimensional medical images of each modality, specifically determining a body center point set and a body head center point set thereof as point cloud sets corresponding to the three-dimensional medical images of each modality, including:

namely, based on the three-dimensional CT medical image of the femur position area, determining a femur central point set and a femoral head central point set as a first point cloud set corresponding to the three-dimensional CT medical image in the CT mode; determining a femur central point set and a femoral head central point set as second point cloud sets corresponding to the three-dimensional MRI medical image in an MRI modality based on the three-dimensional MRI medical image of the femur position area;

the center point of the femur and the center point of the femoral head can be better used as reference points, so that the point cloud sets corresponding to the three-dimensional medical images of various modes are calculated and determined on the basis of the points.

The process for determining the point cloud set corresponding to each modality three-dimensional medical image specifically comprises the following steps:

based on the three-dimensional CT medical image of the femur position area, a femur central point set and a femoral head central point set are determined to be used as a first point cloud set M corresponding to a CT mode. According to the two-dimensional CT medical image of the femur position region output by the model, the femur region is displayed on a two-dimensional cross section, the femoral head layer is approximately circular, so that the femoral head central point can be directly calculated, and then the medullary cavity central point of each layer is determined on the medullary cavity layer to form the femoral central point. These points can also be derived from a three-dimensional CT medical image of the femoral position region after three-dimensional reconstruction from the two-dimensional image. And the three-dimensional CT medical images of the plurality of femoral position areas obtain a femoral central point set and a femoral head central point set, and then the femoral central point set and the femoral head central point set are combined to form a first point cloud set M.

Similarly, based on the three-dimensional MRI medical image of the femur position region (including the three-dimensional MRI medical image of the femur necrosis position region), the femur central point set and the femoral head central point set are determined as the second point cloud set N corresponding to the MRI modality.

And S133, performing point cloud registration and fusion on the point cloud sets corresponding to the three-dimensional medical images of each modality based on a point cloud registration algorithm to obtain a multi-modality fused medical image.

Namely, based on an ICP point cloud registration algorithm, point cloud registration fusion is carried out on two groups of point clouds, namely a first point cloud set M and a second point cloud set N, so that a three-dimensional medical image fused with a CT mode and an MRI mode is obtained, the registration accuracy is higher, and the registration time cost is low.

The ICP point cloud registration algorithm can adopt a three-dimensional point cloud registration method: calculating a first reference coordinate system corresponding to the cloud set of the points to be registered and a second reference coordinate system corresponding to the cloud set of the reference points based on a principal component analysis method; based on a first reference coordinate system and a second reference coordinate system, carrying out initial registration on a point cloud set to be registered and a reference point cloud set; based on a multi-dimensional binary search tree algorithm, searching a point closest to the cloud set of points to be registered in the reference point cloud set after initial registration to obtain a plurality of groups of corresponding point pairs; respectively calculating the direction vector included angles among the corresponding point pairs; and performing fine registration on the point cloud set to be registered and the reference point cloud set based on a preset included angle threshold and a direction vector included angle, and finally obtaining a three-dimensional medical image fused by a CT mode and an MRI mode.

For further explanation with reference to fig. 4-8, fig. 4 is a three-dimensional medical image fused by a CT modality of a femoral position region and an MRI modality of the femoral position region; FIG. 5 is a three-dimensional medical image of a CT modality of a femoral position region and a fused MRI modality of the femoral position region and an MRI modality of a femoral necrosis position region; FIG. 6 is a three-dimensional CT medical image of a femoral position region after image segmentation and three-dimensional reconstruction by the multi-modal medical image fusion method based on deep learning provided by the present invention; FIG. 7 is a three-dimensional MRI medical image of a femoral position region after image segmentation and three-dimensional reconstruction by the deep learning based multi-modal medical image fusion method provided by the present invention; FIG. 8 is a three-dimensional MRI medical image of a femoral necrosis location region after image segmentation and three-dimensional reconstruction by the deep learning based multi-modal medical image fusion method provided by the present invention;

FIG. 2C is a CT modality and MRI modality fused two-dimensional medical image of the patient's femoral position region obtained after point cloud registration fusion; FIG. 4 is a three-dimensional medical image of the patient's femur position region obtained by three-dimensional reconstruction in FIG. 2C using a CT modality and an MRI modality fusion; fig. 5 is a three-dimensional medical image fused by the CT modality of the femoral position region of the patient, the MRI modality of the femoral position region and the MRI modality of the femoral necrosis position region, which are obtained after the point cloud registration fusion; FIG. 6 is a three-dimensional CT medical image of the patient's femoral location area obtained after three-dimensional reconstruction through the above steps; fig. 7 is a three-dimensional MRI medical image of the femoral position region of the patient obtained after the three-dimensional reconstruction through the above-mentioned steps, and fig. 8 is a three-dimensional MRI medical image of the femoral necrosis position region of the patient obtained after the three-dimensional reconstruction through the above-mentioned steps.

Fig. 6 and 7 can be obtained by fusing first to obtain fig. 4, fig. 4 shows a fused three-dimensional image under the condition that the femur is not necrotic, and fig. 8 is a three-dimensional MRI medical image of the femur necrosis location area after image segmentation and three-dimensional reconstruction by the multi-modal medical image fusion method based on deep learning provided by the invention, and can also be understood as a three-dimensional MRI medical image of the independent femur necrosis location area. The three-dimensional MRI medical image of the femoral necrosis location area of figure 8 and the three-dimensional MRI medical image of the femoral location area of figure 7 may be combined as one three-dimensional medical image entity, but when the point cloud registration fusion processing is carried out, the three-dimensional medical images of the point cloud registration fusion processing and the point cloud registration fusion processing are essentially used as a three-dimensional medical image whole after the three-dimensional medical images are fused, the point cloud registration and fusion are further carried out on the new three-dimensional MRI medical image whole of the femur position area and the three-dimensional CT medical image of the femur position area, of course, it is also possible to fuse the three-dimensional CT medical image of the femoral position region shown in fig. 6 and the three-dimensional MRI medical image of the femoral position region shown in fig. 7 to obtain fig. 4, fuse fig. 4 and fig. 8 to obtain fig. 5, namely, a three-dimensional CT medical image of a femoral position area, a three-dimensional MRI medical image of the femoral position area and a three-dimensional MRI medical image of a femoral necrosis position area are finally obtained.

Registering the points together according to an ICP point cloud registration algorithm, and then performing three-dimensional reconstruction to obtain a comprehensive result: the CT modality and the MRI modality fuse three-dimensional medical images. The CT modality and the MRI modality are fused with the three-dimensional medical image, so that different characteristics of the images of the CT modality and the MRI modality are fused accurately, a real femoral necrosis position region of the patient can be reflected (as shown in a small special-shaped region part above the inner part of the femoral head in fig. 5), and accurate reference basis before treatment of the patient suffering from the hip joint disease can be provided for medical staff. It should be noted that the embodiment of the present application only shows the femur shape schematic in the three-dimensional CT medical image and the three-dimensional MRI medical image of the femur position region of the patient, and each point set based on the point cloud registration fusion needs to be combined with each image in the computer system to establish a corresponding coordinate system and obtain a corresponding coordinate point value, and the specific parameters are set according to the actual application scenario, which is not limited herein.

According to the multi-modal medical image registration and fusion method based on deep learning provided by the invention, the principles of the pre-training process of each image segmentation network model corresponding to the two-dimensional medical image under each modality are consistent, and the embodiment of the invention only takes the pre-training process of the CT image segmentation network model and the pre-training process of the MRI image segmentation network model as examples for explanation.

Fig. 9 is a schematic flow chart of a pre-training process of a CT image segmentation network model in the method provided by the present invention, and as shown in fig. 9, the pre-training process of the CT image segmentation network model in the method includes:

s610, acquiring a two-dimensional CT medical image data set of a plurality of patients, wherein the two-dimensional CT medical image data set comprises a plurality of two-dimensional CT medical images;

a large number of two-dimensional CT medical image datasets of patients suffering from hip joint disease are acquired, wherein the two-dimensional CT medical image datasets comprise a plurality of two-dimensional CT medical images.

S620, marking the femoral position area in each two-dimensional CT medical image by adopting at least one mode of automatic marking and manual marking;

the femoral position region is automatically or manually marked for each two-dimensional CT medical image in the two-dimensional CT medical image data set respectively, and the femoral position region is used as the base of a database. The automatic labeling can be performed by means of labeling software. Thereby obtaining a two-dimensional CT medical image data set formed by the marked two-dimensional CT medical images.

S630, dividing each two-dimensional CT medical image after labeling into a CT training data set and a CT testing data set according to a preset proportion;

before the training data set and the test data set are divided, each two-dimensional CT medical image in the labeled two-dimensional CT medical image data set needs to be subjected to corresponding format conversion, so that the two-dimensional CT medical images can smoothly enter an image segmentation network for processing. Specifically, the two-dimensional cross-section DICOM format of each two-dimensional CT medical image in the labeled two-dimensional CT medical image dataset is converted into a picture in the JPG format.

Dividing each two-dimensional CT medical image which is labeled and subjected to format conversion into a CT training data set and a CT testing data set according to the pre-ratio of 7: 3. The CT training dataset is used as input for a CT image segmentation network to train a CT image segmentation network model. And the CT test data set is used for testing and optimizing the performance of the CT image segmentation network model subsequently.

S640, training a CT image segmentation network model based on a CT training data set and in combination with a neural network algorithm and deep learning;

based on a CT training data set and combined with a neural network algorithm and deep learning, deep features of image data in the CT training data set are identified by utilizing multiple downsampling, the learned deep features are reversely stored into the image data by utilizing multiple upsampling, so that an image rough segmentation result is obtained through a first image segmentation network (a main image segmentation network), and multiple points with uncertain classification are accurately segmented through a second image segmentation network (a subordinate image segmentation network) to obtain an accurate segmentation result. And finally training a CT image segmentation network model.

Alternatively, fig. 10 is a schematic flow chart of a pre-training process of an MRI image segmentation network model in the method provided by the present invention, and as shown in fig. 10, the pre-training process of the MRI image segmentation network model in the method includes:

s710, acquiring two-dimensional MRI medical image data sets of a plurality of patients, wherein the two-dimensional MRI medical image data sets comprise a plurality of two-dimensional MRI medical images;

acquiring a two-dimensional MRI medical image dataset of a large number of patients with hip joint disease (the same patients are required in step S610), wherein the two-dimensional MRI medical image dataset comprises a plurality of two-dimensional MRI medical images;

s720, marking the femoral position area in each two-dimensional MRI medical image by adopting at least one mode of automatic marking and manual marking;

the femur position area is automatically or manually marked for each two-dimensional MRI medical image in the two-dimensional MRI medical image data set, and if femur necrosis occurs, the femur necrosis position area is also marked, and the femur necrosis position area is also used as a database foundation. The automatic labeling can be performed by means of labeling software. Thereby obtaining a two-dimensional MRI medical image data set formed by the marked two-dimensional MRI medical images.

S730, dividing each two-dimensional MRI medical image after being labeled into an MRI training data set and an MRI test data set according to a preset proportion;

before the training data set and the test data set are divided, each two-dimensional MRI medical image in the labeled two-dimensional MRI medical image data set needs to be subjected to corresponding format conversion, so that the two-dimensional MRI medical images can smoothly enter an image segmentation network for processing. Specifically, the original format of each two-dimensional MRI medical image in the labeled two-dimensional MRI medical image dataset is converted into a PNG-format picture.

And dividing each two-dimensional MRI medical image which is labeled and subjected to format conversion into an MRI training data set and an MRI test data set according to a pre-ratio of 7: 3. The MRI training dataset is used as input to an MRI image segmentation network to train an MRI image segmentation network model. And the MRI test data set is used for subsequently testing and optimizing the performance of the MRI image segmentation network model.

And S740, training an MRI image segmentation network model based on the MRI training data set and combining a neural network algorithm and deep learning.

Based on an MRI training data set and combined with a neural network algorithm and deep learning, deep features of image data in the MRI training data set are identified by utilizing multiple downsampling, the learned deep features are reversely stored into the image data by utilizing multiple upsampling, so that an image rough segmentation result is obtained through a first image segmentation network (a main image segmentation network), and multiple points with uncertain classification are accurately segmented through a second image segmentation network (a subordinate image segmentation network) to obtain an accurate segmentation result. And finally, training an MRI image segmentation network model. According to the multi-modal medical image registration fusion method provided by the invention, a CT image segmentation network model or an MRI image segmentation network model is trained based on a CT training data set or an MRI training data set and combined with a neural network algorithm and deep learning, fig. 11 is a deep learning training network structure diagram of a training process shown in fig. 9 and fig. 10, and is further combined with fig. 11, the training process of the model specifically comprises the following steps:

(1) carrying out coarse segmentation processing on the CT training data set or the MRI training data set through a first image segmentation model: performing a plurality of downsamplings of image data in the CT training dataset or the MRI training dataset to identify deep features of each image data through processing of the convolutional layer and the pooling layer; performing up-sampling on the down-sampled image data for a plurality of times to reversely store the deep features into the image data through processing of an up-sampling layer and a convolution layer; carrying out image rough classification processing by using an Adam classification optimizer to obtain an image rough segmentation result;

first, a first image segmentation model (unet backbone neural network, abbreviated as unet backbone neural network) is used to perform Coarse segmentation processing (Coarse segmentation) on a CT training data set or an MRI training data set. The first stage performs 4 downsampling to learn the deep features of each image data of the CT training data set or the MRI training data set. Each downsampling layer comprises 2 convolutional layers and 1 pooling layer, the size of a convolution kernel in each convolutional layer is 3 x 3, the size of a convolution kernel in each pooling layer is 2 x 2, and the number of convolution kernels in each convolutional layer is 128, 256, 512 and the like. The downsampled image data is further upsampled 4 times to restore deep features of the respective image data learned by downsampling into the image data. Each upsampling layer comprises 1 upsampling layer and 2 convolutional layers, wherein the size of a convolution kernel of each convolutional layer is 3 x 2, the size of a convolution kernel in each upsampling layer is 2 x 2, and the number of convolution kernels in each upsampling layer is 512, 256, 128 and the like. The above-described sampling process of the convolutional neural network is a process of extracting features of each image data. The characteristic parts need to be identified in each original image, and specifically, deep characteristics of each original image are continuously and repeatedly learned through a convolutional neural network and are finally reversely stored on the original image. And carrying out image rough classification processing by using an Adam classification optimizer to obtain an image rough segmentation result.

(2) And performing fine segmentation processing on the image rough segmentation result through a second image segmentation model: feature point data with preset confidence coefficient is screened from the deep features, bilinear interpolation calculation is carried out on the feature point data, the category of the deep features is identified based on the calculated feature point data, and a final image segmentation result is obtained.

And further performing fine segmentation on the image rough segmentation result obtained after the unet main neural network processing by using a second image segmentation model (pointrend slave neural network, referred to as pointrend slave neural network for short). Performing up-sampling learning calculation on the image rough segmentation result through a Biliner Bilinear interpolation method to obtain an intensive feature map of each image; selecting a plurality of points with unknown affiliated classifications from the dense feature map of each image, namely selecting N points with the most uncertain affiliated classifications, such as selecting a plurality of points with confidence/probability of 0.5, then calculating a table and extracting deep feature representation of the N points, and predicting the affiliated classifications of the N points after fine segmentation point by using an MLP multilayer sensor, such as judging whether the points belong to a femur region or a non-femur region; and repeating the steps until the classification of each point in the N points after the fine segmentation is predicted one by one. When the MLP multi-layer perceptron is used for predicting the classification of each point after the N points are finely divided point by point, a small classifier is used for judging which classification the point belongs to, and the prediction is equivalent to the prediction by using 1-by-1 convolution. However, for points with confidence close to 1 or 0, the classification of the points is still clear, so that the points do not need to be predicted point by point. The number of the needed prediction points is reduced, and the accuracy of the final image segmentation result is improved on the whole. Thus, an optimized fine image segmentation result (optimized prediction) is finally obtained.

(3) Calculating a loss function based on the final image segmentation result and the CT training dataset or the MRI training dataset;

(4) and adjusting parameters of the CT image segmentation network model or the MRI image segmentation network model based on the loss function until the CT image segmentation network model or the MRI image segmentation network model is successfully trained.

The effect of setting the loss function is that the size of the number of samples trained each time can be adjusted according to the change of the loss function in the model pre-training process. Specifically, in the course of the coarse segmentation process of the CT training dataset or the MRI training dataset through the unet main neural network, the initial value of the Size of the sample number Batch _ Size of each training is set to 6, the learning rate is set to 1e-4, the optimizer uses the Adam optimizer, and the loss function DICE loss is set. When the CT training data set or the MRI training data set is completely sent into the unet main neural network for training, the Size of the sample number Batch _ Size of each training can be effectively adjusted in real time according to the change condition of the loss function in the training process, so that the processing accuracy is improved in the coarse segmentation processing stage.

According to the multi-modal medical image registration fusion method based on deep learning provided by the invention, the method further comprises the following steps:

setting an activation function after each convolution layer;

wherein, all convolution layers are provided with activation functions behind, such as relu activation function, Sigmoid activation function, tanh activation function, leave relu activation function, etc., to enhance the nonlinear factor of the convolutional neural network, so that the more complicated calculation processing process can be better solved through the convolutional neural network.

And/or in the course of roughly segmenting a CT training data set or an MRI training data set through a first image segmentation model, a dropout layer is arranged after the last upsampling is finished;

after the last upsampling is finished, or after the last upsampling layer, a dropout layer is arranged and used for temporarily discarding some neural network units from the network according to a certain probability in the training process of the deep learning network so as to further improve the accuracy of model training. Wherein the probability of dropout layer is set to 0.7.

In the following, the multimodal medical image fusion system based on deep learning provided by the present invention is explained, and the multimodal medical image fusion system based on deep learning corresponds to the multimodal medical image fusion method based on deep learning of any of the above embodiments, and the principles thereof can be referred to one another, so that the details are not repeated herein.

The invention also provides a multimodal medical image fusion system based on deep learning, fig. 12 is one of the structural schematic diagrams of the multimodal medical image fusion system based on deep learning provided by the invention, as shown in fig. 12, the system includes: a multi-modality image acquisition unit 1010, a two-dimensional image output unit 1020, a two-dimensional image fusion unit 1030, and a three-dimensional reconstruction unit 1040, wherein,

a multi-modality image acquisition unit 1010 configured to acquire two-dimensional medical images of at least two modalities of a patient;

a two-dimensional image output unit 1020 configured to input the two-dimensional medical images of the at least two modalities to a pre-trained image segmentation network model to obtain outputs of the two-dimensional medical images of respective modality body position regions, respectively;

a two-dimensional image fusion unit 1030 configured to perform point cloud registration fusion on the two-dimensional medical images of the respective modality body position regions based on a point cloud registration algorithm to obtain a multi-modality fused two-dimensional medical image;

a three-dimensional reconstruction unit 1040 configured to perform three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain a multi-modal fused three-dimensional medical image.

According to the multi-modal medical image fusion system based on deep learning, the modules are matched with each other, so that the system can perform image segmentation processing on two-dimensional medical images of the same part of the same patient in different modes respectively, perform accurate point cloud registration fusion on the two-dimensional medical images in different modes, and finally perform three-dimensional reconstruction to obtain the multi-modal fused three-dimensional medical image. The system has high registration precision and low time cost for multi-modal medical image fusion registration, can also process more complex multi-modal image fusion conditions, can also be applied to non-rigid registration conditions, and can provide accurate treatment reference for medical personnel.

Fig. 13 is a schematic structural diagram of the electronic device provided in the present invention, and as shown in fig. 13, the electronic device may include: a processor (processor)1310, a communication Interface (Communications Interface)1320, a memory (memory)1330 and a communication bus 1340, wherein the processor 1310, the communication Interface 1320 and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 may invoke logic instructions in the memory 1330 to perform all or part of the steps of the deep learning based multi-modal medical image fusion method, the method comprising:

acquiring two-dimensional medical images of at least two modalities of a patient;

respectively inputting two-dimensional medical images of at least two modes into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the mode bodies;

performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modal bodies based on a point cloud registration algorithm to obtain a multi-modal fused two-dimensional medical image;

and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image.

In addition, the logic instructions in the memory 1330 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention or parts thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the multimodal medical image registration and fusion method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present invention further provides a computer program product including a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer being capable of executing all or part of the steps of the deep learning based multi-modal medical image fusion method provided by the above embodiments, the method including:

acquiring two-dimensional medical images of at least two modalities of a patient;

respectively inputting two-dimensional medical images of at least two modes into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the mode bodies;

performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modal bodies based on a point cloud registration algorithm to obtain a multi-modal fused two-dimensional medical image;

and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image.

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon computer instructions, which when executed by a processor, implement all or part of the steps of the deep learning based multi-modal medical image fusion method provided by the above embodiments, the method comprising:

acquiring two-dimensional medical images of at least two modalities of a patient;

respectively inputting two-dimensional medical images of at least two modes into corresponding image segmentation network models trained in advance so as to respectively obtain the output of the two-dimensional medical images of the position areas of the mode bodies;

performing point cloud registration fusion on the two-dimensional medical images of the position areas of the various modal bodies based on a point cloud registration algorithm to obtain a multi-modal fused two-dimensional medical image;

and performing three-dimensional reconstruction processing on the multi-modal fused two-dimensional medical image to obtain the multi-modal fused three-dimensional medical image.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on such understanding, the above technical solutions may be essentially or partially implemented in the form of software products, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the multimodal medical image registration and fusion method according to the various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.