阅读说明：本技术 一种3d颅脑mri医学影像报告自动生成方法及系统 (Automatic generation method and system for 3D craniocerebral MRI medical image report ) 是由 杨承林 陈先来 彭鹏 于 2021-07-22 设计创作，主要内容包括：本发明公开了一种3D颅脑MRI医学影像报告自动生成方法及系统,包括：获取待生成医学影像报告的3D颅脑MRI医学图像；将所述待生成医学影像报告的3D颅脑MRI医学图像输入到预先训练好的基于Transformer的3D颅脑MRI医学影像报告自动生成模型中,生成医学影像报告；所述预先训练好的模型,是通过患者的3D颅脑MRI医学图像数据及对应的医学影像报告文本数据训练得到的。本发明公开的3D颅脑MRI医学影像报告自动生成方法在三维图像处理上做出了探索,采用3D CNN对图像特征进行有效的提取,并在Transformer解码器部分,引入一个额外的记忆网络模块,以提高医学影像报告自动生成的质量,用以辅助医生进行颅脑的精确诊断及相应报告的书写,减少阅片和撰写报告的时间。(The invention discloses a method and a system for automatically generating a 3D brain MRI medical image report, which comprises the following steps: acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated; inputting the 3D brain MRI medical image of the medical image report to be generated into a pre-trained automatic generation model of the 3D brain MRI medical image report based on a Transformer to generate a medical image report; the pre-trained model is obtained by training 3D craniocerebral MRI medical image data of a patient and corresponding medical image report text data. The automatic generation method of the 3D brain MRI medical image report disclosed by the invention explores in three-dimensional image processing, adopts 3D CNN to effectively extract image characteristics, and introduces an additional memory network module in a transform decoder part to improve the automatic generation quality of the medical image report, so as to assist doctors in accurate diagnosis of the brain and writing of corresponding reports and reduce the time for reading and writing the reports.)

1. A3D craniocerebral MRI medical image report automatic generation method is characterized by comprising the following steps:

acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

inputting the 3D brain MRI medical image of the medical image report to be generated into a pre-trained transform-based 3D brain MRI medical image report automatic generation model to obtain a medical image report result output by the transform-based 3D brain MRI medical image report automatic generation model; assisting doctors to carry out diagnosis of craniocerebral related diseases and writing of reports;

the pre-trained automatic generation model of the 3D brain MRI medical image report based on the Transformer is obtained based on training of a training set, and the training set comprises 3D brain MRI medical image data of a patient and corresponding text data of the medical image report.

2. The method for automatically generating the 3D brain MRI medical image report according to claim 1, wherein the transform-based 3D brain MRI medical image report automatic generation model comprises the following training steps:

acquiring 3D craniocerebral MRI (magnetic resonance imaging) medical image data of a patient and corresponding medical image report text data; matching and corresponding the acquired 3D brain MRI medical image data and the medical image report text data, and deleting the record of data loss;

preprocessing the acquired 3D brain MRI medical image data and corresponding medical image report text data, wherein the preprocessing comprises establishing image report pairs, Chinese word segmentation, text description and other long-distance completions and establishing a dictionary for processing; storing the preprocessed result;

constructing a data set, wherein the data set is 3D craniocerebral MRI medical image data of a patient and corresponding medical image report text data;

the data set is divided into the following parts according to the proportion of 3:1: a training set, a verification set and a test set;

constructing a Transformer-based 3D craniocerebral MRI medical image report to automatically generate an initial model;

inputting the training set into a Transformer-based 3D brain MRI medical image report automatic generation initial model, training the Transformer-based 3D brain MRI medical image report automatic generation model, and obtaining the Transformer-based 3D brain MRI medical image report automatic generation model after training.

3. The method for automatically generating a 3D brain MRI medical image report according to claim 1, wherein the pre-trained transform-based 3D brain MRI medical image report automatic generation model has a network architecture mainly comprising:

the system comprises a visual feature extractor, an encoder and a decoder;

the input end of the visual feature extractor is used for inputting a 3D brain MRI medical image, the visual feature extractor is used for carrying out feature extraction on the 3D brain MRI medical image, and a compact feature map for capturing space and depth information and abundant local 3D context features are generated; efficient embedding of rich local 3D context features is fed to the encoder, which can then be exploited to model long-range dependencies in global space; compressing the space dimension and the depth dimension into one dimension, introducing position embedding, and fusing the space dimension and the depth dimension with the feature map through a residual error module; the output of the encoder is a hidden state encoded in input features extracted from a visual extractor; the output of the encoder is taken as input to the decoder, which will generate a description for each possible region of interest.

4. The Transformer-based 3D craniocerebral MRI medical image report auto-generation model of claim 3, wherein the visual feature extractor, built using 3D CNN (convolutional neural network), generates compact feature maps that capture spatial and depth information, allowing the use of intermediate high-resolution CNN feature maps in the decoding path better than using a Transformer alone as an encoder.

5. The Transformer-based 3D craniocerebral MRI medical image report auto-generation model of claim 3, wherein the encoder employs a standard encoder of a Transformer to model long-range correlations in global space.

6. The Transformer-based 3D craniocerebral MRI medical image report automatic generation model as claimed in claim 3, wherein the decoder mainly adopts the decoder part of the Transformer and introduces an additional memory network module in layer normalization, so that the Transformer can model long-distance information of the medical image report and model the modeling information therein.

7. The Transformer-based 3D craniocerebral MRI medical image report automatic generation model of claim 6, wherein the memory network module is used for recording information from previous generation processes and integrating the memory network module into the Transformer through layer normalization, so that similar patterns in different medical image reports can be modeled and memorized, thereby facilitating the decoding of the Transformer.

8. A 3D cranial MRI medical image report automatic generation system, comprising:

an acquisition module configured to: acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

a report generation module configured to: inputting a 3D brain MRI medical image of a medical image report to be generated into a pre-trained transform-based 3D brain MRI medical image report automatic generation model to generate a medical image report; assisting doctors to accurately diagnose the craniocerebral related diseases and write reports;

the pre-trained automatic generation model of the 3D brain MRI medical image report based on the Transformer is obtained by training a training set, and the training set comprises 3D brain MRI medical image data of a patient and corresponding medical image report text data.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 7.

Technical Field

The invention relates to the technical field of generating medical image reports based on an artificial intelligence technology, in particular to a method and a system for automatically generating 3D brain MRI medical image reports based on a Transformer.

Background

In the medical diagnosis process, medical images are important disease diagnosis tools, and cranial Magnetic Resonance Imaging (Magnetic Resonance Imaging) can provide information about brain tissues and brain structures, has the advantage of rich Imaging information, and provides powerful support for the judgment of doctors. Due to the complexity of the brain structure and the diversity of diseases, it is very difficult to correctly judge the diseases reflected on the medical images, which may lead to misdiagnosis and missed diagnosis, and it is necessary for experienced doctors to carefully examine to make correct judgment. Moreover, training a professional and experienced doctor often takes years, which results in experienced doctor short supply.

With the development of artificial intelligence technology, the method provides possibility for constructing a high-performance medical image auxiliary diagnosis system. For the field of medical image processing, a corresponding text report can be quickly and automatically generated according to a medical image, and the comprehensive diagnosis of diseases and the writing of the report by doctors are facilitated. But abnormal regions in medical images are difficult to identify, especially 3D craniocerebral MRI; moreover, a complete image report contains abundant information and a plurality of sentences, which are usually long, and this brings great challenges to the automatic generation of medical image reports. In addition, the research on the three-dimensional medical image processing and report automatic generation technology based on the artificial intelligence technology is still in the beginning stage, and the latest research published internationally is very few.

In order to solve the challenge of long medical image report length, the existing research method uses a hierarchical recurrent neural network, so that long text information can be better modeled, and the problem of long text generation is solved to a certain extent. However, they do not take good advantage of the features of the medical image report generation task. Despite these challenges, medical image reports often feature a modeling feature, i.e., similar literary patterns exist between different medical image reports, and this modeling information can effectively help medical image report generation.

Disclosure of Invention

Aiming at the problems, the invention provides a method and a system for automatically generating a 3D craniocerebral MRI medical image report; by performing model training on the 3D craniocerebral MRI medical image data and the corresponding medical image report text data, the trained model can generate a report for assisting a doctor in diagnosing brain diseases. The invention explores a three-dimensional medical image analysis processing technology and provides a method for automatically generating a 3D brain MRI medical image report, a transform model is not simply adopted, but a 3D CNN is firstly adopted to effectively extract image characteristics, an additional memory network module is introduced into a transform decoder part, and modeling is better performed on long-distance information of the medical image report, modeling information in the medical image report is performed, so that the quality of automatic generation of the 3D brain MRI medical image report is improved, the generation process of the medical image report is improved, a doctor is finally assisted to correctly judge the brain MRI medical image and write a corresponding report, the time for reading and writing the report is reduced, and the quality of the brain MRI medical image report is improved.

In a first aspect, the invention provides a method for automatically generating a 3D brain MRI medical image report;

the automatic generation method of the 3D craniocerebral MRI medical image report comprises the following steps:

acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

inputting the 3D brain MRI medical image of the medical image report to be generated into a pre-trained automatic generation model of the 3D brain MRI medical image report based on a Transformer, and automatically generating the model of the 3D brain MRI medical image report based on the Transformer to generate the medical image report; assisting doctors to carry out diagnosis of craniocerebral related diseases and writing of reports;

the pre-trained automatic generation model of the 3D brain MRI medical image report based on the Transformer is obtained by training a training set, wherein the training set comprises 3D brain MRI medical image data of a patient and corresponding medical image report text data.

In a second aspect, the invention provides a system for automatically generating a 3D craniocerebral MRI medical image report;

the automatic generation system of 3D craniocerebral MRI medical image report comprises:

an acquisition module configured to: acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

a report generation module configured to: inputting a 3D brain MRI medical image of a medical image report to be generated into a pre-trained automatic generation model of the 3D brain MRI medical image report based on a Transformer, and generating a corresponding medical image report by the automatic generation model of the 3D brain MRI medical image report based on the Transformer; assisting doctors to carry out diagnosis of craniocerebral related diseases and writing of reports;

the pre-trained automatic generation model of the 3D brain MRI medical image report based on the Transformer is obtained by training a training set, wherein the training set comprises 3D brain MRI medical image data of a patient and corresponding medical image report text data.

In a third aspect, an embodiment of the present invention provides an electronic device, including:

a memory, a processor, and a computer program stored on the memory and executable on the processor; when the electronic device is running, the processor executes one or more computer programs stored in the memory to make the electronic device execute the automatic generation method of the 3D craniocerebral MRI medical image report.

In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program, when being executed by a processor, implementing the steps of any one of the automatic generation methods for 3D brain MRI medical image reports.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can obtain other drawings according to these drawings without creative efforts.

FIG. 1 is a flow chart of a method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a model provided in an embodiment of the present invention;

FIG. 3 is a schematic diagram of model training provided by an embodiment of the present invention;

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. Specific details are described to provide a thorough understanding of various embodiments of the invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present inventions. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the description of the present invention, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, article, or apparatus.

The embodiment of the invention provides a flow chart of a method for automatically generating a 3D brain MRI medical image report, which comprises the following steps of:

acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

inputting the 3D brain MRI medical image of the medical image report to be generated into a pre-trained transform-based 3D brain MRI medical image report automatic generation model to obtain a medical image report result output by the transform-based 3D brain MRI medical image report automatic generation model; assisting doctors to carry out diagnosis of craniocerebral related diseases and writing of reports; the transform-based 3D brain MRI medical image report automatic generation model is obtained based on training of a training set, wherein the training set comprises 3D brain MRI medical image data of a patient and corresponding medical image report text data.

The training set of the transform-based 3D brain MRI medical image report automatic generation model is constructed based on sample set data obtained by performing a series of preprocessing on brain MRI medical image data of a patient and corresponding medical image report text data, and comprises the steps of performing matching correspondence on the 3D brain MRI medical image data and the medical image report text data, deleting data missing records and preprocessing a Chinese text report.

Based on the embodiment, the Transformer-based 3D craniocerebral MRI medical image report automatic generation model is obtained through the following steps:

acquiring a 3D craniocerebral MRI medical image of a patient and corresponding medical image report text sample set data, and preprocessing the sample set data to obtain preprocessed sample set data;

generating a data set for training based on the pre-processed sample set data;

constructing a Transformer-based 3D craniocerebral MRI medical image report to automatically generate an initial model;

inputting the data set into the transform-based 3D brain MRI medical image report to automatically generate an initial model for training, and obtaining the transform-based 3D brain MRI medical image report automatic generation model after training.

The method includes the steps of obtaining a 3D craniocerebral MRI medical image of a patient and corresponding medical image report text sample set data, preprocessing the sample set data to obtain preprocessed sample set data, and specifically includes the following steps:

acquiring 3D brain MRI medical images for training and corresponding medical image report text sample set data;

matching and corresponding the acquired 3D brain MRI medical image data and the medical image report text data, deleting the record of data loss, performing long-term complementation on Chinese word segmentation and text description of the medical image report, and establishing a dictionary for processing;

as one or more embodiments, as shown in fig. 2, the transform-based 3D brain MRI medical image report automatic generation model includes:

a visual feature extractor, an encoder and a decoder;

the visual feature extractor extracts abundant local 3D context features of the brain MRI medical image by using a 3D CNN (convolutional neural network), generates a compact feature map for capturing space and depth information, allows the CNN feature map with intermediate high resolution to be used in a decoding path, and has better effect than the effect of simply using a Transformer as an encoder;

the ENCODER (encoDER) uses a standard ENCODER of a Transformer for global feature modeling.

The DECODER (DECODER) adopts the main part of a transform DECODER, and introduces an additional network Memory module (Memory) in layer normalization, so that modeling information in the medical image is modeled while long-distance information is reported;

the visual feature extractor is established by adopting a 3D CNN method, and specifically comprises the following steps:

the visual feature extractor 3D CNN is a neural network with a 3D convolutional layer and a 3D pooling layer, 3D brain MRI medical images acquired in a training set are input into the 3D CNN for rich local 3D context feature extraction to obtain a feature representation F, and thus the rich local 3D context features are effectively embedded into the F; and folding the space dimension and the depth dimension into a single dimension to obtain a characteristic vector f. In order to encode vital position information in the task of report generation, learnable position embeddings are introduced and fused with the feature map f by a residual module, creating feature embeddings.

Wherein, the encoder adopts a standard encoder of a Transformer to model the global characteristics, and further comprises:

the encoder employs a transform's standard encoder in the model, where the output is a state hidden from the input features extracted by the feature extractor; transformer encoder is perceived by N-layer Multi-Head Attention (MHA) and Multi-layersA block of devices (MLP). Thus, the output of the first layer may be written in the form of z'_l＝MHA(LN(z_l-1))+z_l-1，z_l＝MLP(LN(z′_l))+z′_lWhere LN (-) denotes the layer normalization operator, z_LRepresenting an encoded image representation.

Wherein, the decoder mainly adopts a transform decoder part, and introduces an additional Memory network module (Memory) in layer normalization, which specifically comprises:

the memory network module aims at learning the modeling information of the medical image report, introduces the memory network module into layer normalization, controls the mean value and the variance of the output characteristics of the Transformer by using memory when the medical image report is generated, and deeply utilizes the modeling information of the medical image report.

The memory network module is a matrix, and in the generation process, when a word is generated each time, the memory network module needs to be updated by the word generated at the last moment;

wherein, the memory network module updating rule specifically includes:

given the Memory at the previous time and the word vectors of the words generated at the previous time, they will be input to multi-headed attention, specifically, as query (Q) of the MHA, and concatenated as key (K) and value (V) of the MHA. The residual error result of the MHA is subjected to updated memory through a multilayer perceptron;

as one or more embodiments, as shown in fig. 3, the pre-trained transform-based 3D brain MRI medical image report automatic generation model includes:

acquiring 3D craniocerebral MRI medical image data and medical image report text data of a patient; matching and corresponding the acquired 3D brain MRI medical image data and the medical image report text data, and deleting the record of data loss;

preprocessing the 3D craniocerebral MRI medical image data and the medical image report text data, wherein the preprocessing comprises establishing image report pairs, Chinese word segmentation, text description and other long-distance completions and establishing a dictionary;

constructing a data set, wherein the data set is 3D craniocerebral MRI medical image data of a patient and corresponding medical image report text data;

the data set is divided into the following parts according to the proportion of 3:1: a training set, a verification set and a test set;

and inputting the training set into a Transformer-based 3D brain MRI medical image report automatic generation model, and training the model to obtain the trained Transformer-based 3D brain MRI medical image report automatic generation model.

In particular, given an input 3D craniocerebral MRI medical imageThe spatial resolution is H W, the depth dimension is D and the channel is C. First, a compact feature map is generated using 3D CNN to capture spatial and depth information, rich local 3D context features are efficiently embedded and fed to a Transformer encoder, which consists of N Transformer layers, each layer having a standard architecture consisting of a multi-head attention (MHA) module and a Feed Forward Network (FFN). And further constructing a brain MRI medical text report generation initial model consisting of a visual feature extractor, a coder and a decoder, carrying out model training on the initial model by using the 3D brain MRI medical image data and the corresponding medical image report text data until the model reaches a convergence condition, and obtaining a final transform-based 3D brain MRI medical image report automatic generation model.

Further, the creating of the image report pair is to create the image report pair through a dictionary object in python, such as: { 'image _ id': picture id, 'report': report corresponding to picture }.

Furthermore, the Chinese word segmentation processing is to put the report text into a word segmentation device to segment the text description; all medical image reports are tokenized using the jieba tokenization package in python, and then the comma and pause in the report are replaced with white spaces.

Further, the text description is subjected to an equal length padding process, symbols S and E representing the start and End are added before and after all medical image reports, and all medical image report lengths are padded to the same length as the longest text report in the data set using the 'End' character.

Further, the dictionary building step is to firstly count the results of all the word segmentation and build the dictionary.

Further, after the step of acquiring 3D brain MRI medical image data and medical image report text data, the method further comprises:

desensitizing sensitive information in medical image report text data, uniquely identifying a patient by using a patient id, and desensitizing private information of the patient contained in the data.

It should be understood that deleting a record with missing data refers to: such data is deleted for only the craniocerebral MRI medical images without corresponding medical image reports.

The acquired 3D brain MRI medical image and the corresponding preprocessed medical image report text data are constructed to be used as a data set, and the data set is divided into a training set, a verification set and a test set according to the proportion of 3:1: 1.

Training the model by using data of the training set; since the model involves both the extraction of the 3D medical image features of the cranium and the generation of the medical image report, it is not appropriate to set the iteration round of the model to be too large during the model training. In the training process, if the preset training round is not reached, continuing training, otherwise, performing the next step; and the verification set data is used for parameter adjustment and model optimization, the model is tested in the training process, and the evaluation indexes of BLEU1_4, METEOR, CIDr and ROUGE are output, so that the model training condition can be observed conveniently. When the model is completely iterated, the model is stored, so that the visualization display of the system is convenient to perform later, and a corresponding medical image report can be generated automatically according to the 3D brain MRI medical image.

Acquiring a new 3D brain MRI medical image, inputting the 3D brain MRI medical image into a trained transform-based 3D brain MRI medical image report automatic generation model, and generating a corresponding medical image report by the model according to the extracted characteristics.

The present embodiment provides a 3D brain MRI medical image report automatic generation system, including:

an acquisition module configured to: acquiring a 3D craniocerebral MRI medical image of a medical image report to be generated;

a report generation module configured to: inputting a 3D brain MRI medical image of a medical image report to be generated into a pre-trained transform-based 3D brain MRI medical image report automatic generation model, and generating and displaying a corresponding medical image report; assisting doctors to carry out diagnosis of craniocerebral related diseases and writing of reports;

the pre-trained automatic generation model of the 3D brain MRI medical image report based on the Transformer is obtained by training a training set, and the training set comprises 3D brain MRI medical images of patients and corresponding medical image report text data.

It should be noted that the modules described above as part of a system may be implemented in a computer system such as a set of computer-executable instructions; the proposed system can be implemented in other ways.

According to the automatic generation method and system for the 3D brain MRI medical image report, provided by the embodiment of the invention, model training is carried out through the 3D brain MRI medical image of the patient and the corresponding medical image report text data, and the trained model can generate a report for assisting a doctor in diagnosing related craniocerebral diseases. The invention explores a three-dimensional medical image analysis processing technology and provides an automatic generation method of a 3D brain MRI medical image report, the method does not simply adopt a transform model, but firstly adopts 3D CNN to effectively extract image characteristics, and introduces an additional memory network module in a transform decoder part, so that the modeling of long-distance information of the medical image report is better realized, the modeling of the modeling information in the medical image report is better, the automatic generation quality of the 3D brain MRI medical image report is improved, the generation process of the medical image report is improved, the method is used for assisting a doctor to accurately diagnose and write reports of brain related diseases, the diagnosis time of the doctor is saved, the workload is reduced, the working efficiency of the doctor is improved, and the occurrence probability of missed diagnosis and misdiagnosis is reduced.

On the other hand, an embodiment of the present invention further provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, a processor is connected with the memory, the one or more computer programs are stored in the memory, and when the electronic device runs, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the automatic generation method of the 3D brain MRI medical image report according to the above embodiment.

It is to be understood that the processor may be a central processing unit CPU, but the processor may also be other general purpose processors, a digital signal processor DSP, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may include both read-only memory and random access memory, and may provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory.

In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software.

The automatic generation method of the 3D brain MRI medical image report in the embodiment can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules may reside in random access memory, flash memory, read only memory, programmable read only memory, or any other storage medium known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor. To avoid repetition, it is not described in detail here.

Embodiments also provide a computer readable storage medium for storing computer instructions, which when executed by a processor, perform the method for automatically generating a 3D brain MRI medical image report according to an embodiment.

The elements of each example, i.e., algorithm steps, that are described in connection with the embodiments disclosed herein may be embodied by one of ordinary skill in the art as electronic hardware or in combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation.

The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.