阅读说明：本技术 一种基于互卷积的生物组织透射图像配准系统 (Biological tissue transmission image registration system based on mutual convolution ) 是由 张宝菊 闫文睿 张翠萍 王忠强 王凤娟 赵志洋 王曼 费靖淇 于 2021-04-23 设计创作，主要内容包括：本发明公开了一种基于互卷积的生物组织透射图像配准系统：其主要包括：基于互卷积的回归器、空间转化器、采样器。将生物组织透射图像配准对输入至系统进行配准,系统输出透射图像配准后的结果图。系统可以通过训练实现更新,优化配准效果。本发明可以对关键点稀少、信噪比低的生物组织透射图像实现有效配准,保留浮动图像的细节纹理等信息,减少浮动图像在配准过程中的失真,提升生物组织透射图像配准的效果,减少系统占用存储空间。为乳腺肿瘤早期诊断等涉及低信噪比、低对比度的生物组织透射图像相关技术研究提供了支持。(The invention discloses a mutual convolution-based biological tissue transmission image registration system, which comprises the following steps: it mainly comprises: a regressor based on a mutual convolution, a space converter and a sampler. And registering the biological tissue transmission image registration pair input into the system, and outputting a result graph after transmission image registration by the system. The system can realize updating through training and optimize the registration effect. The invention can realize effective registration of the biological tissue transmission image with rare key points and low signal-to-noise ratio, retain the information of detail texture and the like of the floating image, reduce the distortion of the floating image in the registration process, improve the registration effect of the biological tissue transmission image and reduce the storage space occupied by the system. Provides support for the research of the related technology of the biological tissue transmission image with low signal-to-noise ratio and low contrast, such as the early diagnosis of the breast tumor.)

1. A system for registering transmission images of biological tissues based on mutual convolution, comprising: the system comprises a regressor, a space converter and a sampler based on mutual convolution, wherein after the floating transmission image and the fixed transmission image are input into the system, the floating transmission image and the fixed transmission image are subjected to dimension linkage and then sequentially pass through the regressor, the space converter and the sampler based on mutual convolution to finally obtain a transmission image after registration; the image registration system is updated through training, and the registration effect is improved; the detailed work flow comprises the following steps:

1) Inputting the transmission image to-be-registered pair into the system;

2) the input floating transmission image and the fixed transmission image are linked along the dimension direction and enter a regressor based on mutual convolution to obtain a deformation parameter;

3) the deformation parameters are continuously sent to a space converter for processing to obtain a space deformation field;

4) resampling the floating transmission image by using the deformation field obtained in the previous link of the sampler as a basis, and outputting a transmission image after registration;

the system updating method comprises the following steps:

inputting the prepared transmission image registration dataset into the system;

the input floating transmission image and the fixed transmission image are linked in batches along the dimension direction and enter a regressor (1) based on mutual convolution to obtain a deformation parameter;

3) the deformation parameters are continuously sent to a space converter for processing to obtain a space deformation field;

4) resampling the floating transmission image by using the deformation field obtained in the previous link of the sampler as a basis, and outputting a transmission image after registration;

5) the appearance difference and the local space change are punished through a loss function, the system parameters are adaptively optimized, and the system is updated;

6) repeating 2) -5) until the registration error is reduced to a required range or the iteration number reaches an upper limit;

At this point training is complete and the system is updated.

2. The system for registering transmission images of biological tissues based on mutual convolution of claim 1, wherein: the unit structure in the regression based on the mutual convolution includes convolution and mutual convolution.

3. The system for registering transmission images of biological tissues based on mutual convolution of claim 1, wherein: a framework in the regression device based on the mutual convolution is a network formed by embedding an inclusion module and a U-net module.

4. The system for registering transmission images of biological tissues based on mutual convolution of claim 1, wherein: the system can be trained through the unmarked biological tissue transmission image data set, self-adaptive updating is carried out, and the registration capability of the system to the biological tissue transmission image is improved.

5. The system for registering transmission images of biological tissues based on mutual convolution of claim 1, wherein: the method is oriented to the biological tissue transmission images with sparse key points and low signal-to-noise ratio, the image registration effect is improved end to end, and the storage space occupation of the system is reduced.

6. The system for registering transmission images of biological tissues based on mutual convolution of claim 4, wherein: the loss function includes a penalty for differences in appearance of the registered transmission image and the fixed transmission image and a penalty for local spatial variations in the deformation field.

7. Use of a deconvolution-based biological tissue transmission image registration system of claim 1 for early diagnosis of breast tumors.

Technical Field

The invention belongs to the field of computer vision, and particularly relates to a mutual convolution-based biological tissue transmission image registration system.

Background

The medical image registration technology is an indispensable key step in medical image analysis, and is a precondition for realizing medical image fusion, segmentation, contrast and reconstruction. Many medical pipelines rely on image registration technology, and the improvement of the registration technology has important significance for clinical application. Transmission Multispectral Imaging (TMI) enables early diagnosis of breast tumors. The optical imaging technology has the characteristics of no wound, real time, strong sensitivity and specificity and the like, can play a role in the field of early screening of human pathological tissues, and has important research value in biomedical imaging. However, the biological tissue has optical characteristics of strong absorption and strong scattering, which causes low image signal-to-noise ratio and contrast, and the image is subjected to image processing

The processing and analysis are very influential. Few studies facing transmission images have been developed from the registration direction. The in-depth research on the registration algorithm of the biological tissue transmission image has a market prospect to be urgently exploited, and for example, the method and the support can be provided for the early diagnosis of the breast tumor.

Image registration methods can be divided into classical traditional methods and emerging neural network methods. Classical traditional methods are further classified into registration methods based on gray information, registration methods based on transform domains, and registration methods based on feature points. Among them, the feature point-based registration method is the most widely applied registration method among the conventional methods. Such as scale invariant feature transform method (SIFT), speeded up robust feature transform method (SURF), binary robust invariant scalable keypoint method (BRISK), and oriented fast rotation algorithm (ORB). However, the signal-to-noise ratio and the contrast of the transmission image of the biological tissue are low due to the characteristics of strong scattering and strong absorption of the biological tissue, so that the number of key points detected by the method is rare, and the registration accuracy is greatly adversely affected. The development of the neural network provides a new direction for the field of image registration. The proposed methods like STN, DIRNet etc. enable unsupervised image registration. However, the method does not relate to the field of strong absorption and strong scattering biological tissue transmission images for early diagnosis of breast tumors, and the registration method also occupies the storage space of the model.

Disclosure of Invention

The invention aims to provide a biological tissue transmission image registration system based on mutual convolution.

The system carries out deep research on the registration algorithm of the biological tissue transmission image, has a market prospect which needs to be developed urgently, and can provide support and a method for early diagnosis of breast tumor.

In order to solve the technical problem, the system for registering the transmission images of the biological tissues based on the mutual convolution comprises a regressor, a space converter and a sampler based on the mutual convolution. After the floating transmission image and the fixed transmission image are input into the system, through dimension linkage, the transmission image after registration is finally obtained through a regressor, a space converter and a sampler based on mutual convolution. The system can be updated through training, and the registration effect is improved.

The detailed system workflow comprises the following steps:

1) the transmission image to be registered pair is input to the system.

2) And the input floating transmission image and the fixed transmission image are linked along the dimension direction and enter a regressor based on mutual convolution to obtain a deformation parameter.

3) And the deformation parameters are continuously sent downwards to a space converter for processing to obtain a space deformation field.

4) And the sampler resamples the floating transmission image by taking the deformation field obtained in the previous link as a basis and outputs a transmission image after registration.

The system updating method comprises the following steps:

1) the prepared transmission image registration dataset is input to the system.

2) The input floating transmission image and the fixed transmission image are linked in batches along the dimension direction and enter a regressor based on mutual convolution to obtain deformation parameters.

3) And the deformation parameters are continuously sent downwards to a space converter for processing to obtain a space deformation field.

4) And the sampler resamples the floating transmission image based on the deformation field obtained in the previous step and outputs the registered transmission image.

5) And (3) penalizing appearance difference and local space change through a loss function, adaptively optimizing system parameters, and updating the system.

6) And repeating 2) -5) until the registration error is reduced to a required range or the iteration number reaches an upper limit. At this point training is complete and the system is updated.

In one embodiment of the deconvolution-based biological tissue transmission image registration system, the cell structure in the deconvolution-based regressor includes convolution and deconvolution. The mutual convolution is a novel asymmetric convolution structure, and the obtained benefit is that the information between images is more fully and reasonably utilized, and the occupied space of a system model is reduced.

In one embodiment of the deconvolution-based biological tissue transmission image registration system, the frame in the deconvolution-based regressor is a network of an inclusion module and a U-net module. The obtained gains are that the depth and the width of the regression network are effectively expanded, the appropriate deformation parameters can be learned more conveniently, and the registration effect is improved.

In one embodiment of the system for registering transmission images of biological tissues based on mutual convolution, the system can be trained and updated adaptively through the data set of the transmission images of the biological tissues without labels. The obtained benefits are that the labeling cost in the data set manufacturing is reduced, the manpower is liberated, the system can complete updating by itself, and the registration capability of the biological tissue transmission image is trained.

In an implementation mode of a mutual convolution-based biological tissue transmission image registration system, an image registration effect is improved end to end and the storage space occupation of the system is reduced for a biological tissue transmission image with sparse key points and low signal to noise ratio. The obtained gains are that the registration technology in the field of biological tissue transmission images is deeply explored, and support is provided for the research of biological tissue transmission image related technologies with low signal-to-noise ratio and low contrast, such as early diagnosis of breast tumors.

In one embodiment of the system for registering transmission images of biological tissues based on mutual convolution, the loss function comprises a penalty for appearance difference between the registered transmission image and the fixed transmission image and a penalty for local spatial variation of the deformation field. The obtained gains are that the smoothness of the deformation field is constrained while paying attention to the structural similarity between the registration result graph and the fixed image, and the registration effect is ensured.

The invention takes a correlation algorithm as a core, and can still obtain an effective registration result through a system consisting of a regressor, a space converter and a sampler based on mutual convolution under the conditions of low signal-to-noise ratio, low contrast and rare detectable key points of the biological tissue transmission images, and effectively reduce and reduce the space occupation.

The invention can face to the biological tissue transmission image, carries out end-to-end image registration, and the registered image can well reserve the details of the floating transmission image, reduce the structural difference with the fixed image, has better smoothness and simultaneously reduces the storage space occupied by the model.

Drawings

FIG. 1 is a block diagram of a system for registering transmission images of biological tissues according to the present invention;

FIG. 2 is an example of a transmission image of a phantom of floating breast tissue;

FIG. 3 is an example of a phantom transmission image of fixed breast tissue;

FIG. 4 is an image of the results of an experiment conducted by the method of the present invention;

FIG. 5 is an image of the experimental results of the SURF method;

table 1 shows comparison of image evaluation indexes.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

The invention explores a registration algorithm for biological tissue transmission images with low signal-to-noise ratio and low contrast, and provides support for the research of early diagnosis of breast tumors and other related technologies relating to the biological tissue transmission images. The invention relates to a biological tissue transmission image registration system based on mutual convolution, which comprises a regressor based on mutual convolution, a space converter and a sampler. After the floating transmission image and the fixed transmission image are input into the system, through dimension linkage, the transmission image after registration is finally obtained through a regressor, a space converter and a sampler based on mutual convolution. The system can be updated through training, and the registration effect is improved. The method comprises the following specific steps:

A system for registering transmission images of biological tissues based on mutual convolution, comprising: the system comprises a regressor, a space converter and a sampler based on mutual convolution, wherein after the floating transmission image and the fixed transmission image are input into the system, the floating transmission image and the fixed transmission image are subjected to dimension linkage and then sequentially pass through the regressor, the space converter and the sampler based on mutual convolution to finally obtain a transmission image after registration; the image registration system is updated through training, and the registration effect is improved; the detailed work flow comprises the following steps:

in the first step, the transmission image to be registered pair is input to the system. The transmission image pair to be registered can be input in batch, and the registered images are output in batch after being processed by the system.

And secondly, linking the input floating transmission image and the fixed transmission image along the dimension direction, and entering a regressor based on mutual convolution to obtain a deformation parameter.

The unit structure in the regression device based on the mutual convolution comprises convolution and mutual convolution. Unlike conventional convolution, mutual convolution is a novel asymmetric convolution structure, as inIs used on one channelEach size isIs operated with a parameter ofWhereas the conventional convolution requires One parameter, almost half more than the mutual convolution. The mutual convolution can more fully and reasonably utilize the information between the images, and the occupied space of a system model is reduced.

A framework in the regression device based on the mutual convolution is a network formed by embedding an inclusion module and a U-net module. The depth and the width of the regression network are effectively expanded, the appropriate deformation parameters can be learned more conveniently, and the registration effect is improved.

And thirdly, continuously sending the deformation parameters downwards to a space converter for processing, and obtaining a space deformation field after the transformation of the space converter, namely obtaining a displacement relation field of the registered image and the floating image.

And fourthly, resampling the floating transmission image by using the deformation field obtained in the previous link of the sampler as a basis, putting pixel points in the sampled floating image into coordinate positions corresponding to the registration result image, and outputting the recombined image from the system to obtain the registration result image.

The system updating method comprises the following steps:

in a first step, a prepared registration dataset of transmission images is input to the system. The data set employed in this example was made from a simulated biological breast tissue phantom image manually collected to fit further study in a follow-up breast tumor self-screening handheld device. The data set acquisition platform is built through an acrylic plate, milk, potato slices and pork tissue slices, and images are acquired through a handheld common mobile phone. Wherein, aiming at the characteristics of biological tissues, the normal tissue fluid of biological mammary glands is simulated by milk, and heterogeneities with different pathological changes are simulated by potato slices and pork tissue slices. The acquired images were scaled down to 144 x 256 resolution and converted to a grayscale map, producing a data set containing 700 phantom images. The built platform simulates the strong absorption and strong scattering characteristics of mammary tissue, images acquired by the equipment have strong fuzziness, and small-amplitude translation, rotation and other transformations exist among the images.

And secondly, linking the input floating transmission image and the fixed transmission image in batches along the dimension direction, and entering a regressor based on mutual convolution to obtain a deformation parameter. The unit structure in the regression device based on the mutual convolution comprises convolution and mutual convolution, and the framework is a network formed by embedding an inclusion module and a U-net module.

And thirdly, continuously sending the deformation parameters downwards to a space converter for processing to obtain a space deformation field.

And fourthly, resampling the floating transmission image by using the deformation field obtained in the previous step as a basis and outputting the registered transmission image.

And fifthly, punishing the appearance difference and the local space change through a loss function, adaptively optimizing system parameters, and updating the system.

And the loss function comprises a penalty for appearance difference between the registered transmission image and the fixed transmission image and a penalty for local spatial change of the deformation field. The loss function is formulated as follows:

in the formula, f and m represent the fixed transmission image and the floating transmission image, respectively, represent the transformation field,the image to be registered represents the transformed image to be registered, is used for measuring the similarity of the fixed image and the transformed image, is a regularization item, and restrains the smooth deformation of a space. The method restrains the smoothness of the deformation field while paying attention to the structural similarity between the fixed image and the registration result image, and ensures the registration effect.

And sixthly, repeating the second step to the fifth step until the registration error is reduced to a required range or the iteration number reaches an upper limit and stopping. In this embodiment, the training is stopped when the number of iterations reaches 1200, and when the training is completed, the network parameters are optimized, and the system is updated.

And after the system training is finished, inputting the mammary tissue phantom transmission image which does not participate in the training as a floating image into the system for testing, outputting the registered image by the system, and finishing the registration. Fig. 2 and 3 are a floating transmission image and a fixed transmission image for testing, respectively, and the system output registration result is shown in fig. 4. In contrast, the present embodiment also performs registration by using SURF method, and obtains the registration result, as shown in fig. 5. The SURF method obtains the transformed image through the registration of the limited key points, but the longitudinal stretching of the image is large, so that the form of the simulated alloplasm is not consistent with that of the fixed image.

In order to objectively describe the registration effect of the invention, the registration effect is evaluated by adopting various evaluation indexes, and the evaluation results are shown in table 1: TABLE 1

The data in the table show that the peak signal-to-noise ratio of the method result graph is improved by nearly one time compared with the floating transmission image, which is about 2.5 times of the SURF method result graph, in the aspect of structural similarity measurement, the method result graph is 14.6 percent higher than the SURF method result graph, and in the mutual information index, the effect of the method result graph is also obviously better than that of the initial floating transmission image and the SURF method result graph.

And (4) conclusion: the invention is oriented to the field of biological tissue transmission images with low signal-to-noise ratio and low contrast, deeply explores a transmission image registration method, and realizes a biological tissue transmission image registration system based on mutual convolution. The system comprises a regressor, a space converter and a sampler based on mutual convolution, can utilize asymmetric field of view information, reduces the model space, overcomes the adverse effect of the sparse detectable key points of the transmission image on registration, and ensures the registration effect. Meanwhile, support is provided for the research of the early diagnosis of the breast tumor and other related technologies related to the biological tissue transmission image, and the breast tumor self-screening handheld device can be adapted to the further research of the follow-up breast tumor self-screening handheld device in the future.