阅读说明：本技术 检索病理图像的系统及方法 (System and method for searching pathological image ) 是由 郭兑荣 李尚勋 金善禹 于 2019-10-04 设计创作，主要内容包括：为了帮助诊断者参照现有的诊断记录,从生物组织的图像准确地诊断疾病,公开了一种生物图像诊断系统及其方法。根据本发明的一个方面,所提供的生物图像诊断系统包括：从存储已诊断的多个生物图像和所述已诊断的多个生物图像的诊断结果的DB检索至少一个具有与预设诊断对象生物图像相似特征的类似生物图像的第1检索模块、从所述DB检索由预设诊断者诊断出的至少一个已诊断生物图像的第2检索模块、将所述至少一个类似生物图像和所述至少一个已诊断生物图像传送至所述诊断者终端的通信模块以及从所述诊断者终端接收到的所述诊断对象生物图像的疾病诊断结果存储于所述DB的储存模块。所述诊断者的终端显示所述诊断对象生物图像、所述至少一个类似生物图像和所述至少一个已诊断生物图像。(In order to help a diagnostician accurately diagnose a disease from an image of a biological tissue with reference to an existing diagnosis record, a biological image diagnosis system and a method thereof are disclosed. According to an aspect of the present invention, there is provided a biological image diagnostic system including: a 1 st retrieval module retrieving at least one similar biological image having a similar characteristic to a preset diagnosis object biological image from a DB storing a plurality of diagnosed biological images and diagnosis results of the plurality of diagnosed biological images, a 2 nd retrieval module retrieving at least one diagnosed biological image diagnosed by a preset diagnostician from the DB, a communication module transmitting the at least one similar biological image and the at least one diagnosed biological image to the diagnostician terminal, and a storage module storing disease diagnosis results of the diagnosis object biological image received from the diagnostician terminal in the DB. The diagnostician's terminal displays the diagnosis object biological image, the at least one similar biological image, and the at least one diagnosed biological image.)

1. A system for retrieving a pathology image, comprising: an automatic encoder including an encoder portion for extracting a feature of an original pathological image by accepting an input of the original pathological image, and a decoder portion for generating a restored pathological image corresponding to the original pathological image by inputting the feature of the original pathological image extracted by the encoder portion; a neural network for diagnosis that receives an input of a restored pathological image generated by the automatic encoder to which the original pathological image is input, and outputs a set disease diagnosis result; and

a learning module for learning the automatic encoder and the neural network for diagnosis after inputting a plurality of pathological images for training respectively marked with diagnosis results to the automatic encoder,

the automatic encoder reflects the diagnosis result of the restored pathology image output by the neural network for diagnosis and learns.

2. The system of claim 1, wherein the loss function of said auto-encoder comprises a difference between said original pathology image and said restored pathology image, and a difference between a label of said original pathology image and a diagnosis result of said restored pathology image outputted through said diagnostic neural network.

3. The system according to claim 1, wherein said system for retrieving a pathological image further comprises a feature generation module for inputting said pathological image for retrieval into said learned automatic encoder for each of a plurality of pathological images for retrieval, then generating features of said pathological image for retrieval by said encoder section, and storing the generated features of said pathological image for retrieval in a DB.

4. The system according to claim 3, wherein said system for retrieving a pathology image further comprises a retrieval module for inputting a suspected pathology image in said learned automatic encoder, generating a feature of said suspected pathology image by said encoder section, and retrieving a feature of a similar pathology image similar to said suspected pathology image in said DB based on said generated feature of said suspected pathology image.

5. The system of claim 1, wherein the disease is prostate cancer.

6. The system according to claim 1, wherein the learning module further inputs a plurality of additional pathological images for training, each of which is labeled with a diagnosis result, into the neural network for diagnosis, and learns the neural network for diagnosis.

7. A system for retrieving a pathology image, comprising: an encoder portion of a learned auto encoder that extracts features of an original pathology image by accepting input of the original pathology image;

a DB constructed to include the features of each of the plurality of pathological images for retrieval; and

a retrieval module for inputting the suspicious pathological image into the automatic encoder, generating the characteristic of the suspicious pathological image by the encoder part, retrieving the similar pathological image characteristic similar to the suspicious pathological image in the DB according to the generated characteristic of the suspicious pathological image,

the automatic encoder learns by the system recited in claim 3, and the DB is constructed by the system recited in claim 3.

8. A system for retrieving a pathology image, comprising: an automatic encoder including an encoder portion for extracting a feature of an original pathological image by accepting an input of the original pathological image, and a decoder portion for generating a restored pathological image corresponding to the original pathological image by inputting the feature of the original pathological image extracted by the encoder portion; a neural network for diagnosis that receives an input of a restored pathological image generated by the automatic encoder to which the original pathological image is input, and outputs a preset disease diagnosis result; and a learning module that inputs the plurality of training pathology images, each of which is labeled with a diagnosis result, into the automatic encoder, and learns the automatic encoder, and inputs the plurality of additional training pathology images, each of which is labeled with a diagnosis result, into the diagnostic neural network, and learns the diagnostic neural network, wherein the automatic encoder reflects a diagnosis result of the restored pathology image output from the diagnostic neural network, and learns the diagnosis result.

9. A method for retrieving pathological images is characterized in that: an automatic encoder including an encoder portion for extracting a feature of an original pathological image by accepting an input of the original pathological image, and a decoder portion for generating a restored pathological image corresponding to the original pathological image by inputting the feature of the original pathological image extracted by the encoder portion;

and a diagnostic neural network for receiving an input of a restored pathological image generated by the automatic encoder to which the original pathological image is input and outputting a preset disease diagnosis result,

the method for performing a pathological image search by this system includes a learning step of learning the automatic encoder and the diagnostic neural network after a plurality of pathological images for training respectively labeled with diagnosis results are input to the automatic encoder,

the automatic encoder reflects the diagnosis result of the restored pathology image output by the neural network for diagnosis and learns.

10. The method as claimed in claim 9, wherein the loss function of the automatic encoder includes a difference between the original pathology image and the restored pathology image, and a difference between a label of the original pathology image and a diagnosis result of the restored pathology image outputted through the neural network for diagnosis.

11. The method according to claim 9, wherein said pathological image retrieval method further comprises a feature generation step of inputting a plurality of retrieval pathological images in said learned automatic encoder for each of said plurality of retrieval pathological images, then generating features of said retrieval pathological images by said encoder section, and storing the generated features of said retrieval images in a DB.

12. The method according to claim 11, wherein the method for retrieving a pathological image further comprises a retrieving step of inputting a suspected pathological image into the learned automatic encoder, then generating features of the suspected pathological image by the encoder part, and retrieving similar pathological image features similar to the suspected pathological image in the DB based on the generated features of the suspected pathological image.

13. The method of claim 10, wherein the learning step further comprises; and a step of inputting a plurality of additional training pathological images each of which is labeled with a diagnosis result to the neural network for diagnosis, and learning the neural network for diagnosis.

14. A method for searching pathological images is characterized in that; an encoder section including a learned automatic encoder that extracts a feature of an original pathology image after the original pathology image is input; and

in order to provide a system including a plurality of DBs constructed by searching for the characteristics of each pathological image,

the method for retrieving pathological images performed by this system includes: inputting a suspicious pathological image into the automatic encoder, and generating characteristics of the suspicious pathological image through the encoder part;

and a step of retrieving, in the DB, features of similar pathology images similar to the suspicious pathology image based on the generated features of the suspicious pathology image,

the automatic encoder learns by the method described in claim 10, and the DB is constructed by the method described in claim 10.

15. A method for retrieving pathological images is characterized in that: an automatic encoder including an encoder portion for extracting a feature of an original pathological image after an original pathological image is inputted thereto, and a decoder portion for generating a restored pathological image corresponding to the original pathological image after the feature of the original pathological image extracted by the encoder portion is inputted thereto; and a diagnostic neural network system for outputting a preset disease diagnosis result after restoring the pathological image generated by the automatic encoder which inputs the original pathological image. The method for retrieving pathological images by the system comprises; a step of inputting a plurality of pathological images for training, each of which is labeled with a diagnosis result, into the automatic encoder and then learning the automatic encoder; and a step of learning the diagnostic neural network after inputting a plurality of additional pathological images for training respectively labeled with the diagnostic result into the diagnostic neural network,

the automatic encoder performs learning by reflecting the diagnosis result of the restored pathology image outputted from the neural network for diagnosis.

16. A computer program recorded in a medium, installed in a data processing apparatus, to perform the method recited in any one of claims 9 to 15.

Technical Field

The invention relates to a system and a method for retrieving pathological images. In particular to a system and a method for searching other similar pathological images from the aspect of effective characteristics in disease diagnosis by utilizing an automatic encoder which reflects the effective characteristics in the disease diagnosis based on pathological images in learning.

Background

One of the major tasks performed by a pathology or pathology department is to read a biological image of a patient (e.g., a slide of the patient's biological tissue) to determine the status or signs of a particular disease. Such diagnosis is a way that relies on the experience and knowledge of long-term skilled medical personnel. A recent trend is an increasing manner of reading slide images generated by digital imaging instead of slides of biological tissues.

On the other hand, with the recent development of machine learning, there has been an active attempt to automate work such as recognition and classification of images by a computer system. A typical example is a deep learning approach that is attempting to use machine learning (e.g., using Convolutional Neural Network (CNN)), which automates diagnosis originally performed by skilled medical personnel, and performs image-based disease diagnosis by deep learning using Neural Network (e.g., CNN). In addition, when a diagnostician performs a disease diagnosis based on an image, an auxiliary means such as a technique for searching for an image having a similar feature to the image is also useful for the disease diagnosis.

On the other hand, an automatic encoder (Autoencoder) which is one of background art of the present invention will be explained. The automatic encoder is a neural network structure mainly used in an unsupervised learning method, and is used for unsupervised learning of high-efficiency data coding. The auto-encoder learns a function that brings the output value close to the input value, performs feature extraction on the input data by the encoder, and reconstructs the original data by the decoder.

Fig. 1 is a schematic diagram of an automatic encoder configuration capable of inputting an image. Referring to fig. 1, an automatic encoder 1 may include an encoder portion 2 including a convolutional layer and a decoder portion 3 including an anti-convolutional layer. After an original image (x) is input from the encoder 1, the original image (x) is encoded in the encoder section 2, thereby generating a feature (z ═ e (x)) of the original image (x). The generated features (z) may be decoded in the decoder portion 3, thereby generating a restored image (x' ═ d (z)) corresponding to the original image (x).

The automatic encoder is also a kind of neural network, and therefore, it is also learned through a large amount of training data, and in the learning stage of the automatic encoder, the following process is performed for each learning data x:

1) the learning data x is input to an automatic encoder, and is subjected to encoding and decoding processes to generate restored data x corresponding to the learning data x.

2) The difference between the learned data x and the restored data x ', i.e., the error e ═ L (x, x') (L is a loss function), is calculated.

3) The weights in the autoencoder are updated according to the error back propagation method.

Disclosure of Invention

Technical subject

The technical problem to be achieved by the present invention is to provide a system and a method for searching a pathological image, which can search other pathological images having similar characteristics to a specific pathological image by using an automatic encoder.

In addition, a system and a method for searching pathological images are provided, which can reflect the effective characteristics of disease diagnosis based on pathological images in the learning of an automatic encoder, and search other pathological images with similar characteristics effective for disease diagnosis by using the automatic encoder learned in this way.

Means for solving the problems

According to an aspect of the present invention, there is provided a system for retrieving a pathology image, including: an automatic encoder including an encoder portion that extracts a feature of an original pathology image upon accepting an input of the original pathology image, and a decoder portion that generates a restored pathology image corresponding to the original pathology image by accepting a feature input of the original pathology image extracted by the encoder portion; a learning module for learning the automatic encoder and the neural network for diagnosis by receiving an input of a restored pathological image generated by the automatic encoder to which the original pathological image is input, and inputting a plurality of training pathological images to which a neural network for diagnosis and a diagnosis result for outputting a preset disease diagnosis result are respectively labeled to the automatic encoder, wherein the automatic encoder learns after reflecting the diagnosis result of the restored pathological image output by the neural network for diagnosis.

In one embodiment, the loss function whose characteristic may be the automatic encoder may be defined by a difference between the original pathological image and the restored pathological image, and a difference between a label of the original pathological image and a diagnosis result of the restored pathological image output by the neural network for diagnosis.

In one embodiment, the system for retrieving a pathological image may further include a feature generation module which inputs the pathological image for retrieval into the learned automatic encoder for each of a plurality of pathological images for retrieval, and then generates features of the pathological image for retrieval by the encoder part, and stores the generated features of the pathological image for retrieval in a DB.

In one embodiment, the system for retrieving a pathology image may further include a retrieval module that inputs a suspicious pathology image in the learned auto encoder, generates a feature of the suspicious pathology image through the encoder part, and retrieves a feature of a similar pathology image similar to the suspicious pathology image in the DB according to the generated feature of the suspicious pathology image.

According to one embodiment of the invention, the disease may be prostate cancer.

According to another aspect of the present invention, there is provided a system for retrieving a pathological image, comprising: an encoder portion of a learned auto encoder that extracts features of an original pathology image by accepting input of the original pathology image; a DB constructed by including the characteristics of each of the plurality of pathological images for search; and a retrieval module for inputting the suspicious pathological image into the automatic encoder, generating the characteristic of the suspicious pathological image through the encoder part, and retrieving similar pathological image characteristics similar to the suspicious pathological image in the DB according to the generated characteristic of the suspicious pathological image. The automatic encoder learns through the system, and the DB is constructed through the system.

According to another aspect of the present invention, there is provided a system for retrieving a pathological image, comprising: an automatic encoder including an encoder portion for extracting a feature of an original pathological image by accepting an input of the original pathological image, and a decoder portion for generating a restored pathological image corresponding to the original pathological image by inputting the feature of the original pathological image extracted by the encoder portion; a neural network for diagnosis that receives an input of a restored pathological image generated by the automatic encoder to which the original pathological image is input, and outputs a set disease diagnosis result; and a learning module that inputs the plurality of training pathological images, each of which is labeled with a diagnosis result, into the automatic encoder, and learns the automatic encoder, and inputs the plurality of additional training pathological images, each of which is labeled with a diagnosis result, into the diagnostic neural network, and learns the diagnostic neural network. The automatic encoder reflects the diagnosis result of the restored pathology image output by the neural network for diagnosis and learns.

According to another aspect of the present invention, there is provided a method of retrieving a pathology image, including an automatic encoder for generating a restored pathology image corresponding to an original pathology image by receiving an input of the original pathology image, an encoder part for extracting features of the original pathology image, and a decoder part for inputting the features of the original pathology image extracted by the encoder part; a method for executing a pathological image retrieval by a system for receiving an input of a restored pathological image generated by an automatic encoder to which an original pathological image is input and outputting a diagnostic neural network in which a disease diagnosis result is preset, the method comprising a learning step of learning the automatic encoder and the diagnostic neural network after a plurality of training pathological images each having a diagnosis result marked are input to the automatic encoder, the automatic encoder learning by reflecting the diagnosis result of the restored pathological image output by the diagnostic neural network.

In one embodiment, the loss function characterized by said automatic encoder may be defined by the difference between said original pathology image and said restored pathology image, and the difference between the label of said original pathology image and the diagnosis result of said restored pathology image output through said diagnostic neural network.

In one embodiment, the method of pathological image retrieval may further include a feature generation step of inputting the retrieval pathological image in the learned automatic encoder for each of a plurality of retrieval pathological images, then generating features of the retrieval pathological image by the encoder portion, and storing the generated features of the retrieval image in a DB.

In one embodiment, the method of pathology image retrieval may further include a retrieving step of inputting a suspected pathology image into the learned automatic encoder, then generating a feature of the suspected pathology image by the encoder portion, and retrieving a similar pathology image feature similar to the suspected pathology image in the DB according to the generated feature of the suspected pathology image.

According to another aspect of the present invention, there is provided a method of retrieving a pathology image, comprising: a system including an encoder portion of a learned automatic encoder that extracts features of an original pathological image after the original pathological image is input, and a DB constructed to contain respective features of a plurality of pathological images for retrieval, the method for performing pathological image retrieval by the system comprising: inputting a suspicious pathological image into the automatic encoder, and generating characteristics of the suspicious pathological image through the encoder part; and a step of retrieving, in the DB, features of similar pathology images similar to the suspicious pathology image based on the generated features of the suspicious pathology image. The automatic encoder learns by the method by which the DB is constructed.

According to another aspect of the present invention, there is provided a method of retrieving a pathology image, comprising: a system including an automatic encoder for generating a restored pathological image corresponding to an original pathological image by receiving an input of the original pathological image, an encoder section for extracting a feature of the original pathological image, and a decoder section for generating a restored pathological image corresponding to the original pathological image by inputting the feature of the original pathological image extracted by the encoder section, a neural network for diagnosis for outputting a preset disease diagnosis result by the restored pathological image generated by the automatic encoder for inputting the original pathological image, a method for performing a pathological image retrieval by the system including: a step of inputting a plurality of training pathological images each of which is labeled with a diagnosis result into the automatic encoder and then learning the automatic encoder, and a step of inputting a plurality of additional training pathological images each of which is labeled with a diagnosis result into the diagnostic neural network and then learning the diagnostic neural network. The automatic encoder performs learning by reflecting the diagnosis result of the restored pathology image outputted from the neural network for diagnosis.

According to another aspect of the present invention, there is provided a computer program recorded on a medium, installed in a data processing apparatus, to perform the above method.

ADVANTAGEOUS EFFECTS OF INVENTION

According to the technical idea of the present invention, a system and method for retrieving a pathological image are provided, which can retrieve other pathological images having similar characteristics to a specific pathological image using an automatic encoder.

Further, it is possible to provide a system and a method for retrieving a pathological image, which can reflect an effective feature of a disease diagnosis based on a pathological image in the learning of an automatic encoder, and by using the thus-learned automatic encoder, can retrieve other pathological images similar in feature, which are effective for the disease diagnosis.

Drawings

For a better understanding of the drawings referred to in the detailed description of the invention, a brief description of each figure is provided.

Fig. 1 is a schematic diagram of the structure of an automatic encoder.

Fig. 2 is a schematic view of an operating environment of a method of retrieving a pathological image according to the technical idea of the present invention.

Fig. 3 is a schematic diagram of a simple structure of a system for retrieving a pathological image according to an embodiment of the present invention.

Fig. 4a to 4c are schematic diagrams respectively illustrating a method for learning an automatic encoder in a system for retrieving a pathological image according to different embodiments of the present invention.

Fig. 5 is a schematic diagram of a process of constructing a DB after extracting features of an image for retrieval in a system for retrieving a pathological image according to an embodiment of the present invention.

Fig. 6 is an explanatory diagram of retrieving a pathological image in a system for retrieving a pathological image according to an embodiment of the present invention.

Detailed Description

While the invention is susceptible to various modifications and alternative embodiments, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. However, the present invention is not limited to the specific embodiments, and all modifications, equivalents, and alternatives falling within the spirit and scope of the present invention are to be understood. In describing the present invention, detailed description will be omitted when it is considered that specific description of known techniques may rather obscure the gist of the present invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another component.

The terminology used in the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular references include plural references unless the context clearly dictates otherwise.

The terms "comprising" or "having" in the specification mean the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and it is understood that the presence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof is not previously excluded.

In addition, in this specification, when one component "transfers" data to another component, it means that the component may directly transfer the data to the other component or may transfer the data to the other component through at least one other component. Conversely, if any one component "transfers" data directly to another component, it means that the data will be transferred from that component to the other component, and not through the other component.

The present invention will be described in detail below with reference to the attached drawings, which surround embodiments of the present invention. Like reference symbols in each of the figures indicate like elements.

Fig. 2 is a schematic diagram of an operating environment of a method of retrieving a pathology image according to the technical idea of the present invention. Referring to fig. 2, a method of retrieving a pathology image according to the technical idea of the present invention may be performed by the system 100 of retrieving a pathology image.

In the technical idea of the present invention, the system 100 for retrieving a pathology image may be installed on the preset server 10, implementing the technical idea of the present invention. The server 10 refers to a data processing apparatus having a computing power for implementing the technical idea of the present invention, and any one of the data processing apparatus connected to a client (terminal; 20 to 20-1) through a network and a personal computer, a mobile terminal, etc. for performing a specific service may be generally defined as a server, for which a general expert in the technical field of the present invention can easily infer.

The server 10 may include a processor and a storage device. The processor may refer to an arithmetic device capable of driving a program that realizes the technical idea of the present invention, and may perform a pathological image retrieval using the program and an automatic encoder defined based on the technical idea of the present invention. The storage device may be a program and a data storage means capable of storing various data required for realizing the technical idea of the present invention, or may be implemented using various storage means according to the embodiment. The storage device may refer not only to a main storage device included in the server 10 but also to a temporary storage device or a memory included in the processor.

The pathological image search system 100 is implemented by any physical device in fig. 2, but if necessary, a plurality of physical devices may be organically combined together to implement the diagnosis system 100 according to the technical idea of the present invention, and it is easy for a general expert in the technical field of the present invention to deduce this.

Fig. 3 is a schematic diagram of a simple structure of the system 100 for retrieving a pathological image according to an embodiment of the present invention.

Referring to fig. 2, the system 100 may include an auto-encoder 110, a diagnostic neural network 120, a learning module 130, a feature generation module 140, and a retrieval module 150. Some of the components may not necessarily correspond to the necessary components necessary to implement the present invention according to an embodiment of the present invention, and the diagnostic system 100 may of course contain more components according to an embodiment. For example, the system 100 may further include a control module (not shown) to control the functions and/or resources of other configurations of the system 100 (e.g., the learning module 130, the feature generation module 140, the retrieval module 150, etc.). In addition, according to an embodiment, the system 100 may further include a Database (Database; DB; 200) for storing various information and/or data required for implementing the technical idea of the present invention. Additionally, according to embodiments, the system 100 may not include the learning module 130 and/or the feature generation module 140.

The system 100 may refer to a logical configuration having hardware resources (resources) and/or software required for implementing the technical idea of the present invention, and does not necessarily refer to a physical component or a device. That is, the system 100 may refer to a logical combination of hardware and/or software equipped to implement the technical idea of the present invention, and may be installed on devices spaced apart from each other as necessary to perform respective functions, thereby implementing a logical configuration set of the technical idea of the present invention. Further, the system 100 may also refer to a collection of components that are separately implemented per function or role for implementing the technical idea of the present invention. For example, the automatic encoder 110, the diagnostic neural network 120, the learning module 130, the feature generation module 140, and the search module 150 may be located on different physical devices or may be located on the same physical device. Further, according to the embodiment, each combination of software and/or hardware constituting the automatic encoder 110, the neural network for diagnosis 120, the learning module 130, the feature generation module 140, and the retrieval module 150 may also be located on different physical devices, and components located on different physical devices may be organically combined with each other and implement the modules, respectively.

Further, the modules in the present specification may refer to a functional and structural combination of hardware for performing the technical idea of the present invention and software for driving the hardware. For example, the module may refer to a logical unit of a preset code and a hardware resource (resource) for executing the preset code, and does not necessarily refer to a physically connected code or a hardware, which can be easily inferred by a general expert in the technical field of the present invention.

The DB200 may store a plurality of pathology images. The pathological image may be a variety of biological images such as a tissue image.

According to an embodiment, the DB200 may include a plurality of pathology images for training for learning the automatic encoder 110 and the neural network for diagnosis 120, which will be described later. In addition, according to an embodiment, the DB200 may include a plurality of pathology images for retrieval. The diagnostic results may be labeled in advance on the pathological images for training, and the labeled diagnostic results may be associated with the corresponding pathological data for training and stored in the DB 200.

The auto-encoder 110 may include an encoder portion 111 and a decoder portion 112.

The encoder section 111 may extract features of the original pathology image after accepting input of the original pathology image.

The decoder portion 112 may accept input of the features of the original pathological image extracted by the encoder portion 111, and generate a restored pathological image corresponding to the original pathological image.

The diagnostic neural network 120 may be an image-based neural network for diagnosing a disease. In particular, the diagnostic neural network 120 may input a restored pathology image generated by the automatic encoder to which the original pathology image is input, and then output a diagnosis result of a preset disease.

For example, the disease may be prostate cancer, and the following description will be made with reference to prostate cancer, but the technical idea of the present invention is not limited to prostate cancer, and it will be easily understood by those of ordinary skill in the art to which the present invention pertains.

"performing diagnosis" in the present specification may mean making a judgment about a specific disease based on a slide of a biological image expressed by a biological tissue or a part thereof (e.g., an image block or an image sub-block). Therefore, the diagnosis result on the biological image may include not only the expression or non-expression of the specific disease but also the degree of development (or a probability corresponding to the degree of development) of the specific disease. For example, when the technical idea of the present invention is applied to diagnosis of prostate cancer, the diagnosis result may include "Gleason Pattern" or "Gleason Score" indicating the degree of progression of prostate cancer. For example, a Gleason score of 2 to 10, typically 6 to 10, can be judged as cancerous, with the greater the number, the more severe the extent of prostate cancer expression. The grignard patterns can be classified into classes 1 to 5. Alternatively, the diagnostic result may include the site of disease expression.

According to an embodiment, the diagnostic result output by the diagnostic neural network 120 may be varied. For example, the diagnostic neural network 120 may perform classification to determine the presence or absence of certain diseases (e.g., prostate cancer). In addition or in accordance with an embodiment, the diagnostic neural network 120 may perform a variety of classifications or regression estimations to determine the severity of a particular disease (e.g., prostate cancer). In addition, the diagnostic neural network 120 may also perform multi-level classification, regression estimation, or semantic-based image segmentation to detect the location of a lesion of a particular disease.

In this specification, a neural network may refer to a collection of information expressing a series of design matters defining the neural network. In this specification, the neural network may be a convolutional neural network.

The convolutional neural network may comprise an input layer, a plurality of hidden layers, and an output layer, as is well known. Each of the plurality of hidden layers may include a convolutional layer and a pooling layer (or sub-sampling layer).

The convolutional neural network may define the various layers by functions, filters, convolution step sizes (stride), weight factors, etc. Further, the output layer may be defined as a fully connected feed forward layer.

The design considerations for the various layers that make up a convolutional neural network are well known. For example, known functions may be used for each of the number of layers included in the plurality of layers, the convolution function, the pool function, and the excitation function that define the plurality of layers, or each function separately defined to realize the technical idea of the present invention may be used.

For example, the convolution function may be a discrete convolution sum or the like. The pool function may be max-pooling, mean-pooling, or the like. The excitation function may be sigmoid, tanh, RELu (Rectified Linear unit), etc.

Having defined the design considerations for these convolutional neural networks, the convolutional neural networks defining the design considerations may be stored on a storage device. And, when the convolutional neural network is learned, the weighting factor corresponding to each layer may be specific.

That is, learning of the convolutional neural network may refer to a process determined by the weight factors of the respective layers. Also, when the convolutional neural network is learned, the learned convolutional neural network may accept input of input data on an input layer and output data through an output layer defined in advance.

The neural network to which embodiments of the present invention relate may be defined by selecting any one or more of the well-known design considerations, or may define separate design considerations for the neural network.

The learning module 130 may input a plurality of pathological images for training, which are respectively labeled with diagnosis results, into the automatic encoder, and learn the automatic encoder and the neural network for diagnosis.

In particular, the feature of the automatic encoder may be a diagnosis result reflecting the restored pathology image output by the neural network for diagnosis and learned.

In one embodiment, a loss function (loss function) of the automatic encoder 110 may be defined to include a difference between the original pathological image and the restored pathological image, and a difference between a label of the original pathological image and a diagnosis result of the restored pathological image output by the neural network for diagnosis.

For example, the loss function of the auto-encoder 110 may be in the form of the following equation:

[ formula ]

L＝w1L1(x,x')+w2L2(y,y')

Where w1 and w2 represent preset weights, L1(a, b) and L2(a, b) represent functions of distances between a and b, respectively, x represents an original pathological image, x ' represents a restored pathological image corresponding to x, y represents a label of x, and y ' represents a diagnosis result of diagnosis x ' in the diagnostic neural network 120.

The distance functions L1 and/or L2 may include a loss function such as Mean Squared Error (MSE), Cross Entropy Error (CEE), or a function embodying a distance between two vectors (e.g., Euclidean distance, n-norm distance, Manhattan distance, etc.). In addition, the two distance functions L1 and L2 may be functions of the same form or functions of different forms. In another embodiment, the distance function L may be in the form of the next [ formula ] further comprising an additional term R ().

[ formula ]

L＝w1L1(x,x')+w2L2(y,y')+R()

Where R () may be a function defined by various parameters of the neural network, additional parameters of the encoding result (z), and the like.

Fig. 4a is a schematic diagram of the process of learning with raw pathology images by training by the automatic encoder 110 and the diagnostic neural network 120.

Referring to fig. 4a, the feed forward process is as follows.

The diagnosis result y is that the original pathological image x is input to the automatic encoder, and the encoder section 111 generates the feature z of the original pathological image x, and the feature z generates the restored pathological image x' after being input from the input decoder section. Then, the restored pathology image x 'is input to the neural network for diagnosis 120, and the diagnosis result y' is output.

In addition, referring to fig. 4a, the error back propagation (back propagation) process is formed as follows:

first, the difference L (y, y ') between the diagnosis result y ' output from the diagnostic neural network 120 and the label y of the original pathological image x is calculated, and the calculation result L (y, y ') is reflected on the diagnostic neural network 120 by a back propagation method. On the other hand, the diagnosis result y' output from the neural network for diagnosis 120 is reflected in the learning of the automatic encoder 110. For example, the difference L1(x, x ') between the original pathological image x and the restored image x' corresponding thereto, and the difference L2(y, y ') between the diagnosis result y' output from the neural network for diagnosis 120 and the label y of the original pathological image x are both reflected on the automatic encoder 110 by the back propagation method, so that the automatic encoder 110 is learned.

On the other hand, according to an embodiment, a plurality of additional pathology images for training that have not been passed through the automatic encoder 110 may be further input to the neural network for diagnosis 120, so that learning of the neural network for diagnosis may be performed. That is, in the present embodiment, the learning module 130 is configured to input a plurality of pathological images for training to the neural network for diagnosis 120 via the automatic encoder 110 in a first step, and to input a plurality of additional pathological images for training to the neural network for diagnosis 120 without via the automatic encoder 100 in a second step, thereby learning the neural network for diagnosis 120.

Fig. 4b is a schematic diagram illustrating the learning process of the automatic encoder 110 and the diagnostic neural network 120 according to the above-described embodiment.

Referring to fig. 4b, the feed forward process is as follows.

The diagnosis result y is that the original pathological image x is marked and then inputted into the automatic encoder 110, the feature z of the original pathological image x is generated in the encoder section 111, and the restored pathological image x' is generated after the feature z is inputted into the decoder section. Then, the restored pathology image x 'is input to the neural network for diagnosis 120, and the diagnosis result y' is output. In addition, the additional original pathology image u labeled with the diagnosis result v is input to the neural network for diagnosis 120 instead of the automatic encoder 110, thereby outputting the diagnosis result v'.

Referring to fig. 4b, the error back propagation process is formed as follows.

When learning is performed on the original pathological image x input to the automatic encoder 110, a difference L (y, y ') between the diagnosis result y' of the restored pathological image x 'output from the diagnostic neural network 120 and the label y of the original pathological image x is calculated, and the calculated L (y, y') is reflected in the diagnostic neural network 120 by a back propagation method. In addition, the diagnosis result y' output from the neural network for diagnosis 120 is reflected in the learning of the automatic encoder 110. For example, the difference L1(x, x ') between the original pathological image x and the restored image x' corresponding thereto, and the difference L2(y, y ') between the diagnosis result y' output from the neural network for diagnosis 120 and the label y of the original pathological image x are both reflected on the automatic encoder 110 by the inverse relay method, so that the automatic encoder 110 learns.

When learning is performed using the additional original pathological image u directly input to the diagnostic neural network 120, a difference L (v, v ') between the diagnostic result v ' output from the diagnostic neural network 120 and the label v of the additional original pathological image u is calculated, and the calculated L (v, v ') is reflected to the diagnostic neural network 120 by a back propagation method.

The additional training pathology image or the additional original pathology image in the present specification is a term different from the training pathology image or the original pathology image (i.e., the pathology image input to the automatic encoder 110 and used for learning of the automatic encoder 110 and the diagnostic neural network 120), and therefore, the pathology image input to the automatic encoder 110 and used for learning of the automatic encoder 110 and the diagnostic neural network 120 is used as the 1 st training pathology image and is used for directly inputting to the diagnostic neural network 120 without passing through the automatic encoder 110, and the pathology image learning the diagnostic neural network 120 is used as the 2 nd training pathology image.

On the other hand, according to an embodiment, the learning module 130 inputs a plurality of 1 st training pathology images into the automatic encoder 110 to learn the automatic encoder 110 and the diagnostic neural network 120, and before that, a plurality of 2 nd training pathology images that do not pass through the automatic encoder 110 are first input into the diagnostic neural network 120 to learn the diagnostic neural network 120.

In addition, in some cases, in the learning of the diagnostic neural network 120, there may be some embodiments in which the diagnostic neural network 120 is learned only by a plurality of training pathology images directly input to the diagnostic neural network 120 without passing through the automatic encoder 110.

Fig. 4c is a schematic diagram illustrating the learning process of the automatic encoder 110 and the diagnostic neural network 120 according to the above embodiment.

Referring to fig. 4c, the feed forward process is as follows.

After the original pathological image x marked as the diagnosis result y is inputted to the automatic encoder 110, the feature z of the original pathological image x is generated in the encoder section 111, and the feature z is inputted to the decoder section to generate the restored pathological image x'. Then, the restored pathology image x 'is input to the neural network for diagnosis 120, and the diagnosis result y' is output. In addition, the additional original pathology image u labeled with the diagnosis result v is input to the neural network for diagnosis 120 instead of the automatic encoder 110, thereby outputting the diagnosis result v'.

Referring to fig. 4c, the error back propagation process is formed as follows.

When the original pathological image x input to the automatic encoder 110 is learned, a difference L (y, y ') between the diagnosis result y' of the restored pathological image x 'output from the neural network for diagnosis 120 and the label y of the original pathological image x is calculated, and the calculated L (y, y') is reflected in the learning of the automatic encoder 110. For example, the difference L1(x, x ') between the original pathological image x and the restored image x' corresponding thereto, and the difference L2(y, y ') between the diagnosis result y' output from the neural network for diagnosis 120 and the label y of the original pathological image x are reflected in the automatic encoder 110 by the back propagation method, so that the automatic encoder learns.

When learning is performed from the additional original pathological image u directly input to the diagnostic neural network 120, a difference L (v, v ') between the diagnostic result v ' output from the diagnostic neural network 120 and the label v of the additional original pathological image u is calculated, and the calculated L (v, v ') is reflected to the diagnostic neural network 120 by a back propagation method.

Referring again to fig. 3, the feature generation module 140 may generate respective feature values of a plurality of retrieval-use pathology images after the learning of the automatic encoder 110 is completed, and construct a DB200 storing them.

That is, the feature generation module 140 may input the retrieval-use pathology image into the learned automatic encoder 110 for each of a plurality of retrieval-use pathology images, generate features of the retrieval-use pathology image by the encoder part, and store the generated features of the retrieval-use image in the DB 200.

Fig. 6 is a schematic diagram illustrating a process of extracting features of each of a plurality of retrieval-use pathology images by the feature generation module 140 and constructing a DB 200.

Referring to fig. 6, the DB200 may include a pathology image data DB (210) and a pathology image index DB (220). The retrieval pathology image may be stored in the pathology image data DB (210) together with the index. For each retrieval pathology image x, the feature generation module 140 may input into the automatic encoder 110 that learned the retrieval pathology image x. Then, the feature z of the retrieval-use pathology image x may be extracted from the learned encoder portion 111 of the automatic encoder 110. Then, the feature generation module 140 may store the feature z of the retrieval-use pathology image x in association with the index id of the retrieval-use pathology image x in the pathology image index DB (220).

On the other hand, referring to fig. 3 again, the retrieval module 150 may retrieve similar images similar to a preset suspicious pathological image after extracting features of each of the plurality of retrieval pathological images.

In this regard, as explained with reference to fig. 6, the retrieval module 150 may input the suspected pathology image q into the learned automatic encoder 110, and then the feature w of the suspected pathology image q may be generated by the encoder portion 111.

Thereafter, the retrieval module 150 may retrieve a feature of a similar pathology image similar to the suspicious pathology image on the DB200 based on the feature w of the suspicious pathology image q. For example, the retrieval module 150 may perform retrieval based on the similarity between the feature w of each of the plurality of retrieval-use pathology images stored in the pathology image index DB (220) and the feature w of the suspected pathology image q. In searching based on the similarity, various techniques for finding the similarity between two vectors can be applied. For example, the similarity between two vectors may be represented by a distance between vectors or a cosine similarity, and various known methods may be applied thereto. On the other hand, when the index id of the similar image similar to the suspected pathology image q is retrieved, the retrieval module 150 may retrieve the similar image from the pathology image data DB (210).

On the other hand, according to an embodiment, the auto-encoder 110 may include a variational auto-encoder (vAE).

On the other hand, although the present specification mainly describes an example in which the technical idea of the present invention is applied to prostate cancer, it is necessary to apply the technical idea of the present invention to other diseases, and it is necessary to diagnose a specific tissue in consideration of not only the specific tissue but also the state of the tissue surrounding the specific tissue, so that a more accurate diagnosis can be performed and the diagnosis result can be visualized, which can be easily inferred by a person having ordinary skill in the art of the present invention.

On the other hand, the method for retrieving a pathological image according to the technical idea of the present invention can be applied to more effectively produce learning data used when a machine for disease diagnosis is learned using a biological image.

In addition, the method for retrieving a pathological image according to the technical idea of the present invention can also be used to realize a pathological diagnosis method by retrieving a similar image. For example, a system and method can be implemented that converts a specific input slide image that is a diagnosis target into one or more retrievable images or images, performs similar image search on each of the converted retrievable images, and then synthesizes diagnosis results for them, generating diagnosis results for the input slide image.

On the other hand, according to an embodiment, the system 100 and/or the terminal 20 for retrieving the pathology image may include a processor and a memory storing a program executed by the processor. The processor may include a single core CPU or a multi-core CPU. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magneto-optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory by the processor and other components may be controlled by a memory controller.

On the other hand, the method of retrieving a pathology image according to an embodiment of the present invention may be implemented in the form of a computer-readable program command and stored in a computer-readable recording medium, and the control program and the object program according to an embodiment of the present invention may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store computer system-readable data.

The program commands recorded on the recording medium may be specially designed and configured for the present invention, or may be known and used by those skilled in the software art.

Computer-readable recording media such as magnetic media (magnetic media) like hard disks, floppy disks, and magnetic tapes, optical media (optical media) like CD-ROMs and DVDs, magneto-optical media (magnetic-optical media) like floppy disks (flexible disks), specially configured hardware devices such as ROM, RAM, flash memory, etc., which store and execute program commands. In addition, the computer-readable recording medium is dispersed in a networked computer system, and the computer-readable code may be stored and executed in a distributed fashion.

Examples of the program command include not only a machine language code created by a compiler but also a device that electronically processes information using an interpreter or the like, such as a high-level language code executable by a computer.

The hardware devices may be configured as one or more software modules to perform the operations of the present invention, and vice versa.

The foregoing description of the present invention is provided for illustration only, and it will be understood by those having ordinary skill in the art to which the present invention pertains that the present invention may be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, the above embodiments are to be understood as examples in all respects and not as limitations. For example, elements described as singular may be implemented separately, and elements described as discrete may also be implemented in combination.

The scope of the present invention is indicated by the scope of the claims to be described later, and is not reflected by the detailed description, and the meaning and scope of the claims and all changes or modifications derived from the unified concept thereof should be construed to be included in the scope of the present invention.

Possibility of industrial application

The present invention can be used for "a system and method for retrieving a pathological image".