阅读说明：本技术 一种病理显微图像实时采集分析系统、方法、装置及介质 (Pathological microscopic image real-time acquisition and analysis system, method, device and medium ) 是由 叶德贤 房劬 姜辰希 于 2019-12-19 设计创作，主要内容包括：本发明涉及图像实时采集和分析技术,属于显微病理辅助诊断的技术领域。提供了一种能够实现实时扫描并快速高效分析得到病理状况的系统和方法,该系统包括：显微图像采集装置；图像分析装置,连接于所述显微图像采集装置,用于实时获取所述显微图像块,并将所述显微图像块输入到训练后的神经网络模型进行分析,得到显微图像块分析结果；显示装置,连接于所述图像分析装置,输出显示显微图像块分析结果。本发明还提供对应上述装置的方法或介质。可实时采集显微图像并实时进行病理分析和输出病理分析结果,取代了现有技术中全部图像采集完毕再进行图像分析处理,为病人争取了极大的时间,克服了当前在手术过程中病理分析所需时间较久的技术问题。(The invention relates to a real-time image acquisition and analysis technology, and belongs to the technical field of microscopic pathology auxiliary diagnosis. A system and method for real-time scanning and fast and efficient analysis of pathological conditions are provided, the system comprising: a microscopic image acquisition device; the image analysis device is connected with the microscopic image acquisition device and used for acquiring the microscopic image blocks in real time and inputting the microscopic image blocks into the trained neural network model for analysis to obtain an analysis result of the microscopic image blocks; and the display device is connected with the image analysis device and outputs and displays the analysis result of the microscopic image block. The invention also provides a method or a medium corresponding to the device. The microscopic image can be acquired in real time, pathological analysis can be performed in real time, pathological analysis results can be output, image analysis processing after all images are acquired in the prior art is replaced, great time is won for patients, and the technical problem that long time is needed for pathological analysis in the operation process at present is solved.)

1. A pathological microscopic image real-time acquisition and analysis system for microscopic image acquisition and real-time analysis and diagnosis of a sample, wherein the sample comprises human cells or tissues, and the pathological microscopic image real-time acquisition and analysis system is characterized by comprising:

(1) a microscopic image acquisition device, comprising:

the objective table is used for bearing a sample;

the camera is used for shooting a sample to obtain a microscopic image; and

the control unit is used for controlling the relative position movement of the objective table and the camera and controlling the camera to sequentially shoot microscopic image blocks at a plurality of positions of the sample;

(2) the image analysis device is connected with the microscopic image acquisition device and used for acquiring one or more microscopic image blocks corresponding to one or more positions of the sample in real time and inputting the acquired one or more microscopic image blocks into the trained neural network model for analysis in real time to obtain one or more microscopic image block analysis results, wherein the microscopic image block analysis results comprise analysis results of whether sample cells corresponding to the microscopic image blocks have pathological abnormalities or not; when all the acquired microscopic image blocks are completely analyzed, obtaining an overall analysis result;

(3) and the display device is connected with the image analysis device and is used for outputting and displaying the analysis result of the microscopic image block and/or the overall analysis result to a user.

2. The system for real-time collection and analysis of pathological microscopic images according to claim 1, wherein the control unit controls the stage to move in a stepping manner and controls the camera to take a shot each time the stage moves to a position, and the step size of the stepping movement of the stage is less than or equal to the width of the microscopic field of view that can be shot by the camera.

3. The pathological microscopic image real-time acquisition and analysis system according to claim 2, wherein the camera shoots the sample line by line and transmits the shooting result in real time.

4. An image analysis apparatus for analyzing a pathological microscopic image in real time, comprising:

the microscopic image block acquisition module is in data transmission connection with a microscopic image acquisition device, sequentially shoots a plurality of positions of a sample synchronously with the microscopic image acquisition device, and acquires a microscopic image block corresponding to one or more positions of the sample in real time;

the microscopic image block analysis module comprises a trained neural network model and is used for sequentially analyzing the microscopic image blocks corresponding to the one or more positions to obtain one or more microscopic image block analysis results, wherein the microscopic image block analysis results comprise analysis results of whether sample cells corresponding to the microscopic image blocks have pathological abnormalities;

the overall analysis module obtains an overall analysis result according to the analysis results of all the microscopic image blocks;

and the output module is used for outputting the analysis result of the microscopic image block and/or the overall analysis result.

5. The image analysis device according to claim 4, further comprising an image stitching module, configured to stitch the microscopic image blocks at the respective positions of the sample to obtain an overall microscopic image of the sample; the output module is also used for outputting the sample overall microscopic image.

6. The image analysis apparatus according to claim 5, further comprising an image labeling module that labels a position of the pathological abnormality in the sample whole microscopic image based on a microscopic image block analysis result; the output module is also used for outputting the marked sample overall microscopic image.

7. The image analysis device according to claim 4, further comprising a microscopic image block analysis result judgment module, configured to judge the one or more microscopic image block analysis results after each analysis result of one or more microscopic image blocks is obtained, and output the microscopic image block analysis result if the analysis result satisfies a set condition; otherwise, outputting the overall analysis result after the overall analysis result is obtained.

8. The image analysis device of claim 4, wherein the neural network model is a convolutional neural network model, and the trained neural network model is obtained by training through the following steps:

obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information;

inputting the training data into a convolutional neural network model for training to obtain a trained convolutional neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a batch normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

。

9. An image real-time analysis method for analyzing pathological microscopic images in real time, comprising:

sequentially shooting a plurality of positions of a sample synchronously, and acquiring microscopic image blocks corresponding to one or more positions of the sample in real time;

sequentially inputting the obtained one or more microscopic image blocks corresponding to the positions into a trained neural network model for analysis to obtain one or more microscopic image block analysis results, wherein the microscopic image block analysis results comprise analysis results of whether sample cells corresponding to the microscopic image blocks have pathological abnormalities;

obtaining an overall analysis result according to the analysis results of all the microscopic image blocks;

and outputting the overall analysis result.

10. The method for real-time image analysis according to claim 9, further comprising:

judging the analysis result of one or more microscopic image blocks after obtaining the analysis result of one or more microscopic image blocks;

if the analysis result meets the set condition, outputting the analysis result of the microscopic image block;

otherwise, outputting the overall analysis result after the overall analysis result is obtained.

11. The real-time image analysis method according to claim 9, wherein the neural network model is a convolutional neural network model, and the trained neural network model is obtained by training through the following steps:

obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information;

inputting the training data into a convolutional neural network model for training to obtain a trained neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

12. The method for real-time image analysis according to claim 9, further comprising: and splicing the microscopic image blocks at all positions of the sample to obtain an integral microscopic image of the sample.

13. The method of claim 12, further comprising marking the location of the pathological anomaly in the sample whole microscopic image based on microscopic image patch analysis results.

14. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed, cause a computer to perform the method of any of claims 9-13.

15. A neural network model training method for pathological microscopic image analysis is characterized by comprising the following steps:

obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information;

inputting the training data into a convolutional neural network model for training to obtain a trained neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

Technical Field

The invention relates to a real-time image acquisition and analysis technology, in particular to a pathological microscopic image real-time acquisition and analysis system, method, device and medium, belonging to the technical field of microscopic pathological auxiliary diagnosis.

Background

Currently, the method of lung-assisted diagnosis is usually a needle biopsy under Computed Tomography (CT) guidance, which is a common method for diagnosing lung. The invention mainly aims to collect sample cells at a focus position in a bronchus or a lung through a fiber bronchoscope lung biopsy so as to further analyze and process the sample cells, thereby assisting in diagnosing an illness state. Currently, the field is commonly used with a Rapid field evaluation technique for specimens, namely ROSE (Rapid on-site evaluation), which refers to a Rapid field examination of specimen cells by a cytopathologist and an evaluation of the quality of fine needle smear and biopsy prints. The examiner can know whether the sample size is enough through the ROSE to judge whether more sample sizes need to be collected, so that repeated puncture of a patient is avoided, enough sample sizes can be collected at one time, and meanwhile, a required analysis result can be provided for diagnosis and treatment of subsequent diseases through ROSE initial diagnosis.

Specifically, the bronchoscopic biopsy print and the fine needle aspiration smear are samples obtained by performing a forceps biopsy and a brush biopsy on a bronchus or an intra-pulmonary lesion or a transbronchial aspiration biopsy on a hilum or mediastinal lymph node in a bronchofiberscope lung biopsy, and a doctor performs ROSE on the samples, fixes and stains the slide, and then observes lung cells under a microscope. However, the prior art has the technical defect that a doctor who performs ROSE on a bronchoscope biopsy print and a fine needle puncture smear is a pathologist, and a respiratory doctor is difficult to complete ROSE, so that the bronchoscope biopsy print and the fine needle puncture smear need to be sent to the pathologist, the ROSE time is prolonged in the process, a patient is on an operating table and is often operated by the respiratory doctor, the requirement for obtaining a detection analysis structure in real time is urgent, the time of the patient on the operating table is precious, and a current microscope scanner generates a complete microscopic image after scanning the complete bronchoscope biopsy print or the fine needle puncture smear, and cannot process an analysis picture while scanning in the scanning process, so that the analysis processing time is prolonged, and the difficulty is brought to the rapid diagnosis of the disease. In summary, the problems faced at present are that it takes time to send the images to the pathology department for manual examination, and that it takes time to collect the large images before the examination can be started.

In reality, pathological microscopic diagnosis is applied to various disease detection and treatment scenes, such as microscopic pathological diagnosis of lung cell pathology, cervical cell pathology, breast pathology or thyroid pathology, and also needs to be examined manually in a pathology department. The existing method is long in time consumption and cannot meet the diagnosis requirement in an emergency.

Disclosure of Invention

The present invention is directed to a system or method for real-time scanning and fast and efficient analysis of pathological conditions, so as to overcome the above-mentioned technical drawbacks and problems.

The invention provides a pathological microscopic image real-time acquisition and analysis system, which is used for carrying out microscopic image acquisition and real-time analysis and diagnosis on a sample, wherein the sample comprises human cells or tissues, and the system comprises:

a microscopic image acquisition device, comprising: the objective table is used for bearing a sample; the camera is used for shooting a sample to obtain a microscopic image; the control unit is used for controlling the relative position movement of the objective table and the camera and controlling the camera to sequentially shoot microscopic image blocks at a plurality of positions of the sample; the microscopic image block refers to one area of the sample which can be shot by the camera at each time, and as the sample size is usually larger than the width of a microscopic field which can be shot by the camera, the microscope needs to shoot the sample in blocks in sequence, and one microscopic image block can be obtained by each shooting;

the image analysis device is connected with the microscopic image acquisition device and used for acquiring the one or more microscopic image blocks in real time and inputting the acquired one or more microscopic image blocks into the trained neural network model for analysis in real time to obtain one or more microscopic image block analysis results, wherein the one or more microscopic image block analysis results comprise an analysis result of whether sample cells corresponding to the one or more microscopic image blocks have pathological abnormalities or not, namely when the sample has pathological abnormalities, one or more microscopic image blocks in the sample comprise information of whether the sample cells have pathological abnormalities or not, and the analysis result of the microscopic image block can show an analysis result of whether the sample cells have pathological abnormalities or not in real time; when all the acquired microscopic image blocks are completely analyzed, obtaining an overall analysis result; when a sample has pathological abnormality and a certain microscopic image block is possibly analyzed, the analysis result of whether the sample cells have pathological abnormality can be obtained, however, in the actual operation, the analysis of all the microscopic image blocks is completed to obtain the overall analysis result, and at the moment, the analysis result of one or some of the microscopic image blocks is consistent with the overall analysis result; when the sample has no pathological abnormality, the analysis result of whether the sample cells have pathological abnormality can be obtained only after the analysis of all the microscopic image blocks of the sample is completed, and at the moment, the analysis result of each microscopic image block is consistent with the overall analysis result;

and the display device is connected with the microscopic image analysis device and is used for outputting and displaying the analysis result of the microscopic image block and/or the overall analysis result to a user.

Further, the control unit controls the object stage to move in a stepping mode and controls the camera to shoot once when the object stage moves to a position, and the step length of stepping movement of the object stage is smaller than or equal to the width of a microscopic field of view which can be shot by the camera.

Furthermore, the camera shoots the samples line by line, and the shooting result is transmitted in real time.

Optionally, the display device is a mobile terminal or a visual screen, and the display device and the image analysis device realize data interaction and output an analysis result in real time for medical staff to observe.

The present invention also provides an image analysis apparatus for analyzing a pathological microscopic image in real time, characterized by comprising:

the microscopic image block acquisition module is in data transmission connection with a microscopic image acquisition device, sequentially shoots a plurality of positions of a sample synchronously with the microscopic image acquisition device, and acquires a microscopic image block corresponding to one or more positions of the sample in real time; the microscopic image block analysis module comprises a trained neural network model and is used for sequentially analyzing the microscopic image blocks corresponding to the one or more positions to obtain one or more microscopic image block analysis results, wherein the one or more microscopic image block analysis results comprise an analysis result of whether sample cells corresponding to the one or more microscopic image blocks have pathological abnormalities; the overall analysis module obtains an overall analysis result according to the analysis results of all the microscopic image blocks; and the output module is used for outputting the analysis result of the microscopic image block and/or the overall analysis result.

Further, the image analysis device further comprises an image splicing module, which is used for splicing the microscopic image blocks at all positions of the sample to obtain an integral microscopic image of the sample; the output module is also used for outputting the sample overall microscopic image.

Furthermore, the image analysis device further comprises an image marking module, which marks the position of the pathological abnormality in the sample whole microscopic image according to the analysis result of the microscopic image block; the output module is also used for outputting the marked sample overall microscopic image.

Preferably, the image analysis device further comprises a microscopic image block analysis result judgment module, configured to judge an analysis result of one or more microscopic image blocks after each analysis result of the one or more microscopic image blocks is obtained, and output the microscopic image block analysis result if the analysis result meets a set condition; otherwise, outputting the overall analysis result after the overall analysis result is obtained.

Further, the neural network model is a convolutional neural network model, and the trained neural network model is obtained by training through the following steps: obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information; inputting the training data into a convolutional neural network model for training to obtain a trained neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a batch normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

。

The invention also provides an image real-time analysis method for analyzing pathological microscopic images in real time, which comprises the following steps: sequentially shooting a plurality of positions of a sample synchronously, and acquiring microscopic image blocks corresponding to one or more positions of the sample in real time; sequentially inputting the obtained one or more microscopic image blocks into a trained neural network model for analysis to obtain one or more microscopic image block analysis results, wherein the microscopic image block analysis results comprise analysis results of whether sample cells corresponding to the microscopic image blocks have pathological abnormalities; obtaining an overall analysis result according to the analysis results of all the microscopic image blocks; and outputting the overall analysis result.

Further, the method further comprises: judging the analysis result of one or more microscopic image blocks after obtaining the analysis result of one or more microscopic image blocks; if the analysis result meets the set condition, outputting the analysis result of the microscopic image block; otherwise, outputting the overall analysis result after the overall analysis result is obtained.

Further, the set condition in the above method is the presence or absence of an abnormality, or the presence or absence of a cancerous change.

Optionally, the sample in the above method is any one of a human lung cell sample, a human cervical cell sample, a human breast cell sample, or a human thyroid cell sample.

More preferably, the sample in the above method is a human cell or tissue obtained by puncture or endoscope.

Optionally, the human lung cell sample in the above method is human lung cells obtained by puncturing or human lung cells obtained by bronchoscopy.

Still further, the analysis result of the microscopic image block in the above method includes that the cancer is qualitatively negative or positive;

alternatively, the microscopic image block analysis result in the above method can be further specifically subdivided into any one or more of lung squamous carcinoma, lung adenocarcinoma, small cell lung cancer, undefined non-small cell lung cancer, other malignant lesions, granuloma and inflammation.

Optionally, the neural network model in the above method is a convolutional neural network model, and the trained neural network model is obtained by training through the following steps: obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information; inputting the training data into a convolutional neural network model for training to obtain a trained neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

。

Optionally, the labeling information in the above method includes at least two of squamous cell lung cancer, adenocarcinoma of lung, small cell lung cancer, non-small cell lung cancer which cannot be defined, other malignant lesions, no obvious abnormality, granuloma and inflammation.

Preferably, the method further comprises: and splicing the microscopic image blocks at all positions of the sample to obtain an integral microscopic image of the sample.

Optionally, the method further comprises marking the position of the pathological abnormality in the whole sample microscopic image according to the analysis result of the microscopic image block.

The microscopic image blocks include non-preprocessed or preprocessed microscopic image blocks, and the preprocessing includes preprocessing modes commonly used in the field, such as normalization and size adjustment, of the microscopic image blocks shot by a microscope.

The present invention also provides a computer-readable storage medium having stored thereon computer-executable instructions that, when executed, cause a computer to perform the real-time acquisition and real-time analysis of microscopic image blocks of the present invention as described above, and output the analysis results.

The invention further provides a neural network model training method for pathological microscopic image analysis, which is characterized by comprising the following steps: obtaining training data, wherein the training data are microscopic images of samples and corresponding marking information; inputting the training data into a convolutional neural network model for training to obtain a trained neural network model, wherein the convolutional neural network model consists of a convolutional layer, a pooling layer and a normalization layer, and the loss function for training the convolutional neural network model is as follows:

or

。

Preferably, the neural network model training method further includes testing the trained neural network model by using test data, and specifically includes: obtaining test data, wherein the test data are microscopic images of the sample which are not repeated with the training data and corresponding marking information; analyzing the microscopic image in the test data by using the trained neural network model to obtain a microscopic image block test analysis result; and comparing the test analysis result of the microscopic image block with the label information in the test data to obtain a test comparison result.

By adopting the pathological microscopic image acquisition device, system and method of the technical scheme, the whole scheme can realize the following advantages in comparison with the prior art in the aspects of image acquisition, analysis and display of pathological cells or tissues, and the following specific summary is as follows:

(1) the method can acquire the images of cells or tissues in real time and carry out pathological analysis, replaces the prior art that the analysis processing of the images is carried out after all the images are acquired, strives for great time for patients, and overcomes the technical problem that the time required by pathological analysis in the operation process is longer in the background technology;

(2) because of the adoption of the neural network training method, the accuracy of pathological analysis is obviously improved, and the pathological result obtained by analysis after the neural network training is adopted is more accurate;

(3) the training method adopted by the invention has quick response, so that the processing speed is greatly improved in the aspect of the processing speed of pathological analysis compared with the traditional method.

Drawings

FIG. 1 is an embodiment of a pathological microscopic image real-time acquisition and analysis system;

FIG. 2 is an embodiment of an image analysis apparatus;

FIG. 3 is an embodiment of a method for real-time image analysis;

FIG. 4 is an embodiment of a neural network model training method for pathology microscopy image analysis;

FIG. 5 is a convolutional neural network model embodiment;

FIG. 6 is an example of a microscopic image of lung cells without significant abnormalities;

fig. 7 is an example of a microscopic image of lung cells with adenocarcinoma.

Detailed Description

In order to better explain the objects of the invention, the implementation of the solution and the advantages of the invention compared to the prior art, the invention will be further elaborated and explained below, by way of example, with reference to the drawings and examples of different embodiments shown. It is to be understood that the specific embodiments described herein, or illustrated herein, are merely illustrative or exemplary of the general inventive concept and are not to be considered as limiting the scope of the claims. It is intended that all equivalents and modifications based on the spirit and subject matter of the invention shall fall within the scope of the invention.

The method provided by the embodiment of the invention can be applied to the system shown in FIG. 1. The image analysis means in the system may be a computer device comprising a processor, a memory connected via a system bus, the memory having stored therein a computer program, which when executed by the processor may perform the steps of the method embodiments described below. Optionally, the computer device may further comprise a network interface, a display screen and an input device. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a nonvolatile storage medium storing an operating system and a computer program, and an internal memory. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. Optionally, the computer device may be a server, a personal computer, a personal digital assistant, other terminal devices such as a tablet computer, a mobile phone, and the like, or a cloud or a remote server, and the specific form of the computer device is not limited in the embodiment of the present application.

The following describes the technical solution of the present invention and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.