阅读说明：本技术 观察装置及观察方法 (Observation device and observation method ) 是由 阿部洋子 于 2018-10-01 设计创作，主要内容包括：为了不需要对取得观察图像和参照图像时的拍摄条件进行调整而简单且高精度地向观察者通知被摄体中的异常的发生部位,提供一种观察装置(1),其具有：拍摄部(2),其对被摄体进行拍摄而取得观察图像；参照图像生成部(7),其根据由拍摄部(2)取得的观察图像,生成与观察图像相比消除了图像中的异常的参照图像；以及显示部(4),其切换显示所生成的参照图像和观察图像。(In order to notify an observer of a location of occurrence of an abnormality in an object easily and accurately without adjusting imaging conditions for acquiring an observation image and a reference image, an observation device (1) is provided with: an imaging unit (2) that images a subject and acquires an observation image; a reference image generation unit (7) that generates a reference image from which abnormalities in the image have been eliminated as compared with the observation image, based on the observation image acquired by the imaging unit (2); and a display unit (4) for displaying the generated reference image and observation image in a switched manner.)

1. An observation device, comprising:

an imaging unit that images a subject to acquire an observation image;

a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image; and

and a display unit that displays the generated reference image and the observation image in a switched manner.

2. An observation device, comprising:

an imaging unit that images a subject to acquire an observation image;

a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image;

a difference image generating unit that generates a difference image between the generated reference image and the observation image; and

and a display unit that displays the generated difference image and the observation image in a switched manner.

3. An observation device, comprising:

an imaging unit that images a subject to acquire an observation image;

a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image;

a difference calculation unit that calculates a magnitude of a difference between the generated reference image and the observation image;

an image correction unit that generates a corrected image obtained by adjusting the color attribute or brightness of the reference image according to the calculated difference; and

a display unit that switches between displaying the generated corrected image and the observation image.

4. The observation device according to any one of claims 1 to 3, wherein,

the observation device has:

a visual field movement state detection unit that detects a movement state of an observation visual field of the imaging unit; and

and a display adjustment unit that adjusts a switching mode of display on the display unit according to the detected movement state of the observation field.

5. The viewing device of claim 4,

the movement state of the observation visual field is stop of the observation visual field, change of movement speed, or visual field return, which is a case of returning to the observation visual field of the one-time observation.

6. The viewing device of claim 4,

the switching pattern is a switching speed.

7. The viewing device of claim 4,

the switching of the display is performed a plurality of times,

the switching pattern is a switching number.

8. The observation device according to any one of claims 1 to 3, wherein,

the reference image generation unit generates the reference image based on teaching processing.

9. A method of observation in which, in the above,

an observation image is acquired by photographing an object,

generating a reference image from the acquired observation image, the reference image being free from abnormalities in the image compared with the observation image,

switching to display the generated reference image and the observation image.

10. A method of observation in which, in the above,

an observation image is acquired by photographing an object,

generating a reference image from the acquired observation image, the reference image being free from abnormalities in the image compared with the observation image,

generating a difference image of the generated reference image and the observation image,

switching to display the generated difference image and the observation image.

11. A method of observation in which, in the above,

an observation image is acquired by photographing an object,

generating a reference image from the acquired observation image, the reference image being free from abnormalities in the image compared with the observation image,

calculating a magnitude of a difference between the generated reference image and the observation image,

generating a corrected image obtained by adjusting the color attribute or brightness of the reference image according to the calculated difference,

switching display of the generated correction image and the observation image.

Technical Field

The present invention relates to an observation device and an observation method.

Background

In the inspection of the subject, the presence or absence of an abnormality such as a shape defect is examined by comparing a reference image which is stored in advance and does not include the abnormality with a newly acquired observation image. (see, for example, patent document 1).

Patent document 1 discloses the following method: by storing the observation conditions for acquiring the reference image in advance, the same observation conditions are reproduced for acquiring the observation image, and the comparison between the observation image and the reference image is facilitated.

Prior art documents

Patent document

Patent document 1: japanese patent laid-open publication No. 2003-066339

Disclosure of Invention

Problems to be solved by the invention

However, in the method of patent document 1, a reference image to be compared is acquired before an observation image is acquired, and even if the observation conditions are made to coincide with each other, there are subtle differences such as differences in color tone of the image due to deterioration of a lamp of an illumination device or mechanical errors of a lens or the like, and positional shifts due to differences in the installation position or angle of an object in the field of view, and it is difficult to make the imaging conditions perfectly coincide with each other.

An object of the present invention is to provide an observation apparatus and an observation method capable of easily and accurately notifying an observer of a location of occurrence of an abnormality in an object without adjusting imaging conditions for acquiring an observation image and a reference image.

Means for solving the problems

One aspect of the present invention is an observation device including: an imaging unit that images a subject to acquire an observation image; a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image; and a display unit that displays the generated reference image and the observation image in a switched manner.

According to this aspect, when the observation image is acquired by the imaging unit, the reference image generating unit generates the reference image from the acquired observation image, and the display unit switches between displaying the reference image and the observation image. When an abnormality is included in an observed image, the observer can be made to pay attention to the abnormality by using the afterimage by switching the reference image displayed so as not to include the abnormality in the image as compared with the observed image.

Since the reference image is an image from which an abnormality in the image is removed as compared with the acquired observation image, the imaging condition completely matches the observation target, and a problem of variation in comparison does not occur. This makes it possible to easily and accurately notify the observer of the occurrence of an abnormality in the subject without adjusting the imaging conditions for acquiring the observation image and the reference image.

Another aspect of the present invention is an observation device including: an imaging unit that images a subject to acquire an observation image; a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image; a difference image generating unit that generates a difference image between the generated reference image and the observation image; and a display unit that displays the generated difference image and the observation image by switching between them.

According to this aspect, when the observation image is acquired by the imaging unit, the reference image generating unit generates the reference image from the acquired observation image, the difference image generating unit generates the difference image between the reference image and the observation image, and the display unit switches between displaying the difference image and the observation image. When an abnormality is included in the observation image, the difference image of the reference image, which does not include the abnormality as compared with the observation image, can make the abnormal portion more clearly apparent, and switching the display can improve the attractiveness of making the observer pay attention to the abnormal portion.

Another aspect of the present invention is an observation device including: an imaging unit that images a subject to acquire an observation image; a reference image generation unit that generates, based on the acquired observation image, a reference image from which an abnormality in the image has been eliminated as compared with the observation image; a difference calculation unit that calculates a magnitude of a difference between the generated reference image and the observation image; an image correction unit that generates a corrected image obtained by adjusting the color attribute or brightness of the reference image according to the calculated difference; and a display unit that switches between displaying the generated corrected image and the observation image.

According to this aspect, when the observation image is acquired by the imaging unit, the reference image generation unit generates the reference image from the acquired observation image, and the difference calculation unit calculates the magnitude of the difference between the reference image and the observation image. Then, the image correction unit generates a corrected image in which the color attribute or brightness of the reference image is adjusted in accordance with the magnitude of the difference, and the display unit displays the corrected image and the observation image in a switched manner. When the difference is small, the abnormal portion can be more clearly made visible by adjusting the color attribute such as saturation or contrast or brightness, and the attractiveness for the observer to pay attention to the abnormal portion can be improved by switching the display.

In the above aspect, the observation device may include: a visual field movement state detection unit that detects a movement state of an observation visual field of the imaging unit; and a display adjustment unit that adjusts a switching mode of display in the display unit according to the detected movement state of the observation field of view.

According to this configuration, the display switching mode is adjusted by the display adjusting unit according to the movement state of the observation field. That is, when the observer observes the observation image while moving the observation field, the moving state of the observation field changes when the observation field suspected of having the abnormal portion is displayed. Therefore, by adjusting the switching pattern based on the movement state, the attractiveness of focusing the observer on the abnormal portion can be improved.

In the above aspect, the moving state of the observation field of view may be a stop of the observation field of view, a change in moving speed, or a field of view return, which is a return to the observation field of view observed once.

According to this configuration, when the observation visual field is stopped, when the moving speed is reduced, or when the visual field is returned to the observation visual field for the first-time observation, it can be determined that the observation visual field in which the abnormal portion is suspected is displayed, and the switching pattern is adjusted, whereby the attractiveness of focusing the observer on the abnormal portion can be improved.

In the above aspect, the switching pattern may be a switching speed.

According to this configuration, the residual image is more likely to remain as the switching speed is higher, and the attractiveness of focusing the observer on the abnormal portion can be improved.

In the above aspect, the display may be switched a plurality of times, and the switching pattern may be a number of times of switching.

In the above aspect, the reference image generation unit may generate the reference image based on a teaching process.

With this configuration, the reference image can be easily generated. For example, a reference image can be generated by inputting an observation image to a network that extracts a feature amount capable of reproducing a pattern that is not abnormal by learning only the pattern that is not abnormal in advance.

Another aspect of the present invention is an observation method in which an observation image is acquired by imaging a subject, a reference image in which an abnormality in an image is eliminated from the observation image is generated based on the acquired observation image, the generated reference image and the observation image are switched and displayed,

another aspect of the present invention is an observation method in which an observation image is acquired by imaging a subject, a reference image in which an abnormality in an image is eliminated from the observation image is generated based on the acquired observation image, a difference image between the generated reference image and the observation image is generated, and the generated difference image and the observation image are displayed in a switched manner.

Another aspect of the present invention is an observation method in which an observation image is acquired by capturing an image of a subject, a reference image in which an abnormality in the image is eliminated from the observation image is generated based on the acquired observation image, the magnitude of a difference between the generated reference image and the observation image is calculated, a correction image in which a color attribute or brightness of the reference image is adjusted is generated based on the magnitude of the calculated difference, and the generated correction image and the observation image are displayed in a switched manner.

Effects of the invention

According to the present invention, the following effects are obtained: the observer can be notified of the occurrence of an abnormality in the subject easily and accurately without adjusting the imaging conditions for acquiring the observation image and the reference image.

Drawings

Fig. 1 is a block diagram showing an observation apparatus according to a first embodiment of the present invention.

Fig. 2 is a flowchart showing an observation method using the observation apparatus of fig. 1.

Fig. 3 is a block diagram showing a first modification of the observation apparatus of fig. 1.

Fig. 4 is a block diagram showing a second modification of the observation apparatus of fig. 1.

Fig. 5 is a block diagram showing an observation apparatus according to a second embodiment of the present invention.

Fig. 6 is a flowchart showing an observation method using the observation apparatus of fig. 5.

Fig. 7 is a flowchart for explaining a display method determining operation of the flowchart of fig. 6.

Fig. 8 is a flowchart illustrating a first modification of the observation method of fig. 6.

Fig. 9 is a flowchart showing a part of a second modification of the observation method of fig. 6.

Fig. 10 is a flow chart of a continuation of the flow chart of fig. 9.

Fig. 11 is a flowchart showing a third modification of the observation method of fig. 6.

Fig. 12 is a flowchart for explaining a display method determining operation in the flowchart of fig. 6.

Fig. 13 is a flowchart showing a fourth modification of the observation method of fig. 6.

Detailed Description

The observation apparatus 1 and the observation method according to the first embodiment of the present invention will be described below with reference to the drawings.

As shown in fig. 1, the observation device 1 of the present embodiment includes: an image acquisition unit (imaging unit) 2 that acquires an observation image by imaging the light condensed by the microscope apparatus 100; a control unit 3 that processes the acquired observation image; and a display unit 4 for displaying the image processed by the control unit 3. The control unit 3 is composed of a processor and a memory.

The microscope apparatus 100 moves a stage on which an object is mounted by a user operation, and forms an image of the object in an observation field based on predetermined imaging conditions. The present invention is not limited to a microscope device having a stage, and any microscope device having any configuration that can move a field of view to observe an object may be applied.

The image acquisition unit 2 is a camera mounted on the microscope device 100, and includes: an imaging unit 5 that captures an image of the specimen in an observation field of view imaged by the microscope device 100 and acquires a live image or a still image as an observation image; and a memory 6 for storing the acquired observation image.

The control unit 3 includes a reference image generation unit 7, a difference image generation unit 8, a display image generation unit 9, a storage unit 10, and a display control unit (display adjustment unit) 11.

The reference image generating unit 7 has a network that learns a pattern (forward solution pattern) of an abnormal object in advance and extracts a parameter that can reproduce the forward solution pattern. An automatic encoder is used for network construction. The parameters obtained by the automatic encoder are stored in the storage unit 10.

When an observed image including an abnormality such as a damage is input to a network in which a forward solution pattern is learned by an automatic encoder, a reference image in which the abnormality in the image is removed from the observed image is generated by performing image restoration using parameters of the forward solution pattern learned in advance. When an observation image including no abnormality is input, the observation image is directly output as a reference image.

The difference image generating unit 8 generates a difference image between the observation image and the reference image by receiving the observation image and the reference image. The difference image is obtained by scanning the sliding window to calculate the mean square error of the observation image and the reference image and imaging the calculated mean square error.

The display image generating unit 9 calculates the sum of the mean square errors obtained by the difference image generating unit 8, and selects the reference image as the display image when the sum is greater than a predetermined threshold, and selects the difference image as the display image when the sum is equal to or less than the predetermined threshold.

The display control unit 11 receives the observation image acquired by the imaging unit 5 and the display image generated by the display image generation unit 9, switches the images a predetermined number of times at predetermined time intervals, and outputs the images to the display unit 4. The predetermined number of times of switching the display on the display unit 4 is preferably set to a plurality of times instead of 1 time.

An observation method using the observation device 1 of the present embodiment configured as described above is described below.

In the observation method according to the present embodiment, as shown in fig. 2, the image obtaining unit 2 captures an image of the subject imaged by the microscope apparatus 100 to obtain an observation image (step S1), and the reference image generating unit 7 generates a reference image in which the abnormality in the image is removed from the observation image based on the obtained observation image (step S2).

The reference image based on the observation image is generated by inputting the observation image into a network in which a correct solution pattern of an object having no abnormality is learned in advance and a parameter capable of reproducing the correct solution pattern is extracted.

Then, the difference image generating unit 8 generates a difference image between the generated reference image and the observation image (step S3), the display image generating unit 9 calculates the magnitude of the difference from the difference image (step S4), and determines whether or not the magnitude of the difference is larger than a predetermined threshold (step S5).

As a result of the determination, when the difference is greater than the predetermined threshold value, the reference image is selected as the display image (step S6), and when the difference is equal to or less than the predetermined threshold value, the difference image is selected as the display image (step S7), and the display control unit 11 switches the display of the observation image and the display image at predetermined time intervals (step S8). Then, it is determined whether or not switching between the observation image and the display image is performed a predetermined number of times (step S9), and if the switching has not been performed the predetermined number of times, the steps from step S1 are repeated, and if the switching has been performed the predetermined number of times, the processing is ended.

According to the observation apparatus 1 and the observation method of the present embodiment, unlike the method of detecting an abnormality in an observation image by comparing the observation image with a reference image generated in advance, a reference image in which an abnormality in an image is eliminated as compared with the observation image is generated from the obtained observation image, and therefore, the imaging conditions such as the size, arrangement, and optical system and light source used for the object can all be matched. That is, the observation apparatus 1 and the observation method according to the present embodiment have the following advantages: the observer can be notified of the occurrence of an abnormality in the subject easily and accurately without adjusting the imaging conditions for obtaining the observation image and the reference image.

When the difference between the observation image and the reference image is large, the reference image is selected as the display image, and the observation image and the reference image are switched and displayed. When the difference is large, the observation image and the reference image are switched and displayed so that only the presence or absence of an abnormality is different from each other, and thus, the user can easily recognize the abnormality by the afterimage.

On the other hand, when the difference between the observation image and the reference image is small, the difference image is selected as the display image, and the observation image and the difference image are switched and displayed. When the difference is small, the possibility of overlooking an abnormality is high, and therefore, by displaying a difference image directly showing the difference, it is possible to prevent a user from overlooking an abnormality.

Further, by performing switching of the observation image and the display image a plurality of times, it is possible to more accurately recognize an abnormality.

In the present embodiment, the control unit 3 is provided with the difference image generation unit 8 and the display image generation unit 9, but instead, as shown in fig. 3, the control unit 3 is provided with a reference image generation unit 7, a storage unit 10, and a display control unit 11, generates a reference image from the observation image by the reference image generation unit 7, and switches the observation image and the reference image a plurality of times by the display control unit 11 to output the images to the display unit 4.

As shown in fig. 4, the display image generation unit 9 may not be provided. In this case, the display control unit 11 may periodically switch the observation image, the reference image, and the difference image and output them to the display unit 4.

Further, although a network is constructed in which parameters capable of reproducing the forward-solving pattern are extracted in advance by learning using the forward-solving pattern, it is also possible to perform preprocessing of conventional image processing such as pattern matching using region extraction as necessary for a subject not having a repetitive structure. In general, since it is expected that the generalization performance and the learning speed are improved by eliminating the variation included in the learning data, the preprocessing is performed by normalization, non-correlation, whitening, or the like, but is not limited thereto.

In addition, although a learning-completed network that is learned in advance before observation is used, instead, learning may be performed by a user specifying an observation object immediately before observation starts.

In addition, the display image is selected based on the magnitude of the difference, but a user interface may be used instead to make a selection by the user.

The start and stop of the switching display of the images may be arbitrarily selected by the user, or the display images may be changed.

Next, an observation device 21 according to a second embodiment of the present invention will be described below with reference to the drawings.

In the description of the present embodiment, the same components as those of the observation device 1 of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted

As shown in fig. 5, the observation device 21 of the present embodiment includes a visual field movement monitoring unit (visual field movement state detection unit) 22 that monitors the stoppage of the visual field and the stoppage time based on the stage position information detected by a sensor such as an encoder provided in the microscope device 100.

As shown in fig. 6, after the observation image is acquired (step S1), the visual field movement monitoring unit 22 determines whether or not the visual field is stopped (step S10), and when the visual field is detected to be stopped, the visual field movement monitoring unit 22 counts the stop time (step S12), and causes the control unit 3 to start generating the reference image, the difference image, and the display image and to start switching between the display observation image and the display image. In a state where the field of view is not detected to be stopped, the control unit 3 is instructed to display the observation image itself on the display unit 4 (step S11), and the steps from step S1 are executed.

When receiving the information on the stop time from the visual field movement monitoring unit 22, the display control unit 11 determines the display method according to the size of the stop time (step S13). Specifically, as shown in fig. 7, it is determined whether or not the stop time T is equal to the first threshold value T_SThereafter (step S131), the stop time T is set to the first threshold value T_SIn the following case, the display control unit 11 switches the speed N_STo switch between the observation image and the display image (step S132). In addition, when the stop time is greater than the first threshold value T_SIn the case of (2), it is determined whether the stop time T is at the second threshold value T_MThen (step S133), the second threshold value T is set_MIn the following case, the display control unit 11 switches the speed N_MTo switch the observation image and the display image (step S134). Furthermore, when the stop time is greater than a second threshold value T_MAt a higher switching speed N than the switching speed N_MIs switched over to speed N_LTo switch between the observation image and the display image (step S135).

The stop time of the visual field relates to the ease of determination of a sense of incongruity resulting from an abnormality such as damage to an object in the observation visual field and a difference from an assumed correct solution pattern, and when the stop time is short, the abnormality is relatively easily found or whether the abnormality is easily determined.

On the other hand, if the field of view has a long stop time, the field of view is slightly damaged or defective or is determined to be hesitant.

According to the observation device 21 of the present embodiment, the longer the stop time, the more likely the afterimage feeling remains, and therefore, the display can be switched quickly, and the attractiveness can be improved.

The switching method is an example, and may be set according to the user's convenience, for example, by first switching the display at a high speed and then temporarily reducing the switching speed in a state where the stop time is long and the determination is hesitant.

In the present embodiment, the image selected as the display image is determined based on the magnitude of the difference between the observation image and the reference image, as in the first embodiment, but may be determined based on the stop time T of the field of view as shown in fig. 8 instead (step S14). In the example shown in fig. 8, the stop time T is a threshold value T_MIn the following case, the reference image is selected as the display image and is larger than the threshold value T_MIn the case of (2), the difference image is selected as the display image. Above a threshold value T_MIn the case of (3), the reference image and the difference image may be selected as the display image.

In a state where the stop time is long and the determination is hesitant, it is possible to confirm how much a minute abnormality appearing in the observation image appears as a difference. In addition, the final determination is easily performed by viewing the difference from the correct solution pattern again. The threshold for selecting a display image may be arbitrary.

The stop of the visual field and the stop time are monitored by the visual field movement monitoring unit 22, but in addition to this, as shown in fig. 9 and 10, the presence or absence of return of the visual field may be monitored (step S15). If the field of view moves in a predetermined movement pattern set in advance, it is determined that "no field of view has returned" (step S16), and if the field of view returns to the observation field of view for one-time observation, it is determined that "field of view has returned" (step S17).

When the return of the field of view has occurred (step S18), the display may be switched at the highest speed, unlike the display method when the return of the field of view has not occurred (step S19). Since the occurrence of the visual field returning is a hesitant state which is difficult to judge, it is possible to easily find the abnormality by the same setting as the setting in which the stop time is the longest.

As shown in fig. 11, the visual field movement monitoring unit 22 may monitor the movement speed of the visual field (step S20). It is determined whether or not the moving speed V is equal to or lower than a predetermined threshold value Vs (step S20), and when the moving speed V is equal to or lower than the predetermined threshold value Vs, the control unit 3 starts generating the reference image, the difference image, and the display image and starts switching between the observation image and the display image. When the moving speed V is greater than the threshold value Vs, an instruction is given to the control unit 3 to cause the display unit 4 to display the observation image itself (step S11).

When receiving the information on the movement speed V from the visual field movement monitoring unit 22, the display control unit 11 determines the display method according to the magnitude of the movement speed V (step S21). Specifically, as shown in fig. 12, it is determined whether or not the moving speed V is 0<V (step S211), when the moving speed V is 0<V (in detail, 0)<V ≦ Vs), the display control section 11 switches the speed N_STo switch between the observation image and the display image (step S212). When the moving speed V is 0, that is, the visual field is stopped, the display control unit 11 switches the speed N_MTo switch the observation image and the display image (step S213).

As a factor of the start of the decrease in the moving speed of the stage, there is a case where an image different from a normal image is found in the field of view. In this case, the attractiveness can be improved by switching between displaying the observation image and the display image to prevent overlooking.

Instead of changing the switching speed according to the stop time, the display control unit 11 may correct the differential image, which is the display image, according to the size of the stop time when the stop time information is received from the visual field movement monitoring unit 22 as shown in fig. 13. Specifically, as shown in fig. 14, it is determined whether or not the stop time T is at the threshold T_MThereafter (step S14), the stop time T is set to the threshold value T_MIn the following case, the correction of the display image is not performed, and the stop time T is greater than the threshold value T_MIn the case of (3), the display image generation unit 9 corrects the display image (step S22).

The correction of the display image is performed, for example, for saturation or contrast. By correcting the saturation or contrast to be high, the attractiveness can be improved. The display image generation unit (difference calculation unit) 9 may calculate the magnitude of the difference between the observation image and the reference image, and change the correction level of the saturation or the contrast according to the magnitude of the difference. If the saturation or contrast is higher as the difference is smaller, the attractiveness can be improved even for an abnormality such as a slight damage. Further, when the object is monochrome, correction for giving a pseudo color may be performed. Further, the reference image may be corrected without generating a difference image.

Description of the reference symbols

1. 21 Observation device

2 image obtaining part (imaging part)

4 display part

7 reference image generating unit

8 differential image generating section 9 display image generating section (image correcting section, differential calculating section) 11 display control section (display adjusting section)

22 View movement monitoring section (View movement State detecting section)