阅读说明：本技术 类似图像显示控制装置、系统及方法、显示控制装置、系统及方法、以及记录介质 (Similar image display control device, system and method, and recording medium ) 是由 松永和久 浜田玲 古贺弘志 皆川茜 于 2019-08-23 设计创作，主要内容包括：本发明的类似图像显示控制装置更清楚地显示类似图像之间的关系。类似图像显示装置具备：类似图像获取部,其获取根据对查询图像进行类似图像检索的结果而得到的类似图像；类别设定部,其设定对由类似图像获取部获取的类似图像进行分类的多个类别；位置决定部,其按照上述维数的种类属性来决定表示类别区域的位置的坐标,该类别区域为两个以上的预定维数的维度空间中的表示由类别设定部设定的各类别的区域；分类部,其将由类似图像获取部获取的类似图像分类为由类别设定部设定的多个类别中的任一种；以及图像显示控制部,其将由分类部分类为各类别的类似图像配置于位于由位置决定部所决定的坐标示出的位置处的类别区域的内部并显示于显示部。(The similar image display control apparatus of the present invention displays the relationship between similar images more clearly. The similar image display device includes: a similar image acquisition unit that acquires a similar image obtained from a result of performing similar image search on a query image; a category setting unit that sets a plurality of categories for classifying the similar images acquired by the similar image acquisition unit; a position determination unit that determines coordinates indicating a position of a category region in accordance with the category attribute of the dimension, the category region being a region indicating each category set by the category setting unit in a dimensional space of two or more predetermined dimensions; a classification unit that classifies the similar image acquired by the similar image acquisition unit into any one of a plurality of categories set by the category setting unit; and an image display control unit that arranges the similar images classified into the respective categories by the classification unit inside the category area located at the position indicated by the coordinates determined by the position determination unit and displays the similar images on the display unit.)

1. A similar image display control apparatus is characterized in that,

the similar image display control device includes:

a similar image acquisition unit that acquires a similar image obtained from a result of performing similar image retrieval on the query image;

a category setting unit that sets a plurality of categories for classifying the similar images acquired by the similar image acquisition unit;

a position determining unit that determines coordinates indicating a position of a category region in accordance with two or more category attributes having a predetermined dimension, the category region being a region in the dimensional space of the dimension that indicates each category set by the category setting unit;

a classification unit that classifies the similar image acquired by the similar image acquisition unit into any one of a plurality of categories set by the category setting unit; and

and an image display control unit that arranges the similar images classified into the respective categories by the classification unit inside the category area located at the position indicated by the coordinates determined by the position determination unit, and displays the similar images on a display unit.

2. The similar image display control device according to claim 1,

the image display control means displays the similar image by arranging the similar image inside the category region based on the similarity between the similar image and the query image.

3. The similar image display control device according to claim 1,

the image display control means displays the category region in a larger circle as the number of similar images classified into the category corresponding to the category region increases.

4. The similar image display control device according to claim 3,

the image display control means displays a circumferential line of a circle indicating the category region as the image display control means displays the larger the degree of similarity between the query image and the similar image most similar to the query image among the similar images classified into the categories corresponding to the category regions.

5. The similar image display control device according to claim 3,

the image display control means concentrically arranges and displays the similar images in the category regions.

6. The similar image display control device according to claim 5,

the image display control means displays the similar image at a position closer to the center of the category region as the similarity between the similar image and the query image increases.

7. The similar image display control device according to claim 5,

the image display control means displays a concentric pattern shape on a background of the similar image arranged in the concentric shape and displayed.

8. The similar image display control device according to claim 1,

the position determining means associates the attributes with respective coordinate axes of the space, and determines coordinates indicating the position of the category region as coordinates corresponding to an attribute value of the attribute of the category of the space.

9. The similar image display control device according to claim 1,

the image display control means displays the image by connecting a connection line based on an attribute of a category corresponding to the category region.

10. The similar image display control device according to claim 1,

the image display control means displays the similar image on a display unit by a tree structure in which the query image is a root node, the category region located at the position indicated by the determined coordinates is a leaf node, and a connection line based on the attribute of the category corresponding to the category region is connected from the root node to the leaf node, and displays an attribute name indicating the attribute information of the category as an internal node of the connection line.

11. The similar image display control device according to claim 9,

the image display control means displays, on the display unit, the thickness of the connecting line connected to the category region corresponding to the category into which the similar image is classified, in a predetermined thickness, based on the similarity between a predetermined similar image and the query image among the plurality of similar images classified into the category corresponding to the category region.

12. The similar image display control device according to claim 1,

the image display control means causes the display unit to display one or more similar images selected by the user among the query image and the similar images in an enlarged manner.

13. The similar image display control device according to any one of claims 1 to 12,

the above categories are names of skin diseases.

14. The similar image display control device according to claim 13,

the above attributes are both benign/malignant and melanocyte-like/non-melanocyte-like.

15. A similar image display control system comprising a similar image display control device and a display section,

the similar image display control device includes:

a similar image acquisition unit that acquires a similar image obtained from a result of performing similar image retrieval on the query image;

a category setting unit that sets a plurality of categories for classifying the similar images acquired by the similar image acquiring unit;

a position determining unit that determines coordinates indicating a position of a category region in accordance with two or more category attributes having a predetermined dimension, the category region being a region in the dimensional space of the dimension that indicates each category set by the category setting unit;

a classification unit that classifies the similar image acquired by the similar image acquisition unit into any one of a plurality of categories set by the category setting unit; and

and an image display control unit that arranges the similar images classified into the respective categories by the classification unit inside the category area located at the position indicated by the coordinates determined by the position determination unit, and displays the similar images on the display unit.

16. A similar image display control method is characterized in that,

the similar image display control method includes:

a similar image acquisition step of acquiring a similar image obtained from a result of performing similar image retrieval on the query image;

a classification step of classifying the similar image acquired in the similar image acquisition step into any one of a plurality of categories; and

and an image display control step of determining coordinates indicating positions of category regions in accordance with two or more category attributes of a predetermined dimension, and displaying the similar images classified into the categories in the classification step, which are regions indicating the categories in the dimensional space, on a display unit while arranging the similar images in the category regions at positions indicated by the determined coordinates.

17. A recording medium storing a program, characterized in that,

the program causes a computer to execute the steps of:

a similar image acquisition step of acquiring a similar image obtained from a result of performing similar image retrieval on the query image;

a classification step of classifying the similar image acquired in the similar image acquisition step into any one of a plurality of categories; and

and an image display control step of determining coordinates indicating positions of category regions in accordance with two or more category attributes of a predetermined dimension, and displaying the similar images classified into the categories in the classification step, which are regions indicating the categories in the dimensional space, on a display unit while arranging the similar images in the category regions at positions indicated by the determined coordinates.

18. A display control apparatus is characterized in that,

the display control device includes:

an acquisition unit that acquires a malignancy indicator indicating a likelihood that a disease attribute of a diagnosis target portion is malignant and a first disease attribute indicator indicating a likelihood that the disease attribute of the diagnosis target portion is a predetermined first disease attribute; and

and a display control unit that displays the acquired malignancy index and the acquired first disease attribute index on a display unit in association with each other.

19. The display control apparatus according to claim 18,

the display control device further includes a risk acquisition unit that acquires a risk index indicating whether or not the risk of the disease is high when the disease attribute is malignant and the disease attribute is the first disease attribute,

the display control means displays the acquired risk index on the display unit in association with the acquired malignancy index and the acquired first disease attribute index.

20. The display control apparatus according to claim 18,

the acquiring means further acquires a benign indicator indicating a possibility that the disease attribute of the diagnosis target portion is benign and a second disease attribute indicator indicating a possibility that the disease attribute of the diagnosis target portion is a second disease attribute different from the first disease attribute,

the display control means displays the acquired malignancy index, the acquired first disease attribute index, the acquired benign index, and the acquired second disease attribute index on the display unit in association with one another.

21. The display control apparatus according to claim 20,

the display control device further includes a disease risk acquisition means for acquiring a risk index indicating whether or not the disease risk of the diagnosis target portion is high,

the display control unit displays the acquired risk index, the acquired malignancy index, the acquired first disease attribute index, the acquired benign index, and the acquired second disease attribute index on a display unit in association with each other.

22. The display control apparatus according to claim 20,

the display control device further includes a position determination unit that determines position coordinates for displaying information related to a disease,

the acquisition unit further acquires a disease index indicating a possibility that the disease of the diagnosis target portion is a predetermined disease,

the display control means displays the acquired indexes on the display unit in a manner of being associated with each other by a tree structure in which a query image is set as a root node, a probability circle based on the size of the acquired disease index located at a position indicated by the determined coordinates is set as a leaf node, and a connection line based on a disease attribute of the diagnosis target portion is connected from the root node to the leaf node.

23. The display control apparatus according to any one of claims 20 to 22,

the diagnostic target part is skin, the first disease attribute is a melanocyte class, and the second disease attribute is a non-melanocyte class.

24. The display control apparatus according to claim 18,

the display control device further includes a recognizer that outputs probabilities that the disease of the diagnosis target portion is associated with the attributes,

the acquisition unit acquires, as an index for each of the attributes, a probability associated with each of the attributes that are output.

25. A display control system includes a display control device and a display unit,

it is characterized in that the preparation method is characterized in that,

the display control device includes:

an acquisition unit that acquires a malignancy indicator indicating a likelihood that a disease attribute of a diagnosis target portion is malignant and a first disease attribute indicator indicating a likelihood that the disease attribute of the diagnosis target portion is a predetermined first disease attribute; and

and a display control unit that displays the acquired malignancy index and the acquired first disease attribute index on the display unit in association with each other.

26. A display control method characterized by comprising, in a display control unit,

the display control method includes:

an acquisition step of acquiring a malignancy indicator indicating a likelihood that a disease attribute of a part to be diagnosed is malignant and a first disease attribute indicator indicating a likelihood that the disease attribute of the part to be diagnosed is a predetermined first disease attribute; and

a display control step of displaying the acquired malignancy index and the acquired first disease attribute index on a display unit in association with each other.

27. A recording medium storing a program, characterized in that,

the program causes a computer to execute the steps of:

an acquisition step of acquiring a malignancy indicator indicating a likelihood that a disease attribute of a part to be diagnosed is malignant and a first disease attribute indicator indicating a likelihood that the disease attribute of the part to be diagnosed is a predetermined first disease attribute; and

a display control step of displaying the acquired malignancy index and the acquired first disease attribute index on a display unit in association with each other.

Technical Field

The invention relates to a similar image display control device, a similar image display control system, a similar image display control method, a display control device, a display control system, a display control method, and a recording medium.

Background

In dermatology, diagnosis of skin diseases requires skill, and is a rather difficult task. In contrast, recently, a technique has been developed in which an affected part is imaged and the imaged image is analyzed by a computer. A database of a large number of cases is created, a similar image search is performed using an image of the affected part of the patient as a query image, and diagnosis is performed with reference to the similar cases.

As an apparatus for displaying similar images, for example, japanese patent application laid-open No. 2010-250529 describes an image retrieval apparatus of one of: similar images similar to the query image are extracted from images registered in the image database, the similar images are arranged around the query image, and a search result of linkage display between the query image and the similar images is presented to a display unit or the like.

In addition, in order to assist diagnosis, a technique for determining the benign or malignant nature of the affected area has been developed. For example, in "Nevisense-a breakthrough in non-innovative detection of melanoma", [ online ], [ search for 6/14/2019 ], Internet < URL: https:// scibase. com/the-nevisense-product /) describes a diagnosis assistance device that visually provides the degree of benign/malignancy of a skin disease with one-axis information. The image search device described in japanese patent application laid-open No. 2010-250529 displays a connection line and an image similar to a query image together around the query image, and when a new query image is specified, it is possible to additionally display the connection line and the image in the past search result.

In addition, "Nevisense-a breakthrough in non-innovative detection of melanoma", [ online ], [ search for 6/14 in 2019 ], Internet < URL: the diagnosis support apparatus described in https:// scibase. com/the-nevisense-product/> provides information that enables the degree of benign/malignancy of a part to be diagnosed to be visually grasped, but has a problem that it is difficult to grasp disease attribute information of the part to be diagnosed only by grasping the degree of benign/malignancy.

Disclosure of Invention

The present invention has been made to solve the above-described problems, and a first object of the present invention is to provide a similar image display control device, a similar image display control system, a similar image display control method, and a recording medium, which can display the relationship between similar images more clearly than before.

A second object of the present invention is to provide a display control device, a display control system, a display control method, and a recording medium that can display disease attribute information of a diagnosis target portion in an easily grasped manner.

In order to achieve the first object, a similar image display control device according to the present invention includes:

a similar image acquisition unit that acquires a similar image obtained from a result of performing similar image retrieval on the query image;

a category setting unit that sets a plurality of categories for classifying the similar images acquired by the similar image acquisition unit;

a position determining unit that determines coordinates indicating a position of a category region in accordance with two or more category attributes having a predetermined dimension, the category region being a region in the dimensional space of the dimension that indicates each category set by the category setting unit;

a classification unit that classifies the similar image acquired by the similar image acquisition unit into any one of a plurality of categories set by the category setting unit; and

and an image display control unit that arranges the similar images classified into the respective categories by the classification unit inside the category area located at the position indicated by the coordinates determined by the position determination unit, and displays the similar images on a display unit.

In order to achieve the second object, a display control device according to the present invention includes:

an acquisition unit that acquires a malignancy indicator indicating a likelihood that a disease attribute of a diagnosis target portion is malignant and a first disease attribute indicator indicating a likelihood that the disease attribute of the diagnosis target portion is a predetermined first disease attribute; and

and a display control unit that displays the acquired malignancy index and the acquired first disease attribute index on a display unit in association with each other.

Effects of the invention

According to the present invention, the relationship between similar images can be displayed in a more easily understandable manner or the disease attribute information of a diagnosis target portion can be displayed in an easily grasped manner.

Drawings

Fig. 1 is a diagram showing a functional configuration of a similar image display device according to a first embodiment of the present invention.

Fig. 2 is a diagram showing an example of determination of the position of the category by the position determination unit according to the first embodiment.

Fig. 3 is a diagram showing an example of similar image display by the image display control unit according to the first embodiment.

Fig. 4 is a flowchart of similar image display processing of the similar image display device according to the first embodiment.

Fig. 5 is a diagram showing an example of a comparative display screen according to the first embodiment.

Fig. 6 is a diagram showing an example of similar image display by the image display control unit according to the first modification of the present invention.

Fig. 7 is a diagram showing an example of similar image display by the image display control unit according to the second modification of the present invention.

Fig. 8 is a diagram showing a functional configuration of a display control device according to a second embodiment of the present invention.

Fig. 9 is a diagram showing an example of display performed by the display control device according to the second embodiment.

Fig. 10 is a flowchart of a display control process of the display control device according to the second embodiment.

Fig. 11 is a flowchart of risk boundary line generation processing in the display control device according to the second embodiment.

Fig. 12 is a diagram showing a functional configuration of a display control device according to a third embodiment of the present invention.

Fig. 13 is a diagram showing an example of display performed by the display control device according to the third embodiment.

Fig. 14 is a flowchart of a display control process of the display control device according to the third embodiment.

Fig. 15 is a diagram showing a functional configuration of a display control device according to a fourth embodiment of the present invention.

Fig. 16 is a diagram showing an example of display performed by the display control device according to the fourth embodiment.

Fig. 17 is a flowchart of a display control process of the display control device according to the fourth embodiment.

Detailed Description

Hereinafter, a similar image display device and the like according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding portions are denoted by the same reference numerals.

(first embodiment)

The similar image display apparatus 100 according to the first embodiment of the present invention collects similar images obtained as a result of performing similar image search on a query image for each predetermined category, and arranges the similar images in the category according to the degree of similarity with the query image. The categories for collecting and arranging the similar images are arranged and displayed in an n-dimensional space defined by a predetermined axis, thereby displaying the relationship between the similar images in an easily understandable manner. The mechanism for performing such display is explained below.

As shown in fig. 1, the similar image display device 100 according to the first embodiment includes a control unit 10, a storage unit 20, an input unit 31, an output unit 32, and a communication unit 33.

The control Unit 10 is constituted by a CPU (Central Processing Unit) or the like, and executes a program stored in the storage Unit 20 to realize functions of each Unit (the similar image acquiring Unit 11, the category setting Unit 12, the position determining Unit 13, the classifying Unit 14, and the image display control Unit 15) described later.

The storage unit 20 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores programs executed by the CPU of the control unit 10 and necessary data.

The input section 31 is a device (keyboard, mouse, touch panel, camera, etc.) for the user of the similar image display apparatus 100 to input an instruction applied to the similar image display apparatus 100 or to input a query image. The control unit 10 acquires an instruction from the user or an inquiry image via the input unit 31. As the input unit 31, any device can be used if the control unit 10 can acquire an instruction from the user or an inquiry image. However, the control unit 10 may acquire the query image via the communication unit 33. The query image is image data that is input by the user when searching for a similar image displayed on the similar image display device 100. The similar image display apparatus 100 presents a plurality of images similar to the query image to the user in an easily understandable manner.

The output section 32 is a device (display, interface for display, etc.) for the control section 10 to present similar images to the user. The similar image display device 100 may include a display (display unit) as the output unit 32, or may display a search result or the like on an external display connected via the output unit 32. Note that the similar image display device 100 not provided with a display (display unit) (the similar image display device 100 in which the output unit 32 serves as an interface for the display (display unit)) is also referred to as a similar image display control device.

The communication unit 33 is a device (such as a network interface) for transmitting and receiving data to and from an external device (such as a server storing a database of image data, a similar image search device, and the like). The control unit 10 can acquire the query image or an image similar to the query image via the communication unit 33.

Next, the function of the control unit 10 will be described. The control unit 10 realizes the functions of the similar image acquiring unit 11, the category setting unit 12, the position determining unit 13, the classifying unit 14, and the image display control unit 15.

The similar image acquiring unit 11 acquires data obtained as a result of performing similar image search on the query image (the similarity between the image data of the similar image and the query image of the image). Specifically, in the similar image retrieval, image data having a similarity to the query image of a predetermined threshold or more is acquired together with the similarity. The similar image acquiring unit 11 may acquire data of similar images obtained as a result of the control unit 10 searching for images similar to the query image, or may, for example, cause an external similar image searching device to search for images similar to the query image via the communication unit 33 and acquire data of similar images searched by the similar image searching device. Each image data is added with information such as a disease name corresponding to the image as tag information. The similar image acquiring unit 11 functions as a similar image acquiring unit.

The category setting unit 12 sets a category group (a plurality of categories) for classifying the image data acquired by the similar image acquiring unit 11. The category group refers to, for example, a disease name (pigmented nevus, melanoma, basal cell carcinoma, etc.), an outer shape (circle, star, ellipse, etc.), a color tone (red, black, brown, etc.), a size, an inner structure, a state of nevus (pigmented spots) (a mesh pattern, a small sphere pattern, a cobblestone pattern, a uniform pattern, a parallel pattern, a starburst pattern, a multi-structure pattern, a non-specific pattern, etc.) and the like in the case of targeting image data of the skin. For example, when disease names are set as a category group, a pigmented nevus, melanoma, basal cell carcinoma, and the like, which are specific disease names, are classified into one category. The information of the classified category group (a plurality of categories) is stored in the storage unit 20 in advance, and the category setting unit 12 sets the category group (a plurality of categories) into which the image data is classified, based on the information of the category group stored in the storage unit 20. The category setting unit 12 functions as a category setting means.

The position determination unit 13 determines, as coordinates in the n-dimensional space, positions indicating regions indicating the respective categories included in the category group (categories) set by the category setting unit 12, according to n types (n is an integer equal to or greater than 1) of attributes. More specifically, n attributes are associated with n coordinate axes defining the coordinates of the n-dimensional space, respectively, one for one, and coordinates showing the positions of regions (category regions) for displaying the categories are determined based on the attribute values of the attributes associated with the coordinate axes in each coordinate axis.

For example, consider a case where the category group set by the category setting unit 12 is a disease name, and the position determination unit 13 determines the position in the two-dimensional space of the category group (disease name) with two attributes, i.e., "benign/malignant" and "melanocyte type/non-melanocyte type". In this case, for example, as shown in fig. 2, the position determination unit 13 determines coordinates of a position for displaying each disease name in a two-dimensional space in which "benign/malignant" is taken as a vertical axis (Y axis) and "melanocyte class/non-melanocyte class" is taken as a horizontal axis (X axis). Here, in the vertical axis (Y axis), benign is set to the lower side and malignant is set to the upper side, and in the horizontal axis (X axis), melanocytes are set to the left side and non-melanocytes are set to the right side.

Specifically, when five types of pigmented nevi, melanoma, seborrheic keratosis, hematoma/hemangioma, and basal cell carcinoma are considered as disease names, among the disease attributes, pigmented nevi are "benign, melanocyte type", melanoma is "malignant, melanocyte type", seborrheic keratosis is "benign, nonmelanocyte type", hematoma/hemangioma is "benign, nonmelanocyte type", and basal cell carcinoma is "malignant, nonmelanocyte type". Therefore, as shown in fig. 2, the position determination unit 13 determines the respective positions so that the pigmented nevus 201 is displayed in the lower left region, the melanoma 202 is displayed in the upper left region, the seborrheic keratosis 203 is displayed in the lower right region (slightly left of the center of the region), the hematoma/hemangioma 204 is also displayed in the lower right region (slightly right of the center of the region), and the basal cell carcinoma 205 is displayed in the upper right region.

Further, the position determination unit 13 may adjust the display position of the category as necessary so as not to display the positions of different categories at the same coordinate. For example in the example shown in fig. 2, both seborrheic keratosis 203 and hematoma/hemangioma 204 are "benign, non-melanocytic type", so both categories are displayed in the same lower right corner area when the display position is not adjusted. Therefore, in the example shown in fig. 2, the position determination unit 13 adjusts the display position so that the seborrheic keratosis 203 is displayed at a position slightly displaced to the left from the center of the area in the lower right corner and the hematoma/hemangioma 204 is displayed at a position slightly displaced to the right from the center of the area in the lower right corner.

N attribute information for determining the coordinate axis of the space for determining the display position of each category by the position determination unit 13, information on the attribute of each category, and arrangement information of each attribute are stored in the storage unit 20 in advance. The position determination unit 13 determines coordinates in an n-dimensional space for displaying the positions of the category group (a plurality of categories) based on the n-attribute information stored in the storage unit 20, the attribute information of each category, and the arrangement information of each attribute. In the example shown in fig. 2, two kinds of attribute information, i.e., an attribute of "benign/malignant" and an attribute of "melanocyte class/non-melanocyte class", are stored in the storage unit 20 as the attribute information. As information on the attributes of each category, information that the pigmented nevus 201 is "benign, melanocyte-like", the melanoma 202 is "malignant, melanocyte-like", the seborrheic keratosis 203 is "benign, nonmelanocyte-like", the hematoma/hemangioma 204 is "benign, nonmelanocyte-like", and the basal cell carcinoma 205 is "malignant, nonmelanocyte-like" is stored in the storage unit 20. Further, as the arrangement information of the attributes, information that "benign" is arranged on the lower side and "malignant" is arranged on the upper side in the attribute of "benign/malignant", and "melanocyte type" is arranged on the left side and "non-melanocyte type" is arranged on the right side in the attribute of "melanocyte type/non-melanocyte type" is stored in the storage unit 20. The position determination unit 13 functions as a position determination means.

The classification unit 14 classifies the image data acquired by the similar image acquisition unit 11 into any one of the group (plurality of classes) of classes set by the class setting unit 12. The classification unit 14 can classify the image data using label information attached to each image data (for example, attaching a disease name to each image data as label information). The classification unit 14 functions as a classification means.

The image display control unit 15 arranges the image data classified into each category by the classification unit 14 based on the similarity with the query image in an area indicating the category of the coordinates determined by the position determination unit 13 in the n-dimensional space, and displays the image data via the output unit 32. The image display control unit 15 arranges and displays a similar image classified as a mole in an area 301 of the mole (in the circle at the lower left of fig. 3), an image classified as a melanoma in an area 302 of the melanoma (in the circle at the upper left of fig. 3), and an image classified as a seborrheic keratosis in an area 303 of the seborrheic keratosis (in the circle at the lower right of fig. 3), for example, as shown in fig. 3, so that the higher the degree of similarity with the query image 300, the closer the center of each area (in the circle), the similar image classified into hematoma/hemangioma is arranged and displayed in a region 304 of hematoma/hemangioma (in a circle slightly to the right in the lower right of fig. 3), and the similar image classified into basal cell carcinoma is arranged and displayed in a region 305 of basal cell carcinoma (in a circle in the upper right of fig. 3). The image display control unit 15 functions as an image display control means.

In the above, the functional configuration similar to the image display apparatus 100 is explained. Next, the content of the similar image display processing performed by the similar image display apparatus 100 will be described with reference to fig. 4. The similar image display processing is started when the user instructs the similar image display apparatus 100 to start the similar image display processing via the input section 31.

First, the control section 10 of the similar image display apparatus 100 acquires a query image (step S101). For example, when the user inputs the query image to the similar image display apparatus 100 via the input section 31 (for example, drags and drops the query image to a specified area of the screen), the control section 10 acquires the query image.

Next, the similar image acquiring unit 11 acquires a similar image obtained as a result of performing a similar image search on the query image (step S102). Specifically, in the similar image retrieval, a similar image having a similarity to the query image of a predetermined threshold or more is acquired. At this time, the similar image acquiring unit 11 acquires the similarity between the similar image and the query image together with the similar image. Step S102 is also referred to as a similar image acquisition step. The similar image search process may be performed by an external similar image search device instead of the similar image display device 100. In this case, the control unit 10 transmits the query image acquired in step S101 to the similar image search device via the communication unit 33, and the similar image acquisition unit 11 acquires the result of the similar image search performed by the similar image search device.

Then, the classification unit 14 classifies the similar images acquired by the similar image acquisition unit 11 into the categories set by the category setting unit 12, based on the label information attached to each similar image (step S103). Step S103 is also referred to as a classification step.

Next, the image display control unit 15 arranges the similar images classified into the respective categories at step S103 in the regions of the respective categories at the positions determined by the position determination unit 13, and displays the images via the output unit 32 (step S104). Specifically, as shown in fig. 3, in each category of area, the images having a higher similarity to the query image are arranged concentrically with each other at the center of the area of the category. In the example shown in fig. 3, of the similar images classified into the respective categories, the image having the highest similarity with the query image is arranged at the center of the category, and the images having the second and subsequent degrees of similarity are arranged clockwise from the upper side thereof, so that the images are arranged concentrically.

In step S104, the image display control unit 15 displays a circle having a size corresponding to the number of similar images classified into each category in the area of each category. By displaying the circle, the scale of each category can be easily and intuitively grasped. Then, in the image display control unit 15, the thickness of the circumference line of the circle is displayed to be thicker as the degree of similarity between the center image (the similar image having the highest degree of similarity with the query image in the category) and the query image is higher. By making the thickness of the circle thick in this way, the user can intuitively grasp the arrangement position of the similar image most similar to the query image. The image display control unit 15 may display the thickness of the line on the circumference of the circle not limited to the center image, but may display the thickness of the line on the basis of the similarity between the predetermined image (for example, an image having a high nth similarity (n is an integer equal to or greater than 1 and equal to or less than the number of similar images classified into the category) in the category and the query image, the lowest image, or the middle image when the images are arranged in the order of similarity) and the query image (for example, the thickness increases as the similarity increases, and the thickness decreases as the similarity decreases). In order to allow the user to easily compare each type of similar image with the query image, the image display control unit 15 further performs a process of displaying the query image 300 in the center of the display screen in step S104, as shown in fig. 3. Step S104 is also referred to as an image display control step.

Next, the control unit 10 determines whether or not the similar image displayed in step S104 is selected (for example, clicked by the user) via the input unit 31 (step S105). If a similar image is not selected (step S105: NO), the flow proceeds to step S108.

If similar images are selected (step S105: YES), the image display control section 15 displays these images in an enlarged manner so that the image selected in step S105 can be compared with the inquiry image (step S106). For example, when the central image of a mole (the image most similar to the query image among similar images classified into moles) in fig. 3 is clicked as the comparison target image, as shown in fig. 5, a comparison display screen on which the query image 51 and the clicked comparison target image 52 are displayed in an enlarged manner is displayed via the output unit 32. Fig. 5 also shows that the image display control section 15 displays, on the lower side of the comparison target image 52, the tag information 53 attached to the comparison target image, the order 54 of the degree of similarity between the comparison target image 52 and the inquiry image 51, and the forward button 55 and the backward button 56 for switching the comparison target image 52 to the order of the degree of similarity.

Then, the image display control unit 15 performs image display in accordance with the user operation (step S107). For example, the image display control unit 15 moves the image in parallel by performing a drag operation on the upper side of the query image 51 or the comparison target image 52, enlarges or reduces the image by rotating the wheel, and returns to the query image display screen shown in fig. 3 by double-clicking on the upper side of the query image 51. Further, the image display control unit 15 switches the comparison target image to the order of similarity with the query image when the forward button 55 or the backward button 56 is clicked.

Next, the control unit 10 determines whether or not the end of the similar image display processing is instructed (step S108). If the end of the similar image display processing is not instructed (step S108: NO), it returns to step S107. If the end of the similar image display processing is instructed (step S108: YES), the similar image display processing is ended. For example, if the user instructs the end of the similar image display processing via the input section 31, the similar image display processing is ended.

As described above, the similar image display apparatus 100 can classify similar images into categories, and arrange and display similar images in descending order of similarity with the query image for each category, and therefore can more clearly display the relationship between similar images.

For example, in the case of an image showing a skin disease, melanoma, basal cell carcinoma, and solar keratosis are all malignant diseases, but the degree of malignancy (the degree of influence on the human body) is greatly different. Therefore, for example, as the attribute information of each category, for example, if the melanoma is "malignancy 10, melanocyte type", basal cell carcinoma is "malignancy 8, non-melanocyte type", solar keratosis is "malignancy 3, non-melanocyte type", the information of malignancy (attribute value) is also stored in the storage unit 20, and when the position determination unit 13 determines the position of the category so that, for example, a circle of the category is displayed on the upper part of the screen in accordance with the category having the higher malignancy, the user can confirm the similar image placed in each category together with the malignancy. Similarly, the other attributes determine the position based on the attribute values of the attributes, and the user can confirm the similar image based on the attribute values of the attributes. These are always shown as examples and do not necessarily mean medically correct. These display positions can be appropriately changed according to the mind and situation of a user such as a doctor who resembles the image display apparatus 100.

(first modification)

In the first embodiment described above, the display of fig. 3 is performed in the similar image display processing, but a first modification for making the similar relationship easier to understand will be described with reference to fig. 6.

In the similar image display device 100 according to the first modification, in step S104 of the similar image display processing (fig. 4), the image display control unit 15 performs the following processing. (in the same manner as in the first embodiment, the larger the number of similar images classified into the category, the larger the size of the circle drawn in the region of each category, for example, as shown in fig. 6, the larger the category circle 311 of the pigmented nevus than the category circle 312 of the melanoma.)

-rendering the background of the circles rendered in the area of each category dense in the center and thinner towards the outside. For example, as shown in fig. 6, the concentric figure shapes 311a, 311b, 311c, and 311d from the center side are displayed on the background of the circle drawn in the category region of the category circle 311 of the mole. In fig. 6, the density is changed in 2 to 4 steps according to the size of each type of circle, but the density may be changed smoothly (gradually) without providing such steps.

Connecting from the query image to the central image of each category by connecting lines.

The thickness of the connecting line is thicker as the similarity between the similar image (the similar image most similar to the query image in the category) arranged at the center of the category to which the connecting line is connected and the query image is higher. For example, the connection line 321 to the type circle 311 of the pigmented nevus is thicker than the connection line 322 to the type circle 312 of the melanoma. (the thickness of the connecting line is not limited to the center image, and may be displayed with a predetermined thickness (for example, the thickness increases as the similarity increases, and the thickness decreases as the similarity decreases) based on the similarity between the predetermined image (for example, an image having a high similarity to the query image in the category (n is an integer equal to or greater than 1 and equal to or less than the number of similar images classified into the category), the lowest image, or the image centered when the images are arranged in the order of similarity) and the query image).

Between both ends of the connection lines 321, 322, 323, 324, 325, attribute information used when the position determining unit 13 determines the position of each category is displayed. For example, attribute information such as malignancy 332 and melanocyte class 334 is shown in the category of melanoma.

The regions representing the respective categories are connected as leaf nodes via a tree structure having the query image as a root node by the connecting lines 321, 322, 323, 324, 325. That is, similar images and the like are displayed by a tree structure having the query image as a root node, the regions representing the respective categories as leaf nodes, and connecting lines from the root node to the leaf nodes based on the attributes of the categories corresponding to the regions representing the respective categories. Further, as internal nodes of the connection lines 321, 322, 323, 324, 325 connected to the respective categories, attribute names (benign 331, malignant 332, melanocyte classes 333, 334, non-melanocyte classes 335, 336, etc.) indicating attribute information of the categories are displayed.

In the case where special attention is desired, the attribute information is displayed largely. (for example, in the case of displaying similar images to be displayed on the image data of a skin disease, if there are more similar images that are malignant than similar images that are benign, then the attribute information shows the malignant 332 larger than the benign ones.furthermore, in fig. 6, since there are more similar images that are benign, the malignant 332 is displayed in the same size as the benign 331).

-each similar image is displayed surrounded by a smaller circle, the higher the similarity of the similar image to the query image, the thicker the thickness of its smaller circle line. For example, the line thickness of the small circle 3141 of the similar image arranged at the center of the classification circle 314 of the hematoma/hemangioma is thicker than the line thickness of the small circle 3142 of the similar image arranged at the periphery thereof.

In step S104 of the similar image display processing (fig. 4), the image display control unit 15 performs the above processing, and displays the similar image shown in fig. 6, for example. By performing such display, the following effects are obtained.

The respective category circles 311, 312, 313, 314, 315 are arranged in the n-dimensional space in accordance with the n attributes, whereby the user can intuitively grasp the nature of the query image.

As the number of pieces classified into the category (number of search pieces) increases, the larger the respective category circles 311, 312, 313, 314, 315 are displayed, whereby the user can intuitively grasp the number of search pieces.

The similar image arranged at the center of the category (similar image most similar to the query image in the category) is displayed thicker in the thickness of the connecting lines 321, 322, 323, 324, 325 to each category as the degree of similarity with the query image is higher, whereby the user can intuitively grasp the arrangement position of the similar image most similar to the query image.

The background of the category circle drawn in the area of each category is drawn to be dense in the center and thinner toward the outside, whereby the user can visually recognize that the importance of the case on the side of the center of the circle is high.

(second modification)

In the first embodiment, the image display control unit 15 concentrically arranges the similar image search results for each category, but the present invention is not limited to this, and may be a radial shape, an elliptical shape, a quadrilateral shape, or the like. For example, when the images are arranged in a rectangular shape, as shown in fig. 7, the images can be displayed more compactly, and all similar images can be displayed with good viewability at the same time even when the number of search results is large. In the example shown in fig. 7, the image display control unit 15 increases the size of the quadrangles 351, 352, 353, 354, 355 of each category as the number of similar images classified into the category increases, arranges the similar images on the right side in descending order of the degree of similarity with the query image 300 from the upper left corner within the quadrangle, and returns to the left end if reaching the right end, and arranges and displays the similar images from the lower side.

In the first embodiment, the description has been given of the case where the number of attributes used when determining the position by the position determining unit 13 is two, and the two attributes are associated with the two axes (X axis and Y axis) of the two-dimensional space to determine the coordinates in the two-dimensional space of the display position of the similar image search result, but the present invention is not limited to this. For example, the number of attributes used for determining the position may be one, and each category may be arranged on a straight line (one-dimensional space). In this case, although the categories are arranged on a straight line, similar images within the categories are arranged concentrically, and are therefore arranged in a two-dimensional space.

Further, the number of attributes used in determining the position may be three, and each category may be arranged in a three-dimensional space. In this case, although the category and the similar image are arranged in a three-dimensional space, they can be displayed on a normal display by outputting a portion projected in a two-dimensional space when being output to the output unit 32. When the number n of types of attributes used in determining a position is set to 4 or more, each type may be virtually arranged in an n-dimensional space and finally projected in a two-dimensional space to be output. Further, as the attribute type, not only the above-described "benign/malignant" and "melanocyte type/nonmelanocyte type" may be used, but also "epithelial type/nonepithelial type", "metastatic type/nonmetastatic type", "invasive/noninvasive type", "viral/nonviral type", "size (for example, a major diameter of an ellipse circumscribing a lesion)", "ellipticity (for example, ellipticity of an ellipse circumscribing a lesion)", "lesion area (area of a lesion)", "length of a contour line (length of a contour line of an outer edge of a lesion)", "depth of a tumor (for example, determined by color (light black, and dark brown, gray blue, and light steel color as the contour line becomes darker)", "color of a lesion (for example, arranged on a color axis in accordance with the depth of a tumor)", "shape (for example, using moment (coordinate value of a lesion area, coordinate value of a color axis, and the like), and" shape (for example, using moment (for a, Coordinate values of the outline of the lesion region, pixel values of the lesion region, and the like, which are calculated by performing a moment calculation), "time (for example, when time is represented on the horizontal axis and a measurement value such as a size is represented on the vertical axis, a temporal change in the measurement value such as the size can be observed)" and the like.

In the first embodiment, although the skin disease is described as an example, the present invention is not limited to the field of dermatology, and can be widely applied to the field of displaying similar images. For example, the present invention can be applied to similarity search of flower images, similarity search of bacterial micrographs, and the like.

In the first embodiment, the control unit 10 performs the similar image display processing, but the communication unit may receive the result of the processing performed by the external server and output the result to the output unit 32.

The first embodiment and the first and second modifications can be appropriately combined. For example, by combining the first modification and the second modification, similar images are displayed in a quadrilateral shape for each category, thereby rendering a connecting line or a quadrilateral background, and obtaining both the effects of the first modification and the effects of the second modification. For example, in this case, the connection line to the similar image at the top left corner (the similar image most similar to the query image in the category) within each category of quadrangle is thicker according to the similarity between the similar image and the query image, and the background of the quadrangle is drawn so that the upper left corner is thick and the lower right corner becomes thinner.

(second embodiment)

The display control apparatus 101 according to the second embodiment of the present invention associates disease attributes (for example, "benign/malignant", "melanocyte type/non-melanocyte type", and the like) of a diagnosis target portion shown in a query image with each coordinate axis, and displays an index indicating the possibility of a disease relating to each attribute by plotting the index in a space having the number of disease attributes as dimensions. By performing such display, the display control apparatus 101 easily grasps the disease attribute information of the diagnosis target portion. In the second embodiment, the case where the disease of the part to be diagnosed is a human skin disease is described as an example, but various parts (diseases) to be diagnosed based on the captured image, such as a human uterus (uterine cancer), an oral cavity (oral cancer), skin (skin cancer) of an animal (cat or dog) other than a human, and an oral cavity (oral cancer), may be present as the part to be diagnosed (disease).

As shown in fig. 8, the display control device 101 according to the second embodiment includes a control unit 10, a storage unit 20, an input unit 31, an output unit 32, and a communication unit 33.

The control unit 10 is configured by a CPU or the like, and executes a program stored in the storage unit 20 to realize functions of each unit (the index acquisition unit 16, the risk acquisition unit 17, and the display control unit 18) described later.

The storage unit 20 is configured by a ROM, a RAM, and the like, and stores programs executed by the CPU of the control unit 10 and necessary data.

The input section 31 is a device (keyboard, mouse, touch panel, camera, etc.) for the user of the display control apparatus 101 to input an instruction applied to the display control apparatus 101 or to input a query image. The control unit 10 acquires an instruction from the user or an inquiry image via the input unit 31. As the input unit 31, any device can be used if the control unit 10 can acquire an instruction from the user or an inquiry image. However, the control unit 10 may acquire the query image via the communication unit 33. The query image is image data of an image obtained by imaging a diagnostic target portion using a skin mirror or the like. The display control apparatus 101 presents the disease attribute information of the diagnosis target portion shown in the query image to the user in an easily understandable manner.

The output section 32 is a device (display, display interface, etc.) for the control section 10 to present the disease attribute information to the user in an easily understandable manner. The display control apparatus 101 may include a display (display unit) as the output unit 32, or may display the disease attribute information and the like on an external display (display unit) connected via the output unit 32.

The communication unit 33 is a device (such as a network interface) for transmitting and receiving data to and from an external device (such as a server storing a database of image data, an image recognition device, and the like). The control unit 10 can acquire the image recognition result of the image recognition apparatus and the like via the communication unit 33.

Next, the function of the control unit 10 will be described. The control unit 10 realizes the functions of the index acquisition unit 16, the risk acquisition unit 17, and the display control unit 18.

The index acquisition unit 16 obtains the probability (likelihood) of each attribute relating to a disease in the diagnostic target portion shown in the query image by using the identifier, and acquires the obtained probability as the attribute index. The recognizer is constituted by, for example, a convolutional neural network, and performs learning using predetermined learning image data in advance. The index acquisition unit 16 may be provided with such a learned recognizer, and for example, may be configured to cause an external image recognition device provided with the learned recognizer to recognize the query image via the communication unit 33, and acquire, as the attribute index, the probability (likelihood) of each attribute obtained as a result thereof. The index acquired by the index acquisition unit 16 is not limited to the probability, and the index acquisition unit 16 may acquire a more general score (the score is not limited to the score corresponding to the probability) as the index, the score being larger as the probability of existence is higher, and conversely, the score being larger as the probability is lower. The index acquisition unit 16 functions as an acquisition means.

Here, the index acquisition unit 16 includes a disease recognizer that outputs a probability (hereinafter referred to as "disease equivalent probability") that the disease of the diagnosis target portion shown in the query image is each of four diseases (melanoma, basal cell carcinoma, pigmented nevus, and seborrheic keratosis). The disease equivalent probability obtained by inputting the query image to the disease recognizer is, for example, 89.0% for melanoma, 4.4% for basal cell carcinoma, 6.4% for pigmented nevi, and 0.2% for seborrheic keratosis. In addition, among these disease attributes, pigmented nevi are "benign, melanocytic type", melanoma is "malignant, melanocytic type", seborrheic keratosis is "benign, nonmelanocytic type", and basal cell carcinoma is "malignant, nonmelanocytic type".

In this example, the probability of "malignancy" being the disease attribute of the part to be diagnosed is calculated to be 89.0% + 4.4% + 93.4%, and the probability of "benign" being 6.4% + 0.2% + 6.6%. The probability of "melanocytes" as a disease attribute of the part to be diagnosed was calculated to be 89.0+ 6.4% + 95.4%, and the probability of "non-melanocytes" was calculated to be 4.4% + 0.2% + 4.6%. The index acquisition unit 16 acquires the probability that the disease attribute of the diagnosis target portion calculated in this manner is various attributes as an index indicating the probability that the disease attribute of the diagnosis target portion is various attributes. In particular, the probability that the disease attribute of the diagnosis target portion is "malignant" and the probability that the disease attribute is "benign" are referred to as a malignant index and a benign index, respectively, and the probability that the disease attribute of the diagnosis target portion is a predetermined disease attribute such as "melanocyte type" or "non-melanocyte type" is referred to as a disease attribute index. In addition, when a plurality of disease attributes are referred to differently, the first, second, and the like are added. For example, when "melanocyte class" is set as the first disease attribute and "non-melanocyte class" is set as the second disease attribute in the disease attribute of the diagnosis target portion, the probability that the disease attribute of the diagnosis target portion is "melanocyte class" is referred to as a first disease attribute index, and the probability that the disease attribute of the diagnosis target portion is "non-melanocyte class" is referred to as a second disease attribute index.

The index acquisition unit 16 does not necessarily need to use a disease identifier for acquiring the probability of equivalent disease in the part to be diagnosed. For example, instead of the disease identifier, the index acquisition unit 16 may use an identifier that outputs a probability (malignancy indicator) that the disease attribute of the diagnosis target portion is "malignant" regardless of the disease type of the diagnosis target portion, or an identifier that outputs a probability (disease attribute indicator) that the disease attribute of the diagnosis target portion is a predetermined disease attribute such as "melanocyte type".

The risk acquisition unit 17 acquires a risk index indicating whether the risk of a disease is high when the disease attribute is malignant and the disease attribute is a predetermined disease attribute. Here, although the risk may be considered to be an overlooked risk (risk of erroneous determination by the identifier (malignancy not detected)) or a prognostic risk (overlooked risk), the risk acquisition unit 17 may distinguish these risks and regard them as individual risk indicators or may regard them as one risk indicator by integrating these values. For example, the control unit 10 obtains a risk index of the overlooked risk using image data (test case data) other than learning data for learning the disease identifier, obtains a risk index of the later risk using data relating to the later risk obtained by experts or the like, and stores the risk index in the storage unit 20 in advance. Alternatively, the risk indicator may be obtained in advance by an external server or the like. The risk acquisition unit 17 acquires in advance a risk index obtained by the control unit 10, an external server, or the like. In the present embodiment, the risk index is an index indicating the level of the risk of being overlooked in the case where the disease attribute is malignant based on the malignancy index of the disease, and is generated in advance by a risk boundary generation process described later.

For example, even if the probability of "malignancy" (malignancy index) is the same, a malignant disease of the melanocyte type is more difficult to identify than a malignant disease of the non-melanocyte type, and the risk of overlooking is high. In the present embodiment, since it is considered that the risk of being overlooked is high if the malignancy index is higher than the risk index, the value of the risk index in the case where the disease attribute is "melanocyte-type" is lower than the risk index in the case where the disease attribute is "non-melanocyte-type". Therefore, when the disease attribute is "melanocyte-type", the risk acquiring unit 17 acquires a lower value risk indicator than when the disease attribute is "non-melanocyte-type". The risk acquisition unit 17 functions as a risk acquisition means.

The display control unit 18 causes the display unit to display the plurality of indices acquired by the index acquisition unit 16 in association with each other by a display control process described later. For example, if the index acquisition unit 16 acquires 93.4% as the "malignancy" index and 95.4% as the "melanocyte class" index for the diagnosis target portion shown in the query image, the point 206 is displayed on the display unit as a point corresponding to (95.4%, 93.4%) as shown in fig. 9. The display control unit 18 functions as display control means.

In fig. 9, attribute names are shown at both ends of each axis, for example, the vertical axis indicates malignant and benign, and the horizontal axis indicates melanocyte class and nonmelanocyte class. However, since each axis is actually based on one index (the attributes of both ends of each axis are in a front-to-back relationship, for example, 100% malignant means 0% benign), for example, only a single name may be described, and for example, the vertical axis may be only malignant, and the horizontal axis may be only melanocytes. In fig. 9, the intersection of the vertical axis and the horizontal axis indicates 50% for both malignant and benign indicators and 50% for both melanocyte-type and non-melanocyte-type indicators.

The display control unit 18 displays the risk index acquired by the risk acquisition unit 17 on the display unit in association with the plurality of indexes acquired by the index acquisition unit 16. For this purpose, the display control unit 18 displays the risk boundary lines generated by the risk boundary line generation process described later as, for example, the risk boundary lines 207 shown by broken lines in fig. 9. In fig. 9, the point 206 is on the upper side of the risk boundary 207, but this represents a case where the risk of a disease is high in the diagnosis target portion shown in the query image. Further, fig. 9 shows an example of the risk boundary 207 based on the overlooked risk, but in the case where the risk acquisition portion 17 acquires not only the overlooked risk but also the prognosis risk, the display control portion 18 may display a risk boundary (not shown) based on the prognosis risk in addition to the risk boundary 207 based on the overlooked risk. In the case where the risk acquiring unit 17 acquires only the prognosis risk, the display control unit 18 may display only the risk boundary line (not shown) based on the prognosis risk instead of the risk boundary line 207 based on the overlooked risk.

The functional configuration of the display control apparatus 101 has been described above. Next, the contents of the display control process performed by the display control apparatus 101 will be described with reference to fig. 10. The display control process is started when the user instructs the display control apparatus 101 to start the display control process via the input unit 31. Before starting the instruction display control process, the user instructs the display control device 101 in advance about the attribute type used in the coordinate axis (for example, "benign/malignant" in the vertical axis and "melanocyte type/non-melanocyte type" in the horizontal axis).

First, the display control unit 18 causes the display unit to display coordinate axes (step S201). The coordinate axes displayed here are based on attributes previously instructed from the user. For example, in the example shown in fig. 9, the vertical axis is a malignant (benign/malignant) axis, and the horizontal axis is a melanocyte class (melanocyte class/non-melanocyte class).

Next, the control unit 10 of the display control apparatus 101 acquires the query image (step S202). For example, when the user inputs an inquiry image to the display control apparatus 101 via the input section 31 (for example, drags and drops the inquiry image to a predetermined area of the screen of the display section), the control section 10 acquires the inquiry image.

Next, the index acquisition unit 16 inputs the query image to the recognizer to acquire the index of each attribute (step S203). Step S203 is also referred to as an acquisition step. Then, the display control unit 18 displays the point 206 on the coordinate axis displayed on the display unit, the coordinate axis being indicated by the index acquired by the index acquisition unit 16 (step S204). Step S204 is also referred to as a display control step.

Next, the display control unit 18 displays the risk boundary line 207, which is generated in advance by the risk boundary line generation process described later and stored in the storage unit 20, on the display unit (step S205), and ends the display control process.

Next, the risk boundary generation process will be described with reference to fig. 11. The risk boundary generation processing is executed in advance before the display control processing (fig. 10) is executed. Specifically, for example, when the user instructs the attributes to be used in the coordinate axes of fig. 9, the risk boundary generation processing is started. However, the risk boundary generation process may be executed by an external server or the like in advance. In this case, the control unit 10 acquires the result (the coordinates of the risk boundary line) via the communication unit 33 and stores the result in the storage unit 20. Next, a case where the risk boundary line generation process is executed by the control unit 10 in advance will be described.

First, the control unit 10 acquires (not used for learning of the disease identifier) image data of a test case from the storage unit 20 or via the communication unit 33 (step S301). Next, the index acquisition unit 16 inputs the image data of the test case to the disease identifier, and acquires the attribute index corresponding to each coordinate axis (step S302). In the example shown in fig. 9, the attributes are "malignant" and "melanocyte-like", and here, the index of "malignant" is referred to as a malignant index, and the index of "melanocyte-like" is referred to as a disease attribute index.

Then, the control unit 10 classifies the malignancy index into each section of the disease attribute index among the indexes acquired in step S302 (step S303). Here, each of the disease attribute index sections is, for example, 10 sections, i.e., a section 1 in which the disease attribute index value is 0% or more and less than 10%, a section 2 and … in which the disease attribute index value is 10% or more and less than 20%, and a section 10 in which the disease attribute index value is 90% or more and 100% or less, when the disease attribute index value is 0% or more and 100% or less and the width of each section is 10%. For example, if the indexes acquired in step S302 are 35% of the malignancy index and 55% of the disease attribute index, the control unit 10 classifies 35% of the malignancy index as the section 6.

Next, the control unit 10 determines whether or not the malignancy indicator classified in step S303 is classified into all the sections (all the sections from section 1 to section 10 in the above example) by a predetermined number (for example, 20) or more (step S304). If there is a section in which only classification is smaller than the predetermined number (step S304: NO), the process returns to step S301, and index classification is repeatedly performed based on the new test case data.

If the classification is performed for a predetermined number or more in all the sections (step S304: "30"), the control unit 10 calculates a malignancy determination threshold value for which the sensitivity of the malignant disease is a predetermined sensitivity (for example, a sensitivity of 95%) (for example, a threshold value for which 95% is determined to be a malignant disease when a certain number of test cases of the malignant disease are identified in the case of a sensitivity of 95%) in each section (step S305). The lower the threshold value, the more easily the disease attribute is judged to be malignant, the sensitivity increases, and the specificity (accuracy of benign cases) decreases.

Then, the control unit 10 sets a line obtained by connecting the values of the malignancy determination threshold values of the respective sections by using a spline curve or the like as a risk boundary line, stores the coordinates of the risk boundary line in the storage unit 20 (step S306), and ends the risk boundary line generation processing. If the point displayed in step S204 of the display control process (fig. 10) is located above the risk boundary line, it means that the risk of a disease in the diagnosis target portion shown in the query image is high.

The risk boundary generation processing described above is always an example, and the following modifications are also considered.

Changes according to the risk of the disease (the metastatic potential of melanocytic species (melanoma, etc.) is much higher than that of non-melanocytic species (basal cell carcinoma, etc.) and the risk of prognosis increases, thus the risk boundary decreases in regions where the probability of disease attribute being melanocytic species is high.)

Up and down according to the size of the part to be diagnosed (the risk increases as the size increases, and therefore the risk boundary decreases.)

Upper and lower depending on the lesion depth of the diagnosis target portion (the lesion depth is estimated by image processing such as determination by the color of the diagnosis target portion, and the risk of prognosis increases when the lesion depth increases, and therefore the risk boundary line decreases.)

Depending on the size of the ulcer as the diagnosis target part, the size of the bleeding region in the diagnosis target part (ulcer, bleeding, and the larger the region, the higher the risk of prognosis, and thus the lower the risk boundary)

As described above, the display control apparatus 101 can display the attribute information of the diagnosis target portion shown in the query image in an easily understandable manner from the coordinates of the point 206 as shown in fig. 9 with respect to the input query image. Further, the risk boundary 207 is displayed, and the height of the risk of the disease in the diagnosis target portion can be grasped based on the positional relationship between the point 206 and the risk boundary 207.

In the display control device 101 according to the second embodiment, as well as the similar image display device according to the first embodiment, as attributes, instead of the "alternative/malignant alternative/melanocyte type/non-melanocyte type" plain cells, the "alternative/non-epithelial", "metastatic/non-metastatic", "invasive/non-invasive", "viral/non-viral" and "size of lesion/color of non-lesion" may be used, and the size of the measured value such as the size can be seen through observation/(for example, when the horizontal axis represents time, and the vertical axis represents the size, the time change of the measured value such as the size can be seen through observation) "may be used. Of these attributes, the attribute having the highest prognostic risk is considered as the melanocyte class, and therefore in the example shown in fig. 9, the horizontal axis is an index representing the possibility that the disease attribute is the melanocyte class (melanocyte class/non-melanocyte class).

In the second embodiment, two types of "sex side/malignant side/melanocyte type/non-melanocyte type" are assigned to the vertical axis and the horizontal axis as attributes, and the point 206 is displayed in a two-dimensional space. However, the attributes to be used may be divided into three types, the points 206 may be arranged in a three-dimensional space, and the projected portions may be output to the output unit 32. When the number n of types of attributes to be used is 4 or more, the points 206 may be virtually arranged in the n-dimensional space, and may be finally projected in the two-dimensional space and output to the output unit 32.

(third embodiment)

The display control apparatus 102 according to the third embodiment of the present invention displays the disease attribute of the diagnosis target portion shown in the query image, together with the probability that the disease of the diagnosis target portion is a predetermined disease, by using the tree structure in which the query image is set as the root node. By performing such display, the display control apparatus 102 easily grasps the disease attribute information of the diagnosis target portion.

As shown in fig. 12, the display control device 102 according to the third embodiment includes a control unit 10, a storage unit 20, an input unit 31, an output unit 32, and a communication unit 33. The storage unit 20, the input unit 31, the output unit 32, and the communication unit 33 are the same as the storage unit 20, the input unit 31, the output unit 32, and the communication unit 33 included in the display control device 101 according to the second embodiment, and therefore, description thereof is omitted.

The control unit 10 is configured by a CPU or the like, and executes a program stored in the storage unit 20 to realize functions of each unit (the index acquisition unit 16, the position determination unit 13, the disease risk acquisition unit 19, and the display control unit 18) described later.

The index acquisition unit 16 obtains probabilities (likelihoods) of the disease of the diagnosis target portion shown in the query image with respect to the attributes by using a disease recognizer that recognizes a predetermined number of diseases, and acquires the obtained probabilities as the indexes of the attributes. The disease identifier is constituted by, for example, a convolutional neural network, and performs learning using predetermined learning image data in advance. The index acquisition unit 16 may be provided with such a learned disease identifier, and for example, may be configured to cause an external image recognition device provided with a learned disease identifier to recognize a query image via the communication unit 33, and acquire, as an index of the attribute, a probability (likelihood) associated with each attribute obtained as a result of recognizing the query image.

Here, as described in the item of the index acquisition unit 16 according to the second embodiment, the index acquisition unit 16 includes, for example, a disease identifier that outputs a disease equivalent probability concerning four diseases (melanoma, basal cell carcinoma, pigmented nevus, and seborrheic keratosis). The disease equivalent probability obtained by inputting the query image to the disease recognizer is, for example, 89.0% for melanoma, 4.4% for basal cell carcinoma, 6.4% for pigmented nevi, and 0.2% for seborrheic keratosis.

In this example, as described in the explanation of the index acquisition unit 16 according to the second embodiment, as the index indicating the possibility that the disease attribute of the diagnosis target portion is various attributes, the malignancy index is 93.4%, the benign index is 6.6%, the disease attribute index of "melanocyte-based" is 95.4%, and the disease attribute index of "5 melanocyte-based" is 4.6%. The index acquisition unit 16 according to the third embodiment also acquires the probability of each disease output from the disease identifier as a disease index. In this example, the disease index for melanoma was 89.0%, the disease index for basal cell carcinoma was 4.4%, the disease index for pigmented nevi was 6.4%, and the disease index for seborrheic keratosis was 0.2%.

The position determination unit 13 determines, as coordinates in the n-dimensional space, positions for displaying information on each disease (category) corresponding to the number of disease indicators acquired by the indicator acquisition unit 16, according to n types (n is an integer equal to or greater than 1) of attributes. More specifically, n attributes are associated with n coordinate axes defining the coordinates of an n-dimensional space, respectively, one for one, and coordinates indicating positions for displaying information on each disease are determined based on an index indicating the possibility of the disease being associated with the attribute associated with each coordinate axis.

For example, when two attributes of "adopted/malignant adopted/melanocyte type/non-melanocyte type" are adopted as the n attributes, the position determination unit 13 determines coordinates on a two-dimensional space for displaying information on each disease corresponding to the disease index acquired by the index acquisition unit 16. For example, as shown in fig. 13, the position determination unit 13 determines coordinates of a circle (probability circle) for displaying the position of the circle indicating the probability magnitude of each disease in the diagnosis target portion in a two-dimensional space in which "the determination/malignancy is defined as the vertical axis (Y axis) and" the melanocyte type/non-melanocyte type "is defined as the horizontal axis (X axis). In fig. 13, the vertical axis (Y axis) shows the lower side for benign and the upper side for malignant, and the horizontal axis (X axis) shows the left side for melanocytes and the right side for nonmelanocytes.

Specifically, when four kinds of pigmented nevi, melanoma, seborrheic keratosis, and basal cell carcinoma are considered as diseases, among the disease attributes AZ, pigmented nevi are "melanotic melanocytes" melanomas and "malignant, melanocytes" melanotic keratosis and "epiphytic, nonmelanocytes" melanotic keratosis, respectively. Therefore, as shown in fig. 13, the position determination unit 13 determines each position so that pigmented nevi are displayed in the lower left-hand region, melanoma is displayed in the upper left-hand region, seborrheic keratosis is displayed in the lower right-hand region, and basal cell carcinoma is displayed in the upper right-hand region.

Further, the position determination unit 13 may adjust the display position of the information on the disease as necessary so that the positions for displaying the information on different diseases do not have the same coordinates. Although not shown in fig. 13, for example, when the index acquiring unit also acquires a disease index of hematoma/hemangioma, the attribute of hematoma/hemangioma is "sexually keratinized melanocyte type" as in seborrheic keratosis, and therefore, if the display position of the information related to the disease is not adjusted, the information related to the disease is displayed in the same lower right corner region for both diseases. In this case, the position determination unit 13 may adjust the display position of the information on each disease, for example, by shifting the display position of the information on seborrheic keratosis slightly to the left of the center of the region in the lower right corner, and shifting the display position of the information on hematoma/hemangioma slightly to the right of the center of the region in the lower right corner.

N attribute information used when the position determining unit 13 determines the coordinate axis of the space in which the information on each disease is displayed, information on the attribute of each disease, and arrangement information on each attribute are stored in the storage unit 20 in advance. The position determination unit 13 determines coordinates in an n-dimensional space in which the position of information relating to each disease is displayed, based on the n-attribute information stored in the storage unit 20, the information on the attribute of each disease, and the arrangement information of each attribute. In the example shown in fig. 13, two kinds of attribute information, i.e., "sex/malignancy information" and "melanocyte-type/non-melanocyte-type" element cells, are stored in the storage unit 20 as the attribute information. As information on the attribute of each disease, information that pigmented nevi are "melanotic melanocytes" melanomas are "malignant, melanotic melanocytes" melanotic keratosis is "hyperkeratomelanocytes" melanotic melanocytes "melanotic diseases and" malignant, non-melanocytes "is stored in the storage unit 20. Further, as the arrangement information of each attribute, in the "arrangement of sex/malignancy, information that" sex is arranged on the lower side and "malignancy side is arranged on the upper side, and" melanocyte type/non-melanocyte type "melanocytes," melanocyte type "is arranged on the left side and" lateral melanocyte type "is arranged on the right side is stored in the storage unit 20.

The disease risk acquisition unit 19 acquires a risk index indicating for each disease whether the risk of the disease is high or not. Here, although there are a prognostic risk (an neglected risk in the case of neglecting a disease) and a neglected risk (a misjudgment risk that the disease identifier does not judge a malignant disease as a malignant disease) among the risks of diseases, the disease risk acquisition unit 19 may distinguish these risks and regard them as individual risk indicators or may regard them as one risk indicator by integrating these risks. For example, melanoma has a high risk of prognosis, overlooked risk, compared to basal cell carcinoma. Therefore, the disease risk acquiring unit 19 acquires, for example, 10% as a risk index of melanoma and 80% as a risk index of basal cell carcinoma. This is an example of the following: if the disease in the part to be diagnosed is melanoma, the risk is high even if the probability (disease index) is 10%, but if the disease is basal cell carcinoma, the risk cannot be said to be high if the probability (disease index) is not 80% or more. The risk index value of each disease may be a value preset by a doctor or the like for each disease, or a disease index having a sensitivity of a predetermined value (for example, 95% or 90%) in each disease may be obtained in advance as a judgment threshold value using test case data different from data used for learning, as in the risk boundary line generation process (fig. 11) of the second embodiment, and the obtained judgment threshold value may be acquired as a risk index. The disease risk acquiring unit 19 functions as a disease risk acquiring means.

The display control unit 18 causes the plurality of indices acquired by the index acquisition unit 16 to pass through the tree structure in a correlated manner by a display control process described later, and displays the indices on the display unit as shown in fig. 13. For example, if the index acquisition unit 16 acquires values of 89.0% of melanoma, 4.4% of basal cell carcinoma, 6.4% of pigmented nevus, and 0.2% of seborrheic keratosis as disease indexes of respective diseases for a part to be diagnosed shown in the query image, as shown in fig. 13, the display control unit 18 displays the probability that the disease of the part to be diagnosed is pigmented nevus with a probability circle 411 corresponding to 6.4%, the probability that the disease of the part to be diagnosed is melanoma with a probability circle 412 corresponding to 89.0%, the probability that the disease of the part to be diagnosed is seborrheic keratosis with a small probability circle 413 corresponding to 0.2%, and the probability that the disease of the part to be diagnosed is basal cell carcinoma with a probability circle 414 corresponding to 4.4%. Further, although a point indicating the center is displayed at the center of each probability circle in fig. 13, whether or not such a point is displayed is arbitrary, and the display of the point at the center may be turned on and off in accordance with a user instruction or the like.

The functional configuration of the display control apparatus 102 is described above. Next, the contents of the display control process performed by the display control apparatus 102 will be described with reference to fig. 14. The display control process is started when the user instructs the display control apparatus 102 to start the display control process via the input unit 31.

First, the control unit 10 of the display control apparatus 102 acquires a query image (step S401). For example, when the user inputs an inquiry image to the display control apparatus 102 via the input section 31 (for example, drags and drops the inquiry image to a predetermined area of the screen of the display section), the control section 10 acquires the inquiry image.

Next, as shown in fig. 13, the display control unit 18 displays the query image 400 at the center portion of the display screen (step S402).

Next, the index acquisition unit 16 inputs the query image to the disease identifier to acquire a disease index of each disease (step S403). Then, as shown in fig. 13, the display control unit 18 displays a probability circle based on the disease index size of each disease at the display position of the information on each disease at the position determined by the position determination unit 13 (step S404).

Then, the display control unit 18 displays a risk circle indicating the size of the risk index of each disease acquired by the disease risk acquisition unit 19 at a position where the center of the risk circle coincides with the center of the probability circle of each disease (step S405). For example, in fig. 13, a risk circle 415 based on a risk indicator size with a sensitivity of 90% for melanoma is shown by a solid line, and a risk circle 416 based on a risk indicator size with a sensitivity of 95% is shown by a broken line. The risk circle 417 based on the risk indicator having a sensitivity of 90% for basal cell carcinoma is indicated by a solid line, and the risk circle 418 based on the risk indicator having a sensitivity of 95% is indicated by a broken line. Here, the "risk index having a sensitivity of disease S of P%" refers to a threshold value of an output (probability value of disease S) of a disease recognizer, of which P% is determined as disease S, when a certain number of test case images of disease S are recognized using the disease recognizer.

In the example shown in fig. 13, the case where the probability circle 412 of melanoma is larger than the risk circle 415 based on the risk indicator size with the sensitivity of 90% and the overlooked risk of melanoma is high is shown. Conversely, the probability circle 414 showing basal cell carcinoma is smaller than the risk circle 418 based on the risk indicator size with a sensitivity of 95%, whereas the overlooked risk of basal cell carcinoma is low.

Next, as shown in fig. 13, the display control unit 18 displays a tree structure in which the query image 400 at the center of the display screen is set as a root node, the probability circles 411, 412, 413, and 414 of the respective diseases are set as leaf nodes, and these are connected using the connection lines 421, 422, 423, and 424 (step S406), and ends the display control process.

In the display of the tree structure in step S406, the display control section 18 displays the malignant node 432 larger than the benign node 431 as shown in fig. 13 if the malignant index is larger than the benign index, based on the index acquired by the index acquisition section 16. In addition, the following tree structure is shown: after the malignant node 432 and the benign node 431, the melanocyte-like nodes 433 and 434 and the nonmelanocyte-like nodes 435 and 436 are provided, and the connecting lines 421, 422, 423, and 424 extend from these nodes to the probability circles 411, 412, 413, and 414 of diseases corresponding to the attributes.

Although not shown in fig. 13, the display control unit 18 may perform display such as adding a green frame to the probability circle of benign disease and adding a red frame to the probability circle of malignant disease, so as to easily understand the risk level of each disease.

In fig. 13, the size of each probability circle corresponds to the probability of the corresponding disease, but the present invention is not limited thereto. Even if the probabilities are the same and the risk levels are different depending on the disease, for example, when the risk index acquired by the disease risk acquisition unit 19 is higher than the probability of the disease, a large probability circle may be displayed even if the probability of the disease is small, and when the risk index acquired by the disease risk acquisition unit 19 is lower than the probability of the disease, a small probability circle may be displayed even if the probability of the disease is large.

As described above, the display control device 102 can easily grasp the disease attribute information of the diagnosis target portion by displaying the probability circle by the tree structure based on each disease attribute by showing the probability that the disease of the diagnosis target portion shown in the query image is a predetermined disease in the size of the probability circle as shown in fig. 13 with respect to the input query image. Further, by displaying the risk circles 415, 416, 417, 418, the height of the risk of the disease in the part to be diagnosed can be grasped from the size relationship between the probability circles 412, 414 and the risk circles 415, 416, 417, 418.

In the display control apparatus 102 according to the third embodiment, as attributes, instead of the "substitute/malignant substitute/melanocyte type/non-melanocyte type" pigment cells, the "color portion-changed/(e.g., when the horizontal axis represents time and the vertical axis represents measured values such as size, a temporal change in measured values such as size can be observed)" may be used, and the like. Of these attributes, the attribute having the highest prognostic risk is considered as the melanocyte class, and therefore in the example shown in fig. 13, the tree structure is shown in which the index representing the possibility that the disease attribute is the melanocyte class (melanocyte class/non-melanocyte class) is shown in the horizontal axis and the index representing the possibility that the disease attribute is malignant (benign/malignant) is shown in the vertical axis.

The display control unit 18 may display only the probability circles 411, 412, 413, and 414, and may display only the nodes and a part of the connection lines constituting the tree structure, instead of displaying the nodes of the malignant node 432, the benign node 431, the melanocyte class nodes 433 and 434, and the non-melanocyte class nodes 435 and 436, and may display only the probability circles 411, 412, 423, and 424, and the connection lines extending from the nodes to the probability circles 411, 412, 413, and 414 of the diseases corresponding to the attributes.

In the third embodiment, as attributes, two types of "will do/vicious will do/melanocytes/non-melanocytes" are assigned on the vertical axis and the horizontal axis, and the tree structure is displayed in two dimensions. However, the tree structure may be arranged in a three-dimensional space with three types of attributes to be used, and a portion obtained by projecting the tree structure in a two-dimensional space may be output to the output unit 32. When the number n of types of attributes to be used is 4 or more, the tree structure may be virtually arranged in the n-dimensional space, and finally projected in the two-dimensional space and output to the output unit 32.

(fourth embodiment)

The display control device 103 according to the fourth embodiment of the present invention displays an image similar to the query image around each probability circle in addition to the display of the tree structure by the display control device 102 according to the third embodiment. By performing such display, the display control apparatus 103 easily grasps the disease attribute information of the diagnosis target portion, and displays the relationship between the similar images in a more easily understood manner.

As shown in fig. 15, the display control device 103 according to the fourth embodiment includes a control unit 10, a storage unit 20, an input unit 31, an output unit 32, and a communication unit 33. The storage unit 20, the input unit 31, and the output unit 32 are the same as the storage unit 20, the input unit 31, and the output unit 32 included in the display control device 102 according to the third embodiment, and therefore, description thereof is omitted. The communication unit 33 is also the same as the communication unit 33 provided in the display control device 102 according to the third embodiment, but the other external device to which data is transmitted/received is also a similar image search device or the like, and the control unit 10 can acquire a similar image search result (for example, an image similar to the query image) of the similar image search device via the communication unit 33.

The control unit 10 is configured by a CPU or the like, and executes a program stored in the storage unit 20, thereby realizing the functions of the respective units (the index acquisition unit 16, the position determination unit 13, the disease risk acquisition unit 19, the similar image acquisition unit 11, the classification unit 14, and the display control unit 18) described later.

The index acquisition unit 16, the position determination unit 13, and the disease risk acquisition unit 19 are the same as the index acquisition unit 16, the position determination unit 13, and the disease risk acquisition unit 19 included in the display control device 102 according to the third embodiment, and therefore, description thereof is omitted.

The similar image acquiring unit 11 acquires data (image data of a similar image and a similarity with a query image of the image) obtained as a result of performing a similar image search on the query image, as in the similar image acquiring unit 11 according to the first embodiment. Specifically, in the similar image retrieval, image data having a similarity with the query image of a predetermined threshold or more is acquired together with the similarity. The similar image acquiring unit 11 may acquire data of similar images obtained as a result of the control unit 10 searching for images similar to the query image, or may cause an external similar image searching device to search for images similar to the query image via the communication unit 33 and acquire data of similar images searched by the similar image searching device, for example. In addition, information such as a disease name corresponding to each image is added as tag information to each image data.

The classification unit 14 classifies the image data acquired by the similar image acquisition unit 11 into any of the diseases identified by the disease identifier used by the index acquisition unit 16. The classification unit 14 can classify the image data into any disease by using the label information added to each image data (for example, by adding a disease name to each image data as label information).

The display control unit 18 performs the following processing in addition to the processing of the display control unit 18 according to the third embodiment, by the display control processing described later: the data of the similar images acquired by the similar image acquiring unit 11 is displayed around the probability circle corresponding to the disease classified by the classifying unit 14 as shown in fig. 16.

The functional configuration of the display control device 103 is described above. Next, the contents of the display control processing performed by the display control device 103 will be described with reference to fig. 17. When the user instructs the display control apparatus 103 to start the display control process via the input unit 31, the display control process is started. Among the display control processes shown in fig. 17, the processes from step S401 to step S406 are the same as those of the display control process (fig. 14) of the display control device 102 according to the third embodiment, and therefore, the description thereof is omitted.

When the tree structure is displayed by the processing up to step S406, next, the similar image acquiring unit 11 acquires a similar image obtained as a result of performing the similar image search on the query image (step S407). Specifically, in the similar image retrieval, a similar image having a similarity to the query image of a predetermined threshold or more is acquired. At this time, the similar image acquiring unit 11 also acquires the similarity between the similar image and the query image together with the similar image.

Then, the classification unit 14 classifies the similar images acquired by the similar image acquisition unit 11 into any one of the diseases identified by the disease identifier used by the index acquisition unit 16, based on the label information (disease name) attached to each similar image (step S408).

Then, the display control unit 18 arranges the similar images acquired by the similar image acquiring unit 11 in step S407 around (or inside) the probability circle corresponding to the disease classified by the classification unit 14 in step S408, and displays the similar images on the display unit (step S409), and ends the display control process.

As shown in fig. 16, the display control unit 18 in step S409 displays the similar images in a concentric circle so that the image having a higher degree of similarity to the query image is disposed at the center of the probability circle around (or inside) the probability circle of each disease. In the example shown in fig. 16, the similar images having the highest similarity to the query image among the similar images classified into the respective diseases are arranged on the centers of the respective probability circles, and are arranged clockwise in the center, thereby being arranged in a concentric circle shape.

In addition, each similar image is displayed surrounded by a small circle, but the higher the similarity of the similar image to the query image, the thicker the thickness of the line of the small circle. For example, in the example shown in fig. 16, the line thickness of the small circle 4121 surrounding the similar image arranged at the upper part of the center of the probability circle 412 of melanoma is shown to be thicker than the line thickness of the small circle 4122 surrounding the similar image arranged adjacent thereto. The thickness of the line of the small circle 4121 surrounded by the similar image disposed on the upper portion of the center of the probability circle 412 for melanoma is displayed as the thickness of the line of the small circle 4111 surrounded by the similar image disposed on the upper portion of the probability circle 411 for moles, the thickness of the line of the small circle 4131 surrounded by the similar image disposed on the upper portion of the probability circle 413 for seborrheic keratosis, or the thickness of the line of the small circle 4141 surrounded by the similar image disposed on the upper portion of the probability circle 414 for basal cell carcinoma. This means that the similar image most similar to the query image among the similar images acquired by the similar image acquiring section 11 is an image of melanoma (an image surrounded by a small circle 4121 surrounding the similar image).

As described above, the display control device 103 displays the probability that the disease of the diagnosis target portion shown in the query image is a predetermined disease in the size of the probability circle as shown in fig. 16, and displays the probability circle by the tree structure according to each disease attribute, thereby easily grasping the disease attribute information of the diagnosis target portion. Further, by displaying the risk circles 415, 416, 417, 418, the height of the risk of the disease in the part to be diagnosed can be grasped from the size relationship between the probability circles 412, 414 and the risk circles 415, 416, 417, 418. Further, since the similar images can be arranged around (or inside) each probability circle and displayed in descending order of the degree of similarity to the query image, the relationship between the similar images can be displayed more clearly.

In the display control device 102 according to the fourth embodiment, as in the display control device 102 according to the third embodiment, various attributes can be used, and the display control unit 18 may display only the probability circles 411, 412, 413, 414, the risk circles 415, 416, 417, 418, the query image 400, the similar images, the nodes constituting the tree structure, and a part of the connection line. Note that the number of types of attributes is not two types of tree structures (tree structures in a two-dimensional space), and n types of attributes may be provided, and tree structures may be arranged in the n-dimensional space and finally projected in the two-dimensional space and output to the output unit 32.

In the second, third, and fourth embodiments, the skin diseases are described as an example, but the present invention is not limited to the field of dermatology, and can be widely applied to the field of recognition of images by a recognizer. For example, identification of flower type by flower image, identification of bacteria by bacteria photomicrograph, and the like can also be applied. The method of implementing these discriminators is arbitrary, and may be implemented by DNN (deep Neural Network) such as CNN (Convolutional Neural Network), or may be implemented by SVM (Support Vector Machine), logistic regression, or the like.

In the second, third, and fourth embodiments, the control unit 10 performs the display control process, but the communication unit 33 may receive the result of the process corresponding to the display control process performed by the external server and output the result to the output unit 32.

Further, the above embodiments and modifications can be combined as appropriate. The fourth embodiment can be said to be an embodiment obtained by combining a part of the first embodiment with the third embodiment, but for example, a part of the third embodiment may be combined with the first embodiment in reverse. In this way, it is possible to replace each category circle shown in fig. 3 with a probability circle indicating the magnitude of the probability of the disease corresponding to the category, and to display the value of the probability of the disease corresponding to the category and the risk circle. With this configuration, the probability and risk of each disease can be visually confirmed, and the similar image can be referred to, thereby making it possible to refer to diagnosis. The shapes of the probability circle and risk circle in the third and fourth embodiments are not limited to circles, and may be other suitable shapes (for example, n-sided shapes such as triangles and quadrangles, and symbol shapes such as hearts and stars).

The functions of the similar image display apparatus 100 and the display control apparatuses 101, 102, and 103 can be implemented by a Computer such as a general PC (Personal Computer). Specifically, in the above-described embodiment, the program of the similar image display processing performed by the similar image display apparatus 100 and the program of the display control processing performed by the display control apparatuses 101, 102, and 103 are stored in advance in the ROM of the storage unit 20. However, a computer capable of realizing the above-described functions may be configured by storing and distributing a program to a computer-readable recording medium such as a floppy disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (digital versatile Disc), an MO (magnetic-Optical Disc), a Memory card, or a USB (Universal Serial Bus) Memory, and reading and installing the program to the computer.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the specific embodiments described above, and the present invention includes the inventions described in the scope of the claims and the equivalent ranges thereof.