阅读说明：本技术 图像搜索方法、图像搜索设备和计算机可读存储介质 (Image search method, image search apparatus, and computer-readable storage medium ) 是由 黄耀海 陶训强 彭健腾 邓伟洪 胡佳妮 于 2019-02-22 设计创作，主要内容包括：本公开涉及一种图像搜索方法、图像搜索设备和计算机可读存储介质。该图像搜索方法包括：获得图像集合,计算图像集合中任何两个对象之间的相似度,根据所计算的相似度和给定的相似度阈值来构建图像集合中的对象的连通子图,至少使用连通子图的边的信息来计算各个连通子图的置信度,对连通子图进行排序以获得排序信息,使用置信度和排序信息来搜索图像集合中的图像。(The present disclosure relates to an image search method, an image search apparatus, and a computer-readable storage medium. The image searching method comprises the following steps: obtaining an image set, calculating the similarity between any two objects in the image set, constructing connected subgraphs of the objects in the image set according to the calculated similarity and a given similarity threshold, calculating the confidence of each connected subgraph by using at least the information of the edges of the connected subgraphs, sorting the connected subgraphs to obtain sorting information, and searching the images in the image set by using the confidence and the sorting information.)

1. An image search method, comprising:

obtaining an image set;

calculating the similarity between any two objects in the image set;

constructing a connected subgraph of the objects in the image set according to the calculated similarity and a given similarity threshold;

calculating a confidence of each connected subgraph using at least information of edges of the connected subgraph;

sorting the connected subgraphs to obtain sorting information;

images in the image collection are searched using the confidence and ranking information.

2. The method of claim 1, wherein constructing a connected subgraph comprises:

treating each object in the image set as a node;

connecting any two nodes satisfying the following conditions with an edge: the similarity between the objects corresponding to the two nodes is greater than a given similarity threshold.

3. The method of claim 1 or 2, wherein the information of the edge is an edge score, the method further comprising:

for each connected subgraph, an edge score is computed using 2 m/(n-1)), where m is the number of edges in each connected subgraph and n is the number of nodes in each connected subgraph.

4. The method of claim 1 or 2, wherein the confidence is calculated using a combination of information of nodes and information of edges of a connected subgraph.

5. The method of claim 4, wherein the combining is

Formulas with weights, such as w1 node information + w2 edge information, where w1 is 0 ≦ 1, and w2 is 1-w 1; or

The combination is realized by a classifier, and the information of the nodes and the information of the edges are used as the input characteristics of the classifier.

6. The method of claim 4, wherein the information of a node is a node score, wherein the node score of a connected subgraph is a ratio of the weight of the connected subgraph to the largest weight in all connected subgraphs.

7. The method of claim 6, wherein the weight is computed by summing a similarity between any two connected nodes in a connected subgraph.

8. The method of claim 6, wherein the weights are computed by summing image ranking scores for respective nodes in each connected subgraph, wherein the image ranking scores are provided by a search engine.

9. The method of claim 6, wherein the weight is computed by a combination of a sum of similarities between any two connected nodes in a connected subgraph and a sum of image ranking scores for the respective nodes in each connected subgraph, wherein the image ranking scores are provided by a search engine.

10. The method of claim 1 or 2, wherein the connected subgraph is ordered according to the number of nodes in the connected subgraph.

11. The method of claim 1 or 2, wherein the connected subgraphs are ordered according to similarity between nodes in the connected subgraphs.

12. The method of claim 1 or 2, wherein the connected subgraph is ordered according to a combination of similarity between nodes in the connected subgraph and the number of nodes.

13. The method of claim 1 or 2, wherein the searching comprises saving a highest ranked subgraph if its confidence is greater than a confidence threshold.

14. The method of claim 1 or 2, wherein the searching comprises performing post-processing if the confidence of the highest ranked subgraph is not greater than a confidence threshold.

15. The method of claim 14, wherein the post-processing comprises removing objects from a group image in the connected subgraph.

16. The method of claim 14, wherein the post-processing comprises giving greater weight to similarity between objects in the set of images from an image containing only one object.

17. The method of claim 14, wherein the post-processing comprises breaking bridge connections in a highest-ranked connected subgraph.

18. An image search method comprising:

dividing the set of images into a plurality of groups according to the labels, wherein the images in each group have the same label;

image search is performed using the method according to any of claims 1-17.

19. An image search apparatus, characterized by comprising a processor and a memory, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of the image search method according to any one of claims 1 to 18.

20. An image search apparatus characterized by comprising means configured to perform the steps of the image search method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 18.

Technical Field

The present disclosure relates generally to image searching, and in particular to the field of connected subgraph-based object recognition and image searching.

Background

In recent years, innovation of big data technology has made it easier to collect real world data from the internet and build large object data sets. As a result of this, research on, for example, Face Recognition (FR) has been vigorously developed. However, the raw data collected typically contains noise and therefore requires tissue and cleansing before use. Thus, building large data sets remains a time consuming and laborious task.

Therefore, there is a need for an automatic or semi-automatic image search method to find a target object or remove noise in order to improve the quality of the collected data set.

Disclosure of Invention

According to an aspect of the present invention, there is provided an image search method. The method comprises the following steps: obtaining an image set, calculating the similarity between any two objects in the image set, constructing connected subgraphs of the objects in the image set according to the calculated similarity and a given similarity threshold, calculating the confidence of each connected subgraph by using at least the information of the edges of the connected subgraphs, sorting the connected subgraphs to obtain sorting information, and searching the images in the image set by using the confidence and the sorting information.

Drawings

The above and other objects and advantages of embodiments of the present disclosure are further described below in conjunction with the specific embodiments, and with reference to the accompanying drawings. In the drawings, the same or corresponding technical features or components will be denoted by the same or corresponding reference numerals.

Fig. 1A shows a schematic flow chart of an image search method in the prior art.

Fig. 1B shows a flowchart of an image search method according to the first embodiment.

Fig. 2 shows a flow chart of a method of constructing a connected subgraph according to the first embodiment.

Fig. 3 shows a flow chart of an example of a method of calculating a confidence from an edge score according to the second embodiment.

Fig. 4 shows a flowchart of an example of a method of calculating a confidence from a combination of a node score and an edge score according to the third embodiment.

Fig. 5 shows a flowchart of an example of a method of calculating a node score of a connected subgraph using a similarity matrix according to the fourth embodiment.

Fig. 6 shows a schematic diagram of calculating the node scores of a connected subgraph using a similarity matrix according to a fourth embodiment.

Fig. 7 shows a flowchart of an example of a method of calculating a node score of a connected subgraph using image ranking scores according to the fifth embodiment.

Fig. 8 shows a schematic diagram of calculating a node score of a connected subgraph using image ranking scores according to a fifth embodiment.

Fig. 9 shows a flowchart of an example of a method of calculating a node score from a combination of a similarity matrix and an image ranking score according to the sixth embodiment.

Fig. 10 shows a flowchart of an example of a method of ordering a connected subgraph according to the number of nodes in the connected subgraph according to the seventh embodiment.

Fig. 11A depicts a flowchart of an example of connected subgraph ordering by summing similarities according to the eighth embodiment.

Fig. 11B shows a flowchart of an example of a method of ordering connected subgraphs under consideration of the weight of the nodes according to the eighth embodiment.

Fig. 12A shows a flowchart of an example of a method of performing an image search according to confidence and ranking information according to the ninth embodiment.

Fig. 12B shows a flowchart of an example of another method of conducting an image search according to confidence and ranking information according to a thirteenth embodiment.

Fig. 12C shows a schematic diagram of four categories into which an image set is divided according to node scores and edge scores according to the thirteenth embodiment.

Fig. 13 shows a flowchart of one example of post-processing according to the tenth embodiment.

Fig. 14 shows a flowchart of another example of post-processing according to the eleventh embodiment.

Fig. 15 shows a flowchart of still another example of post-processing according to the twelfth embodiment.

Fig. 16 shows a flowchart of an example of an image search method according to the fourteenth embodiment.

Fig. 17 shows a block diagram of an image search apparatus according to a fifteenth embodiment.

Fig. 18 shows a block diagram showing a hardware configuration of a computer system capable of implementing an embodiment of the present disclosure according to a sixteenth embodiment.

Detailed Description

Exemplary embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. In the interest of clarity and conciseness, not all features of an embodiment have been described in the specification. It should be appreciated, however, that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with device-related and business-related constraints, which may vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.

Here, it should also be noted that, in order to avoid obscuring the present disclosure with unnecessary detail, only process steps and/or system structures germane to at least the scheme according to the present disclosure are shown in the drawings, and other details not germane to the present disclosure are omitted.

Next, various aspects of the present disclosure will be described.

Patent document CN105512638A entitled "method for cleaning face recognition data based on maximum connected subgraph" discloses a method for cleaning face recognition data based on maximum connected subgraph. Fig. 1A shows a schematic flow diagram of the cleaning method.

As shown in fig. 1A, a set of images is obtained in step S101.

In step S102, the similarity between any two faces in the image set is calculated, and a similarity matrix is obtained.

Subsequently, in step S103, a connected subgraph of the face is constructed from the calculated similarity and a given similarity threshold.

Subsequently, in step S104, the largest connected subgraph, i.e. the connected subgraph containing the largest number of nodes, is found among all connected subgraphs.

Subsequently, in step S105, the image in the largest connected subgraph is saved as a search result.

This approach enables some degree of cleansing of the data in the image collection. However, this approach has a problem that sometimes the maximum connected subgraph is unreliable. For example, in a group photo image taken over a long distance, some faces look very similar due to low resolution, a uniform scene, etc., so that faces of different persons may be erroneously matched. The resulting maximally connected subgraph will contain facial image noise from different people. Since the noisy face image from the group photo will generally not match the face image within a single image, the connectivity of the edges of the connected subgraph containing such a noisy face is relatively sparse. Therefore, unlike the existing method, we do not directly use the largest connected subgraph (the connected subgraph with the largest number of nodes, where each face image is considered as a node) as the face search result, but use at least the information of the edges of the connected subgraph to define a confidence level, and use the confidence level to judge whether the highest-ranked connected subgraph obtained by the ranking algorithm contains noise. If there is noise, post-processing is used to reduce the noise.

First embodiment

Fig. 1B shows a flowchart of an image searching method according to the present embodiment.

As shown in fig. 1B, in step S111, a set of images is obtained. The set of images refers to a set of images from the same identity that contain one or more objects. Among them, an image containing more than one object is called a group image. The identifier here may be, for example, a name of the object or a preset number of the object. In one embodiment, the set of images is obtained by searching using keywords in a search engine. Alternatively or additionally, the image set may be obtained using other methods. In one embodiment, the object in the image may be a human face, for example. However, the object may be an animal or other object.

Subsequently, in step S112, the similarity between any two objects in the image set is calculated.

In one embodiment, calculating the similarity between two objects may be performed by comparing features of the two objects. For example, when the object is a human face, the calculation of the similarity may be implemented using, for example, a human face recognition system already existing in the prior art. The face recognition system specifically comprises the following steps performed in sequence: face detection, face alignment, face feature extraction and face feature comparison. The face feature comparison is to compare the similarity of the features of two input faces. The features of the face may be, for example, vectors of particular dimensions, the dimensions of which depend on the particular face feature method. The calculation of the similarity SIM may use, for example, the following formula:

SIM(i,j)＝[s(v_i,v_j)],i,j＝1,2,...N1 (1)

wherein i, j are two different input face images in the image set; v is a feature vector; n1 is the total number of images in the image set; s () is a method of comparing the degree of similarity of two vectors, and may be, for example, calculating a euclidean distance, a cosine distance, or the like between the two vectors.

Subsequently, in step S113, a connected subgraph of the objects in the image set is constructed from the computed similarity and a given similarity threshold.

In this step, the input is the calculated similarity SIM. And constructing a connected subgraph according to the similarity. As shown in FIG. 2, in one embodiment, constructing a connected subgraph comprises: in step S201, each object in the image set is regarded as a node, and in step S202, any two nodes satisfying the following conditions are connected with an edge: the similarity between the objects corresponding to the two nodes is greater than a given similarity threshold. Thus, any two nodes in a connected subgraph can be connected through a path formed by one or more edges. In this way, all connected subgraphs are found from the connection information of the nodes.

Subsequently, in step S114, the confidence of each connected subgraph is calculated using at least the information of the edges of the connected subgraph.

In step S115, the connected subgraphs are ranked to obtain ranking information.

Subsequently, in step S116, images in the image set are searched using the confidence and the ranking information.

Preferably, whether the confidence of the connected subgraph with the highest ranking is larger than a confidence threshold value is judged, and the connected subgraph and/or the image set are/is processed according to the judgment result.

The calculation of confidence, the obtaining of ranking information, and the image search will be described in detail later in this disclosure.

Note that, in fig. 1B, step S114 and step S115 are regarded as being sequentially performed, but are not limited thereto. For example, the execution order of step S114 and step S115 may be changed, or may be executed in parallel.

In this embodiment, by searching for images in the image set using the confidence and the ranking information, the problem of decreased accuracy of search results due to the presence of a group image in the image set is effectively suppressed.

Second embodiment

In this embodiment and the subsequent embodiments, the differences from the previous embodiments are mainly described and the description of the parts overlapping with the previous embodiments is not repeated.

In this embodiment, in step S114, the confidence of each connected subgraph is calculated using only the edge scores of the connected subgraph. The information of the edge may be, for example, an edge score.

Fig. 3 shows a flow diagram of an example of a method of computing confidence from edge scores according to an embodiment of the present disclosure.

As shown in fig. 3, in step S301, a connected subgraph is obtained.

Subsequently, in step S302, an edge score of the connected subgraph is calculated.

Subsequently, in step S303, the edge score is output as the confidence of the connected subgraph.

In calculating the edge score, each connected subgraph is input, and the confidence of each connected subgraph is output. The "edge score ES" is used as the confidence. In one embodiment, the "edge score ES" of the connected subgraph may be calculated, for example, using the following formula, but is not limited thereto:

wherein N2 represents the number of connected subgraphs; c_kThe kth connected subgraph is shown;

representing the number of edges of the kth connected subgraph; n is_kThe number of nodes of the kth connected subgraph is indicated.

The confidence of the connected subgraph is calculated using equation (2) because a noisy object (e.g., a human face) in the group image usually cannot match with most objects in the image of a single object, and therefore, if the connected subgraph contains more objects of the group image, the edges of the connected subgraph are relatively sparse. And ES (k) in equation (2) may characterize how dense the edges in the connected subgraph are. That is, when the value of es (k) is higher, the density of edges of the connected subgraph is also higher, i.e., there are fewer objects from the group image in the connected subgraph, and thus the confidence is higher. And vice versa.

The edge scores of the connected subgraphs are used as confidence degrees, and the sparsity degree of the edges in the connected subgraphs can be effectively measured, so that the dense connected subgraphs can be obtained, and the accuracy of image searching is improved.

Note that equation (2) is just one example of a method of calculating confidence using edge scores. In practical applications, the confidence level may be calculated by obtaining an edge score from the information of the edge using any suitable method.

Third embodiment

Unlike the second embodiment, in the present embodiment, in step S114, the confidence is calculated using a combination of information of nodes and information of edges of a connected subgraph.

In one embodiment, the combination may be a formula with weight, e.g., information of w1 node + information of w2 edge, where 0 ≦ w1 ≦ 1, and w2 ≦ 1-w 1.

In another embodiment, the combination is implemented by a classifier, where the information of the nodes and the information of the edges are used as input features of the classifier.

Note that the implementation of this combination is not limited to the above two ways. In practical applications, the information of the nodes and the information of the edges may be combined using any suitable method.

In one embodiment, the information of the edge may be, for example, an edge score. In one embodiment, the information of the node may be, for example, a node score.

Fig. 4 shows a flowchart of an example of a method of calculating confidence from a combination of node scores and edge scores, according to an embodiment of the present disclosure.

As shown in fig. 4, in step S401, a connected subgraph is obtained.

Subsequently, in step S402, the edge score and the node score of the connected subgraph are calculated.

An example of a method of calculating the edge score has been described in detail in the second embodiment of the present disclosure, and is not repeated here.

Preferably, the node score of a connected subgraph can be the ratio of the weight of the connected subgraph to the maximum weight in all connected subgraphs, but is not limited thereto. In practical applications, any suitable method may be used to compute the node scores of the connected subgraph.

In one embodiment, the similarity matrix may be used to compute the weights of the connected subgraphs. In another embodiment, the image ranking scores provided by the search engine may be used to compute the weights of the connected subgraphs. In yet another embodiment, a combination of the similarity matrix and the image ranking score may be used to compute the weight of the connected subgraph. In practical applications, any suitable method may be used to compute the weights of the connected subgraphs.

The specific manner in which the node scores are calculated will be described in detail below in this disclosure.

Subsequently, in step S403, a combination of the edge score and the node score is output as the confidence of the connected subgraph.

In the embodiment, by comprehensively considering the edge score and the node score, the confidence degree calculation method can be freely adjusted as required, and the accuracy and flexibility of image search are further improved.

Fourth embodiment

In this embodiment, the node scores of the connected subgraph are computed, for example, using a similarity matrix.

FIG. 5 shows a flowchart of an example of a method of computing a node score of a connected subgraph using a similarity matrix, according to an embodiment of the present disclosure.

As shown in fig. 5, in S501, a connected subgraph and a similarity matrix are obtained. The similarity matrix may be derived using, for example, formula (1) in the first embodiment, but is not limited thereto.

Subsequently, in step S502, the weights of the connected subgraphs are computed by summing the similarity between any two connected nodes in each connected subgraph.

Subsequently, in step S503, a node score is calculated by comparing the weight of each connected subgraph with the largest weight in all connected subgraphs. In one embodiment, the comparison may be a ratio of the weight of each connected subgraph to the largest weight in all connected subgraphs, but is not so limited.

Fig. 6 shows a schematic diagram of calculating the node scores of the connected subgraph using the similarity matrix according to the present embodiment.

As shown in fig. 6, the connected sub-diagram SG 1 has four nodes ND 1 to ND 4, the connected sub-diagram SG 2 has three nodes ND ' 1, ND ' 2, and ND ' 3, and the connected sub-diagram SG 3 has two nodes ND "1 and ND" 2. The similarity matrix between the nodes of the connected subgraph SG 1 is shown in the figure. It can then be calculated from this that the weight of the connected subgraph SG 1 is equal to the sum of the similarities of the nodes, i.e. W_SG1＝0.7+0.6+0.8+0.8+0.7+0.6＝4.2。

Similarly, the weight W of the connected subgraph SG 2 can be computed_SG20.8+0.7+ 0.6-2.1, and weight W of SG 3_SG3＝0.6。

And finally, dividing the weight of each connected subgraph by the maximum weight to obtain the node score of the connected subgraph. It can be seen that the node score of the connected sub-graph SG 1 is 4.2/4.2 equals 1, the node score of the connected sub-graph SG 2 is 2.1/4.2 equals 0.5, and the node score of the connected sub-graph SG 3 is 0.6/4.2 equals 0.14.

In the embodiment, the node score of the connected subgraph is calculated according to the similarity between the nodes, so that the confidence of the connected subgraph is effectively reflected, and more accurate search results are obtained.

Fifth embodiment

Unlike the fourth embodiment, in the present embodiment, the node scores of the connected subgraph are calculated using the image ranking scores provided by the search engine.

Fig. 7 illustrates a flowchart of an example of a method of computing a node score for a connected subgraph using image ranking scores, according to an embodiment of the present disclosure.

As shown in fig. 7, in S701, connected subgraphs and image ranking scores are obtained.

Subsequently, in step S702, a weight is calculated by summing the image rankings corresponding to any one node in each connected subgraph.

Subsequently, in step S703, a node score is calculated by comparing the weight of each connected subgraph with the maximum weight in all connected subgraphs.

Fig. 8 shows a schematic diagram for calculating the node score of a connected subgraph using the image ranking score according to the present embodiment.

As shown in fig. 8, the connected sub-diagram SG 1 has four nodes ND 1 to ND 4, the connected sub-diagram SG 2 has three nodes ND ' 1, ND ' 2, and ND ' 3, and the connected sub-diagram SG 3 has two nodes ND "1 and ND" 2. The image ranking scores for the various nodes are shown. Thus, the weight of the connected subgraph SG 1 is calculated to be equal to the sum of the image ranking scores of all nodes in SG 1, namely W_SG1＝100+90+80+70＝340。

Similarly, the weight W of the connected subgraph SG 2 can be computed_SG260+50+ 40-150, and weight W of SG 3_SG3＝30+20＝50。

And finally, dividing the weight of each connected subgraph by the maximum weight to obtain the node score of the connected subgraph. It can be seen that the node score of the connected sub-graph SG 1 is 340/340 ═ 1, the node score of the connected sub-graph SG 2 is 150/340 ═ 0.44, and the node score of the connected sub-graph SG 3 is 50/340 ═ 0.15.

In the present embodiment, the node score is calculated using the image ranking score. The image ranking score represents the confidence that the image given by the search engine is the search result for a given token. The images at the top of the search listing have a higher image ranking score, that is, the confidence in the search results for a given token is also high. Therefore, more accurate search results can be obtained.

Sixth embodiment

Unlike the fourth and fifth embodiments, in the present embodiment, the node score is calculated, for example, by a combination of the sum of the similarities between any two connected nodes in a connected subgraph and the sum of the image ranking scores of the respective nodes in each connected subgraph.

Fig. 9 is a flow diagram of an example of a method of computing a node score from a combination of a similarity matrix and an image ranking score, according to an embodiment of the disclosure.

As shown in fig. 9, in step S901, connected subgraphs, similarities, and image ranking scores are obtained.

In step S902, a node score is calculated by using a combination of the similarity and the image ranking score. Preferably, a combination of similarity and image ranking scores may be used to compute the weight of a connected subgraph, and then its node scores may be computed from the weights of the connected subgraph using, for example, a method similar to the fourth and fifth embodiments. The combination may be, for example, a weighted sum, but is not limited thereto.

Note that in the exemplary description of the fourth to sixth embodiments, the node score is calculated for each connected subgraph, but is not limited thereto. In one embodiment, since the determination and processing are performed in step S116 of fig. 1 only for the confidence of the connected subgraph ranked the highest (e.g., having the greatest weight) in step S115, only the node score of the highest ranked connected subgraph may be calculated. For example, the node score of the connected subgraph with the greatest weight can be calculated using the following formula:

where W _ max is the maximum weight and W _ second is the maximum weight (i.e., the second largest weight) other than W _ max.

As shown in Table 1, connectivity sub-graph C₄With maximum weight of 10.0, connected subgraph C₂Has the second largest weight of 9.4, and thus, mayTo compute only the connectivity sub-graph C₄The node score of (2): 10.0/9.4 is 1.06.

By adopting the method, the computing resources can be effectively saved, the computing time is shortened, and the searching efficiency is improved.

TABLE 1 example of computing only the node scores for the connected subgraph with the greatest weight

Seventh embodiment

In the present embodiment, an example of a method of sorting connected subgraphs in step S115 will be described.

Preferably, the connected subgraphs can be sorted according to the number of nodes in the connected subgraphs.

FIG. 10 shows a flowchart of an example of a method of ordering a connected subgraph according to the number of nodes in the connected subgraph, according to an embodiment of the present disclosure.

In this process, the input is each connected subgraph. And outputting the sorting information for each subgraph.

As shown in fig. 10, in step S1001, a connected subgraph is obtained.

Subsequently, in step S1002, the number of nodes of each connected subgraph is calculated.

Subsequently, in step S1003, the connected subgraphs are sorted according to the number of nodes in the connected subgraphs, and the sorting information is obtained.

Preferably, in step S1003, the connected subgraphs are sorted in the order of the node number from large to small.

Table 2 gives an example of the sorted results with 4 connected subgraphs.

TABLE 2 ranking results of four connected subgraphs

Connected subgraph	Number of nodes	Results of the sorting
			C1	6	rank3
C2	12	rank2
			C3	5	rank4
C4	14	rank1

As shown in Table 2, C₄Has the maximum number of nodes 14 and is therefore the first candidate connected subgraph. C₂Has the second largest number of nodes 12 and is thus the second candidate connected subgraph. And so on.

Eighth embodiment

Unlike the seventh embodiment, in the present embodiment, connected subgraphs are sorted according to the similarity between nodes in the connected subgraphs.

Preferably, the connected subgraphs can be sorted by summing the similarity between any two connected nodes in each connected subgraph.

FIG. 11A depicts a flow diagram of an example of sorting by summing similarity.

As shown in fig. 11A, in step S1101, a connected subgraph and a similarity between any two nodes in the connected subgraph are obtained.

Subsequently, in step S1102, the weight of the connected subgraph is calculated by summing the similarities between any two connected nodes in each connected subgraph.

Subsequently, in step S1103, the connected subgraphs are sorted according to the weights of the connected subgraphs, and sorting information is obtained.

Preferably, the weight of each node can be calculated by using the similarity between the nodes, and then the weight of the connected subgraph is calculated according to the weight of the node and is sorted according to the weight.

Fig. 11B illustrates a flow diagram of an example of a method of ordering connected subgraphs taking into account the weights of the nodes according to an embodiment of the present disclosure.

As shown in fig. 11B, first, in step S1111, a weight value of each node is calculated.

Subsequently, in step S1112, the weight of each connected subgraph is calculated according to the weight of the node.

Subsequently, in step S1113, all connected subgraphs are sorted from large to small according to the weight of the connected subgraphs. This is explained in turn below.

1. Calculating the weight of each node

In one embodiment, in step S1111, for any given node a, the weight w (a) of the node a may be calculated using the following formula:

wherein b is a node directly connected to node a with an edge, N_NUMIs the total number of all nodes that have an edge directly connected to node a. SIM (a, b) is the similarity between nodes a and b, e.g. the euclidean distance of the features of nodes a and b. Therefore, w (a) characterizes the average degree of similarity of node a to the respective nodes with which it communicates.

In another embodiment, the image collection is from an image search engine and the image ranking score for each image given by the search engine is recorded. Therefore, in step S1111, the image ranking score R may be considered in calculating the weight of the node.

Preferably, the weight w (a) of the node a is the image ranking score r (a).

Alternatively, the weight w (a) of the node a may be determined according to the similarity SIM and the image ranking score R, for example.

One exemplary description of the method of determining the weight of node a according to the similarity SIM and the image ranking score R is as follows:

for all nodes of a connected subgraph, first, kernel functions are usedThe image ranking score R of each node is mapped to the initial weight wr of the node. Kernel function

Is a monotonous non-negative function in a finite positive integer domain, and can be, for example, a constant kernel, a linear kernel, an exponential kernel, an inverse kernel, and the like. For any given node a, the mapping formula may be expressed, for example, as follows:

wherein, R (a) is the image ranking score of the image corresponding to the node a; n4 is the total number of nodes in the connected subgraph.

Then, the weight w (a) of the node a is calculated as follows:

expressed as a matrix of the form:

w＝SIM·wr (8)

in order to obtain a robust result, for example, weight iteration may be performed, and the flow of one iteration is as follows:

firstly, updating the weight wr by w;

subsequently, w is normalized and iterated using the following formula:

multiple iterations may be performed to obtain the weight w of the node.

2. Computing weights for connected subgraphs

In one embodiment, in step S1112, the weight of each connected subgraph can be calculated from the weight of the node using the following formula:

wherein, w_k,aRepresenting the weight of the a node of the k connected subgraph; n is a radical of_kThe total number of nodes for the kth connected subgraph is represented.

3. Weight ordering according to connected subgraphs

Table 3 gives an example of the results of ordering 4 connected subgraphs.

As can be seen from Table 3, C₄Is the highest, and thus the rank is the first candidate. C₂Is weighted next to C₄The second candidate, and so on.

TABLE 3 example ordering respective connected subgraphs according to weight

Connected subgraph	Weight of	Sorting
			C1	3.2	rank3
C2	9.4	rank2
			C3	2.5	rank4
C4	10.0	rank1

It should be noted that the method of ordering connected subgraphs described in the seventh and eighth embodiments is merely exemplary. In practical applications, the connected subgraphs can be sorted using any suitable method. For example, the connected subgraph may be ordered according to the results of a combination (e.g., a weighted sum) of similarities between nodes in the connected subgraph and the number of nodes, or any other method may be used.

Ninth embodiment

In the present embodiment, a method of searching for an image in an image set using confidence and ranking information in step S116 of the first embodiment will be described in detail.

Fig. 12A shows a flowchart of an example of a method of performing an image search according to the confidence and ranking information according to the present embodiment.

As shown in fig. 12A, in step S1201, the sorted connected subgraphs and confidences are obtained.

Subsequently, in step S1202, a first candidate connected subgraph is found.

Subsequently, in step S1203, it is determined whether the confidence of the first candidate connected subgraph is greater than a confidence threshold. If so, proceed to step S1204; if not, the process proceeds to step S1205. The confidence threshold may be obtained empirically or experimentally, for example.

In step S1204, the first candidate connected subgraph is saved as a search result, and the process ends.

In step S1205, post-processing is performed on the connected subgraph whose confidence is smaller than the confidence threshold.

Tenth embodiment

In the present embodiment, the post-processing method in step S1205 of the ninth embodiment will be described in detail.

Preferably, the post-processing comprises removing objects from the group image in the connected subgraph.

Fig. 13 shows a flowchart of one example of post-processing according to the present embodiment.

As shown in fig. 13, in step S1301, a first candidate connected subgraph is found.

Subsequently, in step S1302, it is determined whether the image where the node in the connected subgraph is located is a group photo image. If yes, go to step S1303; if not, the process ends.

In step S1303, the group image is removed.

Subsequently, in step S1304, the connected subgraph from which the ghost image is removed is saved, that is, the result of the image search is obtained.

The connected subgraph obtained by the method has no object in the group picture image, so that the searching accuracy is effectively improved.

Alternatively, after step S1303, step S1304 is not performed, but the image search may be re-performed using the methods described in the embodiments of the present disclosure as needed to obtain more accurate search results.

Eleventh embodiment

Unlike in the tenth embodiment, in the present embodiment, the post-processing includes: the connected subgraph is reconstructed to perform the search. For example, in step S112 of the first embodiment, a greater weighting value may be given to the degree of similarity between objects from images containing only one object in the image set.

Fig. 14 shows an example of another method of post-processing according to an embodiment of the present disclosure.

As shown in fig. 14, in step S1401, an image set is obtained.

In step S1402, the similarity between any two objects in the image is calculated, resulting in a similarity matrix.

In step S1403, a greater weight (e.g., 1.1 times) is given to the similarity between objects from the images containing only one object in the image set to modify the similarity matrix.

Subsequently, in step S1404, the modified similarity is compared with an adjusted higher threshold (e.g., 1.1 times), thereby constructing a connected subgraph.

Note that the threshold value may be adjusted in the same manner or in a different manner as the similarity weighting value, or the threshold value and the weighting value may be adjusted using any suitable method as needed.

Subsequently, in step S1405, image search is performed anew using the constructed connected subgraph.

Additionally or alternatively, a smaller weighting value (e.g., 0.9 times) may be assigned to the similarity between objects from images containing more than one object (group images) in the image set.

By the method, the influence of the false recognition in the group photo image in the image set on the accuracy of the search result can be effectively inhibited.

Twelfth embodiment

Unlike the tenth and eleventh embodiments, in the present embodiment, the post-processing includes breaking the bridge connection in the highest-ranked connected subgraph.

The concept of a bridge is first defined. For the first candidate connected subgraph G, the set of edges is defined as E. If there is an edge set E1, the subgraph in the graph G after deleting all the edges in E1 is not connected, and the subgraph obtained after deleting any proper subset in E1 is connected, then the E1 is called an edge cut set of G. If a certain edge forms an edge cut set, the edge is called a bridge.

Fig. 15 shows a flowchart of yet another example of a method of post-processing according to an embodiment of the present disclosure.

As shown in fig. 15, in step S1501, the bridge in the highest-ranked connected subgraph is disconnected, and a plurality of connected subgraphs that do not communicate with each other are obtained.

In step S1502, the plurality of connected subgraphs obtained in step S1501 are ranked to find a first candidate connected subgraph.

Preferably, the plurality of connected subgraphs are sorted using a sorting method as described in various embodiments of the present disclosure.

In step S1503, the first candidate connected subgraph (the highest-ranked connected subgraph) obtained by ranking is saved as a search result.

By the method, the interference of the object in the noise image (such as a group photo image) can be effectively reduced, and more accurate search results can be obtained.

Thirteenth embodiment

In the present embodiment, another method of searching for images in an image set using confidence and ranking information in step S116 of the first embodiment will be described in detail.

Different from the ninth embodiment, in this embodiment, the first candidate connected subgraph is obtained according to the ranking information, the first candidate connected subgraph is classified according to the edge score and/or the node score used in the confidence calculation, and the image search is performed using the classification result.

Fig. 12B shows a flowchart of an example of another method of performing an image search according to the confidence and ranking information according to the present embodiment.

As shown in FIG. 12B, in step S1211, the sorted connected subgraphs are obtained.

Subsequently, in step S1212, a first candidate connected subgraph is found according to the sorting result of each connected subgraph.

Subsequently, in step S1213, edge scores and/or node scores of the first candidate connected subgraph are calculated.

Preferably, the edge score may be calculated using formula (2) of the present disclosure, and the node score may be calculated using formula (3), but is not limited thereto.

Subsequently, in step S1214, the set of images is classified using a classifier.

In one embodiment, the input image set may be divided into four categories based on the node score and the edge score of the first candidate connected subgraph, as shown in fig. 12C.

Class 1 (frangible set 1221) in which the nodes representing real objects can be densely connected by edges to form the largest connected subgraph, while the nodes representing objects in noisy pictures are fairly sparsely connected (less connected to each other).

Category 2 (multiple identity set 1222): in this class of images, two to three main connected subgraphs may be formed. A typical example of this category is searching for a member of a band, which, in general, is likely to frequently appear in the same photo. Thus, many similar connected subgraphs may occur.

In this case, preferably, the method of the eleventh embodiment of the present disclosure may be adopted for denoising, that is, a greater weight may be given to the similarity between objects from an image containing only one object in the image set.

Category 3 (weak link set 1223): in this class of images, although only one main connected subgraph is formed, the nodes of this connected subgraph are not densely connected by edges. There are two reasons for this. First, the nodes may be from a co-photograph where all facets look similar. Secondly, two small connected subgraphs happen to be connected by a "bridge", thus forming a large connected subgraph. In this case, the methods of the tenth and twelfth embodiments of the present disclosure may be adopted for denoising.

Category 4 (scatter gather 1224): in images of this category, almost all nodes are scattered (there are fewer connections between each other). Thus, the images of the category can be directly removed. Here, Category 1 is referred to as clean (clear) set; category 2, category 3, and category 4 are collectively referred to as a rough (rough) set.

In another embodiment, a substantial portion of category 2 and category 3 will transition to category 1 or to category 4 as a result of multiple iterations through equation (9) of the present disclosure. Therefore, preferably, a classifier (such as an SVM) can be used to classify the input image set into two categories: class 1 and non-class 1. Here, Category 1 is referred to as a clean set; non-class 1 is referred to as a coarse set.

Subsequently, in step S1215, search processing is performed according to the classification result.

Preferably, for a clean set, a first candidate connected subgraph is selected as the search result. For a coarse set, the first candidate connected subgraph may be selected as a search result, or a certain proportion of data (e.g., images) may be removed as a search result. For example, the lowest score of 60% of the images may be removed. Preferably, the score here may be a weight of the node determined by, for example, formula (4) or formula (6), or may be an image ranking score provided by a search engine, but is not limited thereto.

In the embodiment, various noise images can be effectively removed by classifying the connected subgraphs and performing different denoising processing and searching processing based on the classes of the connected subgraphs, so that a very accurate searching result is obtained.

Fourteenth embodiment

In the present embodiment, an image search method will be described. The image searching method comprises the following steps: dividing the set of images into a plurality of groups according to the labels, wherein the images in each group have the same label; and performing an image search using a method according to any one or any combination of embodiments of the present disclosure.

Fig. 16 shows a flowchart of an example of an image search method according to an embodiment of the present disclosure.

As shown in fig. 16, in step S1601, the image set is divided into a plurality of groups according to the tags. Subsequently, in step S1602, an image search is performed using a method according to any one or any combination of the embodiments of the present disclosure.

It should be noted that the steps of the methods according to the various embodiments of the present disclosure are not necessarily performed in the order shown, but may be performed in parallel or in other orders.

Fifteenth embodiment

As shown in fig. 17, in the present embodiment, an image search apparatus 1700 is described. The apparatus 1700 includes: image obtaining means 1701 for obtaining an image collection, similarity calculating means 1702 for calculating a similarity between any two objects in the image collection, connected subgraph constructing means 1703 for constructing connected subgraphs of the objects in the image collection according to the calculated similarity and a given similarity threshold, confidence calculating means 1704 for calculating a confidence of each connected subgraph using at least information of the edges of the connected subgraph, ranking means 1705 for ranking the connected subgraphs to obtain ranking information, and searching means 1706 for searching the images in the image collection using the confidence and the ranking information.

The apparatus described in the fifteenth embodiment above is an exemplary and/or preferred apparatus for carrying out the methods described in this disclosure. These devices can achieve similar effects to the corresponding methods. These means may be hardware elements, such as field programmable gate arrays, digital signal processors, application specific integrated circuits or computers, etc., and/or software means, such as computer readable programs. The apparatus for performing the various steps has not been described in detail above. However, as long as there is a step of performing a certain process, there may be corresponding means (implemented by hardware and/or software) for implementing the same process. All the technical solutions defined by all the combinations of the described steps and the devices corresponding to the steps are included in the disclosure of the present disclosure as long as they constitute the technical solutions are complete and applicable.

Further, the above-described apparatus constituted by various means may be incorporated as a functional module into a hardware device such as a computer. In addition to these functional modules, the computer may of course have other hardware or software components.

Sixteenth embodiment

Fig. 18 is a block diagram showing a hardware configuration of a computer system capable of implementing an embodiment of the present disclosure.

As shown in fig. 18, the computer system includes a processing unit (processor) 1801, a read-only memory 1802, a random access memory 1803, and an input/output interface 1805 connected via a system bus 1804, and an input unit 1806, an output unit 1807, a storage unit 1808, a communication unit 1809, and a driver 1810 connected via the input/output interface 1805. The program may be recorded in advance in a ROM (read only memory) 1802 or a storage unit 1808 as a recording medium built in the computer. Alternatively, the program may be stored (recorded) in the removable medium 1811. Herein, the removable medium 1811 includes, for example, a flexible disk, a CD-ROM (compact disc read only memory), an MO (magneto optical) disk, a DVD (digital versatile disc), a magnetic disk, a semiconductor memory, and the like.

The input unit 1806 is used to input a user request and is configured with a keyboard, a mouse, a touch screen, a microphone, a camera, and the like. In addition, the output unit 1807 is configured with an LCD (liquid crystal display), a speaker, and the like.

The communication unit 1809 may be, for example, a wireless communication unit including at least one transceiver module and a positioning module. The transceiver module is used to send requests to the remote server and receive responses from the remote server. The positioning module is, for example, a GPS module for acquiring a position.

The storage unit 1808 or the ROM 1802 stores images, audio, and the like. The RAM 903 may store temporary state information and intermediate calculation results.

Further, in addition to the configuration in which the program is installed from the above-mentioned removable medium 1811 to the computer system through the drive 1810, the program may be downloaded to the computer system through a communication network or a broadcast network to be installed in the built-in storage unit 1808. In other words, the program may be transmitted from a download point to the computer system by a satellite for digital satellite broadcasting, for example, in a wireless manner, or may be transmitted to the computer system by a wired manner through a network such as a LAN (local area network) or the internet.

If a command is input to the computer system via the input/output interface 1805 by user manipulation or the like to the input unit 1806, the CPU 1801 executes a program stored in the ROM 1802 in accordance with the command. Alternatively, the CPU 1801 loads a program stored in the storage unit 1808 on the RAM 1803 to execute the program.

Therefore, the CPU 1801 executes some processing according to the above-mentioned flowchart or processing executed by the above-mentioned configuration of the block diagram. Next, the CPU 1801 allows the result of the processing to be output from the output unit 1807, transmitted from the communication unit 1809, recorded in the storage unit 1808, and the like, if necessary, for example, through the input/output interface 1805.

In addition, the program may be executed by a computer (processor). In addition, the program may be processed by a plurality of computers in a distributed manner. In addition, the program may be transferred to a remote computer for execution.

The computer system shown in FIG. 18 is illustrative only and is in no way intended to be limiting of the present disclosure, its application, or uses. The computer system shown in fig. 18 may be implemented in any embodiment, as a stand-alone computer, or as a processing system in a device, and one or more unnecessary components may be removed or one or more additional components may be added thereto.

In one example, a computer system is implemented as an apparatus for image searching. The apparatus comprises a processor and a memory, the memory storing a computer program which, when executed by the processor, is capable of causing the apparatus to perform a method according to an embodiment of the disclosure.

The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination thereof. The order of the method steps described above is merely illustrative, and the method steps of the present disclosure are not limited to the order specifically described above unless explicitly stated otherwise. Furthermore, in some embodiments, the present disclosure may also be implemented as a computer program stored in a computer-readable storage medium. The computer programs, when executed by the processor 1801, enable the processor 1801 to perform methods in accordance with any embodiment or any combination of embodiments of the present disclosure. Thus, the present disclosure also covers a computer-readable storage medium storing a computer program for implementing the method according to the present disclosure.

Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the foregoing examples are illustrative only and are not limiting upon the scope of the disclosure. It will be appreciated by those skilled in the art that the above-described embodiments may be modified without departing from the scope and spirit of the disclosure. The scope of the present disclosure is defined by the appended claims.