阅读说明：本技术 支持大模板匹配的迭代多方向图像搜索 (Iterative multi-directional image search supporting large template matching ) 是由 邱天 于 2019-08-07 设计创作，主要内容包括：描述了迭代多方向图像搜索的系统和方法。实施例使用多方向搜索模式来确定源图像内的一个或多个搜索区域。为在多方向搜索模式的多个方向上搜寻模板图像,实施例的搜索区域提供源图像内的搜索区域。可以迭代地更新源图像内搜索区域的位置,例如基于从搜索导出的运动向量,直到在源图像内识别出模板图像匹配位置。实施例将模板图像和源图像内的搜索区域转换为与多方向图像搜索的每个方向相对应的1D表示。实施例适应源图像内的模板图像的主体(如感兴趣目标)的旋转和缩放变化。(Systems and methods of iterative multi-directional image search are described. Embodiments use a multidirectional search pattern to determine one or more search regions within a source image. To search for a template image in multiple directions in a multi-directional search mode, the search area of an embodiment provides a search area within a source image. The location of the search area within the source image may be iteratively updated, e.g., based on motion vectors derived from the search, until a template image match location is identified within the source image. Embodiments convert a search area within a template image and a source image into a 1D representation corresponding to each direction of a multi-directional image search. Embodiments accommodate rotational and scaling variations of a subject (e.g., an object of interest) of a template image within a source image.)

1. A method for template image matching within a source image, comprising:

converting the template image into a plurality of one-dimensional (1D) representations corresponding to a plurality of image search directions;

defining a search area within the source image for searching the template image in the plurality of image search directions, wherein the search area is a sub-portion of a search range and is configured for image searching in each of the plurality of search directions;

converting the search area of the source image into a plurality of 1D representations corresponding to the plurality of image search directions;

performing a correlation between a 1D representation of the template image for each of the plurality of image search orientations and a 1D representation of the search area of the source image for each respective one of the plurality of image search orientations;

analyzing the correlation to determine whether a match of the template image within the source image is indicated;

outputting matching pattern information for the template image if a match of the template image within the source image is indicated.

2. The method of claim 1, wherein the plurality of image search directions comprises a first direction of the multi-directional search and a second direction of the multi-directional search, wherein converting the template image to a 1D representation comprises:

converting the template image into a first direction template image pixel 1D vector for a first direction of the multi-directional search;

converting the template image into a second direction template image pixel 1D vector for a second direction of the multi-directional search.

3. The method of claim 2, wherein converting the search area of the source image into a plurality of 1D representations comprises:

converting the search area of the source image into a first direction source image pixel 1D vector for a first direction of the multi-directional search; and

converting the search area of the source image to a second direction source image pixel 1D vector for a second direction of the multi-directional search.

4. The method of claim 3, wherein a first direction of the multi-directional search and a second direction of the multi-directional search are orthogonal.

5. The method of claim 4, wherein a first direction of the multi-directional search is a horizontal direction and a second direction of the multi-directional search is a vertical direction.

6. The method of claim 4, wherein the search area comprises a first direction search portion corresponding to the multi-directional search first direction and a second direction search portion corresponding to the multi-directional search second direction, wherein the first direction search portion and the second direction search portion intersect each other.

7. The method of claim 4, wherein the search area comprises a plurality of first direction search portions corresponding to first directions of the multi-directional search and a plurality of second direction search portions corresponding to second directions of the multi-directional search, wherein each of the plurality of first direction search portions intersects at least one of the plurality of second direction search portions.

8. The method of claim 1, further comprising:

rotating the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a rotational variation of a subject of the template image, wherein the converting the template image into a plurality of 1D representations corresponding to a plurality of image search directions comprises:

converting each of the plurality of instances of the template image into a plurality of 1D representations corresponding to the plurality of image search directions for determining whether a match of the template image within the source image is indicated when undergoing a rotational change of a subject of the template image.

9. The method of claim 1, further comprising:

scaling the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a change in scale of a subject of the template image, wherein the converting the template image into a plurality of 1D representations corresponding to a plurality of image search directions comprises:

converting each of the plurality of instances of the template image into a plurality of 1D representations corresponding to the plurality of image search directions for determining whether a match of the template image within the source image is indicated when undergoing a change in scale of a subject of the template image.

10. The method of claim 1, further comprising:

iteratively performing a multi-directional image search within the search range using a 1D representation of the template image if the template image is not indicated as matching within the source image.

11. The method of claim 10, wherein the iteratively performing a multi-directional image search comprises:

repositioning the search area within a search range in the source image;

performing the further iteration of converting the search region of the source image into a plurality of 1D representations, performing a correlation between the 1D representation of the template image and the 1D representation of the search region of the source image, and analyzing the correlation.

12. The method of claim 11, wherein said repositioning of the search area within the search range in the source image is based on: a motion vector for each of the plurality of image search directions derived from analyzing a correlation between the 1D representation of the template image for each of the plurality of image search directions and the 1D representation of the source image search area for each corresponding one of the plurality of image search directions.

13. The method of claim 12, wherein said repositioning a search area within said source image comprises:

moving a first direction search portion of the search area corresponding to a first direction of the multi-directional search based on a second motion vector of a second direction of the plurality of search directions; and

moving a second direction search portion of the search area corresponding to a second direction of the multi-directional search based on a first motion vector of a first direction of the plurality of search directions.

14. The method of claim 1, wherein the correlation comprises a cross-correlation.

15. The method of claim 1, wherein a correlation criterion for the correlation is selected from the group consisting of:

sum of Absolute Differences (SAD);

mean Absolute Error (MAE); and

mean Square Error (MSE).

16. The method of claim 1, wherein the width or height of the template image is at least 1/3 of the search range, so the template image match is a large template case.

17. A system for template image matching within a source image, the system comprising:

at least one processor; and

a memory connected to the at least one processor, wherein the at least one processor is configured to:

converting the template image into a plurality of one-dimensional (1D) representations corresponding to a plurality of image search directions;

defining a search region within the source image for searching the template image in the plurality of image search directions, wherein the search region is a sub-portion of the search range and is configured for image searching in each of the plurality of search directions;

converting the search area of the source image into a plurality of 1D representations corresponding to the plurality of image search directions;

performing a correlation between a 1D representation of the template image for each of the plurality of image search orientations and a 1D representation of a search area of the source image for each corresponding one of the plurality of image search orientations;

analyzing the correlation to determine whether a match of the template image within the source image is indicated;

outputting matching pattern information for the template image if a match of the template image within the source image is indicated.

18. The system of claim 17, wherein the plurality of image search directions comprises a first direction of the multi-directional search and a second direction of the multi-directional search, and wherein the at least one processor is further configured to:

converting the template image into a first direction template image pixel 1D vector for a first direction of the multi-directional search;

converting the template image into a second direction template image pixel 1D vector for a second direction of the multi-directional search;

converting a search area of the source image to a first direction source image pixel 1D vector for a first direction of the multi-directional search;

converting the search area of the source image to a second direction source image pixel 1D vector for a second direction of the multi-directional search.

19. The system of claim 17, wherein the search area comprises a first direction search portion corresponding to a first direction of the multi-directional search and a second direction search portion corresponding to a second direction of the multi-directional search, wherein the first direction search portion and the second direction search portion intersect each other.

20. The system of claim 17, wherein the at least one processor is further configured to:

rotating the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a rotational variation of a subject of the template image;

converting each of the plurality of instances of the template image into a plurality of 1D representations corresponding to the plurality of image search directions to determine whether a match of the template image within the source image is indicated when experiencing a rotational change in a subject of the template image.

21. The system of claim 17, wherein the at least one processor is further configured to:

scaling the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a change in scale of a subject of the template image;

converting each of the plurality of instances of the template image into a plurality of 1D representations corresponding to the plurality of image search directions to determine whether a match of the template image within the source image is indicated when undergoing a change in scale of a subject of the template image.

22. The system of claim 17, wherein the at least one processor is further configured to:

iteratively performing a multi-directional image search within the search range using a 1D representation of the template image if a match of the template image within the source image is not indicated.

23. The system of claim 22, wherein the at least one processor configured to iteratively perform a multi-directional image search is configured to:

repositioning the search area within the search range of the source image;

performing the further iteration of converting the search area of the source image into a plurality of 1D representations, performing a correlation between a 1D representation of the template image and a 1D representation of the search area of the source image, and analyzing the correlation.

24. The system of claim 23, wherein repositioning a search area within the search range of the source image is based on a motion vector for each of the plurality of search directions derived from analyzing a correlation between the template image 1D representation for each of the plurality of image search directions and a 1D representation of the source image search area for each corresponding direction of the plurality of image search directions.

25. The system of claim 24, wherein the at least one processor configured to reposition the search area within the source image is configured to:

moving a first direction search portion of the search area corresponding to the multi-directional search first direction based on a second motion vector for a second direction of the plurality of search directions;

moving a second direction search portion of the search area corresponding to the multi-directional search second direction based on a first motion vector for a first direction of the plurality of search directions.

26. A method for template image matching within a source image, the method comprising:

converting the template image into a horizontal template image pixel one-dimensional (1D) vector and a vertical template image pixel (1D) vector for a multi-directional search within the source image;

defining a search area within the source image for searching the template image in horizontal and vertical search directions, wherein the search area is a sub-portion of a search range and includes a horizontal search portion corresponding to the horizontal search direction and a vertical search portion corresponding to the vertical search direction, wherein the horizontal search portion and the vertical search portion intersect one another;

converting the search area of the source image into a horizontal source image pixel 1D vector and a vertical source image pixel 1D vector for the multidirectional search;

performing a correlation between the horizontal template image pixel 1D vector and the horizontal source image pixel 1D vector;

performing a correlation between the vertical template image pixel 1D vector and the vertical source image pixel 1D vector;

analyzing the correlation to determine whether a match of the template image within the source image is indicated;

outputting matching pattern information for the template image if a match of the template image within the source image is indicated.

27. The method of claim 26, further comprising:

rotating the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a rotational variation of a subject of the template image;

converting each of the plurality of instances of the template image into a respective horizontal template image pixel 1D vector and a respective vertical template image pixel 1D vector to determine whether a match of the template image within the source image is indicated when undergoing a rotational change of a body of the template.

28. The method of claim 26, further comprising:

scaling the template image to provide a plurality of instances of the template image, wherein one or more of the plurality of instances of the template image provide a change in scaling of a subject of the template image;

converting each of the plurality of instances of the template image into a respective horizontal template image pixel 1D vector and a respective vertical template image pixel 1D vector to determine whether a match of the template image within the source image is indicated when undergoing a change in zoom of a body of the template.

29. The method of claim 26, further comprising:

iteratively performing a multi-directional image search within the search range using the horizontal template image pixel 1D vector and the vertical template image pixel 1D vector if a match of the template image within the source image is not indicated.

30. The method of claim 29, wherein iteratively performing a multi-directional image search comprises:

repositioning a search area within the search range of the source image;

performing the other iteration of converting the search area of the source image into a horizontal source image pixel 1D vector and a vertical source image pixel 1D vector, performing a correlation between the horizontal template image pixel 1D vector and the horizontal source image pixel 1D vector, performing a correlation between the vertical template image pixel 1D vector and the vertical source image pixel 1D vector, and analyzing the correlation.

31. The method of claim 30 wherein repositioning the search area within the search range of the source image is based on horizontal and vertical motion vectors derived from said analyzing the correlation between the horizontal template image pixel 1D vector and the horizontal source image pixel 1D vector, the correlation between the vertical template image pixel 1D vector and the vertical source image pixel 1D vector.

32. The method of claim 31, wherein repositioning a search area within the source image comprises:

moving the horizontal search portion based on a vertical motion vector;

the vertical search portion is moved based on a horizontal motion vector.

33. The method of claim 26, wherein the correlating comprises cross-correlating.

[ technical field ] A method for producing a semiconductor device

The present invention relates to image searching, and more particularly to iterative multi-directional image searching supporting large template matching.

[ background of the invention ]

Various techniques are used to identify portions of interest within an image. For example, in digital image processing, template matching techniques are known for finding a small portion of an image within a digital source image scene (referred to herein as a "source image") that corresponds to a digital reference image template (referred to herein as a "template image"). Such techniques may be implemented, for example, for defect inspection in various manufacturing systems, motion estimation in video compression systems, and so forth.

In operation, the template matching technique locates the position of the template image within the larger source image by matching the template image to a portion of the source image. As such, template matching techniques typically scan the source image pixel-by-pixel and compare with the template image to estimate the position of the template image within the source image. The Full Search (FS) template matching method exhaustively evaluates all possible candidate locations within one search area of the source image. It can be readily appreciated that an exhaustive search for FS template matches is very computationally intensive and consumes a significant amount of computing power. The normalized gray scale correlation can be used to compare gray scale values associated with their positions in the template image and the source image. The number of pixels can be reduced using a pyramidal approach. The hexagonal search method uses two different sizes of hexagonal search patterns, where a larger search pattern consisting of 7 check points is used for coarse search and a smaller reduced hexagonal pattern consisting of 4 check points is used for fine search. The hexagonal search method reduces the total number of checkpoints, thereby making the template search process faster. However, using normalized gray-scale correlation and image pyramids is computationally very expensive, especially when the template image is relatively large, because every pixel in the template image needs to be correlated with every pixel in the source image, whereas a hexagonal search may fall into a trap that is a local best match rather than a global best match.

A template image of 1/3 or greater having a width or height that is the search range (i.e., the area within the source image to be searched) is defined herein as a large template. Thus, the large template is relatively large compared to the search range. Although the computationally expensive nature of these techniques is exacerbated by the use of such large templates in conventional template matching techniques, the use of such large templates is still common. For example, large template matching may be part of a coarse and fine matching process, such as may be used for video compression, image registration, object localization, shape defect inspection, and stereo matching. In common cases, the rough location of the target within the image is known, but the target location needs to be accurately located within the image. In this case, the search range within the source image is relatively small, and the problem of large template search occurs using the template matching technique. Conventional template matching techniques are computationally intensive, especially for such large templates, and therefore searching for the best template match under these circumstances is often time consuming.

An example of a large template case is that the template image size may be 64 x 84 pixels, while the search range size is 177 x 236 pixels. In this example, the correlation (search) matrix size is 114 × 153 (i.e., (177-64+1) × (236-84+1), where the boundaries of the template image are constrained to be entirely within the source image), with the number of computations being (114 × 153) × (64 × 84) ═ 93,768,192 units. As can be seen from this example, template matching in the case of large templates is computationally very expensive.

In addition to being computationally expensive, existing template matching techniques are often problematic in terms of the accuracy of the results provided. For example, conventional template matching techniques typically provide a local optimum, rather than a global optimum, which reduces accuracy/precision. In addition, conventional template matching techniques often result in the loss of edge pixels in the template matching, which can affect the accuracy of the results.

[ summary of the invention ]

The invention relates to a system and a method for iterative multi-directional image search. The iterative multidirectional image search technique implemented according to the inventive concepts supports large template matching, enabling fast and accurate matching of template images (e.g., that may include one or more objects of interest) within source images even in the presence of large templates. Iterative multi-directional image search implemented according to the inventive concepts provides template matching in a computationally efficient multi-directional search process, even with large templates, which provides high accuracy matching results.

Embodiments of an iterative multi-directional image search use a multi-directional search pattern to determine one or more search regions within a source image. The search area of an embodiment provides a search area within a source image for searching a template image in multiple directions of a multi-directional search pattern. For example, embodiments may use one or more cross search patterns to identify a cross search region within a source image, e.g., may include a first direction search portion (e.g., a horizontal search portion) and a second direction search portion (e.g., a vertical search portion) that cross each other, wherein template images are sought in the search region along multiple directions of a multi-directional search pattern. The location of the search area within the source image may be iteratively updated, e.g., based on motion vectors derived from the search, until a template image match location is identified within the source image. Directionally-determined search regions and corresponding multi-directional-based template image searches implemented in accordance with embodiments of the present invention facilitate efficient searching within a source image, wherein iterative updating of the search regions enables efficient convergence of template image matching locations within the source image.

To facilitate efficient searching within a source image, embodiments of an iterative multi-directional image search convert a template image and a search area within a source image into a one-dimensional (1D) representation corresponding to each direction of the multi-directional image search. For example, summation, Fast Fourier Transform (FFT), or other transformation techniques may be used to convert the image data of the template image and the multi-directional image search region within the source image from a two-dimensional (2D) domain to a 1D representation to represent the image data one-dimensionally. For example, in a case where the intersection search area includes a horizontal search section and a vertical search section, the embodiment may convert the template image into a 1D representation in the horizontal direction and a 1D representation in the vertical direction. Similarly, the intersection search area of the source image may be converted into a 1D representation in the horizontal direction and a 1D representation in the vertical direction. Thus, a search for a template image in a search area can be performed along multiple (e.g., vertical and horizontal) directions of a multi-directional search pattern by associating the template image with an appropriate 1D representation of the search area within the source image.

[ description of the drawings ]

For a more complete understanding of this disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawing, in which:

FIG. 1A illustrates an example template image that may be used in an iterative multi-directional image search of an embodiment of the present invention;

FIG. 1B illustrates an example source image that may be used in an iterative multi-directional image search of an embodiment of the present invention;

FIG. 1C illustrates a determined position of an example template image within an example source image that may be provided by an iterative multi-directional image search of an embodiment of the present invention;

FIG. 2 illustrates a flow diagram of an iterative multi-directional image search operation of an embodiment of the present invention;

FIG. 3 illustrates a processor-based system configured for iterative multi-directional image search operations in accordance with an embodiment of the present invention;

FIG. 4A illustrates an example of the generation of a 1D representation of a template image, wherein the multi-directional search includes horizontal and vertical directions;

FIG. 4B illustrates one configuration example of an iterative multi-directional image search for rotational variation of a target of a template image within a source image, in accordance with an embodiment of the present invention;

FIG. 4C illustrates one configuration example of an iterative multi-directional image search for a zoom change of a target of a template image within a source image, in accordance with an embodiment of the present invention;

FIG. 5 illustrates an example of one search area provided by a search range for an iterative multi-directional image search in accordance with an embodiment of the present invention;

FIG. 6 illustrates an example of a search area determined in a source image of an iterative multidirectional image search in accordance with embodiments of the present invention;

FIGS. 7A and 7B illustrate one example of the generation of a 1D representation of a search area of a source image, where the multi-directional search includes horizontal and vertical directions;

FIG. 8 illustrates a flow diagram of an iterative multi-directional image search operation of an embodiment of the present invention.

FIG. 9 illustrates one example location information for a determined location of a template image within a source image that may be provided by an iterative multi-directional image search of an embodiment of the present invention;

FIG. 10 illustrates an exemplary multi-directional search pattern that may be used by an iterative multi-directional image search of an embodiment of the present invention.

The iterative multi-directional image search of an embodiment accommodates rotational and scaling variations of a subject (e.g., an object of interest) of the template image within the source image. For example, the template image may be rotated and/or scaled (e.g., 75%, 100%, 150%, etc.) in multiple directions (e.g., 45 °, 90 °, 135 °, 180 °, 225 °, 270 °, 315 °, etc.), wherein multiple rotated and/or scaled instances of the template are searched in a search area within the source image in the multiple directions. Embodiments of the interactive multi-directional image search may translate each such rotated instance and/or scaled instance of the template image to perform the multi-directional image search as described above.

The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the detailed description of the invention that follows may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. It should be appreciated by those skilled in the art that the conception and specific embodiment disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. The novel features which are believed to be characteristic of the invention, both as to its organization and method of operation, together with further objects and advantages will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present invention.