阅读说明：本技术 使用变换以验证计算机视觉质量 (Using transformations to verify computer vision quality ) 是由 J.布罗伊达 于 2019-11-08 设计创作，主要内容包括：公开了用于使用图像数据集变换以验证计算机视觉系统的质量的技术。在一些示例实施例中,计算机实施的方法包括：访问数据库以获得参考图像；使用多个变换基于参考图像生成多个新图像,多个变换中的每一个被配置为改变参考图像的对应视觉属性；将多个新图像馈送到图像分类器中,以生成用于多个新图像中的每一个的对应分类结果；基于生成的用于多个新图像分类结果,确定图像分类器不满足一个或多个准确度标准；以及基于确定图像分类器不满足一个或多个准确度标准,选择性地执行功能。(Techniques for using image dataset transforms to verify the quality of a computer vision system are disclosed. In some example embodiments, a computer-implemented method includes: accessing a database to obtain a reference image; generating a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image; feeding the plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images; determining, based on the generated results for the plurality of new image classifications, that the image classifier does not satisfy one or more accuracy criteria; and selectively performing a function based on determining that the image classifier does not satisfy the one or more accuracy criteria.)

1. A computer-implemented method, comprising:

accessing a database to obtain a reference image;

generating, by at least one hardware processor, a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image;

feeding, by at least one hardware processor, a plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images;

determining, by the at least one hardware processor, that the image classifier does not satisfy the one or more accuracy criteria based on the generated classification results for the plurality of new images; and

based on determining that the image classifier does not meet the one or more accuracy criteria, a function is selectively performed by the at least one hardware processor.

2. The computer-implemented method of claim 1, wherein the plurality of transforms comprises rotating the reference image.

3. The computer-implemented method of claim 1, wherein the plurality of transforms includes scaling a visual size of the reference image.

4. The computer-implemented method of claim 1, wherein the plurality of transforms includes changing a compression quality level of the reference image.

5. The computer-implemented method of claim 1, wherein generating a plurality of new images comprises:

receiving a corresponding transformation parameter for each of a plurality of transformation types; and

a plurality of transforms is generated based on the transform parameters, the plurality of transforms including different combinations of transform types and different combinations of transform values corresponding to the transform types for each of the different combinations of transform types.

6. The computer-implemented method of claim 5, wherein receiving corresponding transformation parameters for each of a plurality of transformation types comprises receiving user input via a user interface, the user input indicating transformation values for each of the plurality of transformation types via the user interface.

7. The computer-implemented method of claim 1, wherein the image classifier comprises a neural network model.

8. The computer-implemented method of claim 7, wherein the neural network model comprises a convolutional neural network model.

9. The computer-implemented method of claim 1, wherein the one or more accuracy criteria include a requirement that at least a particular portion of the generated classification result match a comparison value representing a correct prediction.

10. The computer-implemented method of claim 1, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

receiving corresponding base fact values for a plurality of new images; and

the generated classification result is compared with the corresponding base fact value.

11. The computer-implemented method of claim 1, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

feeding the reference image into an image classifier to generate a corresponding classification result for the reference image; and

the classification results of the plurality of new images are compared to the classification results of the reference image.

12. The computer-implemented method of claim 1, wherein the function comprises displaying a notification on a user interface of the computing device that the image classifier does not meet the one or more accuracy criteria.

13. The computer-implemented method of claim 1, wherein the function comprises training an image classifier using a plurality of new images as training data in one or more machine learning operations.

14. A system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that, when executed, cause at least one processor to perform operations comprising:

accessing a database to obtain a reference image;

generating a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image;

feeding the plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images;

determining, based on the generated classification results for the plurality of new images, that the image classifier does not satisfy one or more accuracy criteria; and

based on determining that the image classifier does not meet the one or more accuracy criteria, a function is selectively performed.

15. The system of claim 14, wherein the plurality of transforms include rotating the reference image, scaling a visual size of the reference image, and changing a compression quality level of the reference image.

16. The system of claim 14, wherein generating a plurality of new images comprises:

receiving a corresponding transformation parameter for each of a plurality of transformation types; and

a plurality of transforms is generated based on the transform parameters, the plurality of transforms including different combinations of transform types and different combinations of transform values corresponding to the transform types for each of the different combinations of transform types.

17. The system of claim 14, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

receiving corresponding base fact values for a plurality of new images; and

the generated classification result is compared with the corresponding base fact value.

18. The system of claim 14, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

feeding the reference image into an image classifier to generate a corresponding classification result for the reference image; and

the classification results of the plurality of new images are compared to the classification results of the reference image.

19. The system of claim 14, wherein the function comprises at least one of:

training an image classifier using the plurality of new images as training data in one or more machine learning operations; and

displaying, on a user interface of a computing device, a notification that an image classifier does not meet one or more accuracy criteria.

20. A non-transitory machine-readable storage medium tangibly embodying a set of instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

accessing a database to obtain a reference image;

generating a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image;

feeding the plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images;

determining, based on the generated classification results for the plurality of new images, that the image classifier does not satisfy one or more accuracy criteria; and

based on determining that the image classifier does not meet the one or more accuracy criteria, a function is selectively performed.

Technical Field

The present application relates generally to the technical field of neural networks and, in various embodiments, to systems and methods for using image dataset transforms to verify the quality of a computer vision system.

Background

Computer vision is a field of computer science, which is directed to enabling computers to view, recognize and process images in the same manner as human vision, and then provide appropriate output. However, before providing the computer vision system as a service, it is difficult to determine whether the computer vision system provides an acceptable level of quality (e.g., accuracy), either formally or experimentally, because the set of marking data that can be used for the quality verification process is limited and the diversity of possible real world input data is very high. Current marker data sets for computer vision systems fail to account for real-world distortions that occur in images that may affect the performance quality of the computer vision system. This situation, which fails to account for real-world distortions in images, is also found in training data used in the training of computer vision systems.

Disclosure of Invention

There is provided a computer-implemented method for verifying the quality of a computer vision system using an image dataset transform, the method comprising: accessing a database to obtain a reference image; generating, by at least one hardware processor, a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image; feeding, by at least one hardware processor, a plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images; determining, by the at least one hardware processor, that the image classifier does not satisfy the one or more accuracy criteria based on the generated classification results for the plurality of new images; and selectively performing, by the at least one hardware processor, a function based on the determination that the image classifier does not satisfy the one or more accuracy criteria.

There is provided a system for verifying the quality of a computer vision system using an image dataset transform, the system comprising: at least one processor; and a non-transitory computer-readable medium storing executable instructions that, when executed, cause at least one processor to perform operations comprising: accessing a database to obtain a reference image; generating a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image; feeding the plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images; determining, based on the generated classification results for the plurality of new images, that the image classifier does not satisfy one or more accuracy criteria; and selectively performing a function based on determining that the image classifier does not satisfy the one or more accuracy criteria.

A non-transitory machine-readable storage medium is provided that tangibly embodies a set of instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: accessing a database to obtain a reference image; generating a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image; feeding the plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images; determining, based on the generated classification results for the plurality of new images, that the image classifier does not satisfy one or more accuracy criteria; and selectively performing a function based on determining that the image classifier does not satisfy the one or more accuracy criteria.

Drawings

Some example embodiments of the disclosure are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

Fig. 1 is a network diagram illustrating a client-server system, according to some example embodiments.

Fig. 2 is a block diagram illustrating enterprise applications and services in an enterprise application platform, according to some example embodiments.

Fig. 3 is a block diagram illustrating a computer vision system, according to some example embodiments.

4A-4C illustrate applying different types of transforms to a reference image in generating a new image, according to some example embodiments.

Fig. 5 illustrates applying a plurality of transforms to a reference image in generating a new image, according to some example embodiments.

FIG. 6 illustrates a Graphical User Interface (GUI) configured to receive transformation parameters, according to some example embodiments.

Fig. 7 is a flow diagram illustrating a method of using an image dataset transform to verify the quality of a computer vision system, according to some example embodiments.

Fig. 8 is a block diagram of an example computer system on which methodologies described herein may be run, according to some example embodiments.

Detailed Description

Example methods and systems for using image dataset transforms to verify the quality of a computer vision system are disclosed. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the example embodiments. It will be apparent, however, to one skilled in the art that the present embodiments may be practiced without these specific details.

In some example embodiments, the computer vision system is configured to apply a transformation that simulates real-world distortion to a reference dataset of the image and produce a new image dataset that is more diverse and closer to the real-world data. The generation and application of this new image dataset enables the computer vision system to better assess (e.g., estimate) the quality level of the computer vision system in possible real world use cases, thereby improving the functionality of the computer vision system.

Embodiments of the features disclosed herein relate to non-generic, non-traditional, and non-conventional operations or combinations of operations. Some technical effects of the systems and methods of the present disclosure are to improve the quality (e.g., accuracy) of computer vision systems by applying one or more of the solutions disclosed herein. As a result, the functionality of the computer vision system is improved. Other technical effects will also be apparent from the present disclosure.

The methods or embodiments disclosed herein may be implemented as a computer system having one or more modules (e.g., hardware modules or software modules). These modules may be executed by one or more hardware processors of the computer system. In some example embodiments, the non-transitory machine-readable storage device may store a set of instructions that, when executed by the at least one processor, cause the at least one processor to perform the operations and method steps discussed within this disclosure.

The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and benefits of the subject matter described herein will be apparent from the description and drawings, and from the claims.

Fig. 1 is a network diagram illustrating a client-server system 100, according to some example embodiments. In the example form of the enterprise application platform 112, the platform (e.g., machines and software) provides server-side functionality to one or more clients via a network 114 (e.g., the internet). FIG. 1 shows, for example, a client machine 116 with a programmatic client 118 (e.g., a browser), a small device client machine 122 with a small device web client 120 (e.g., a browser without a scripting engine), and a client/server machine 117 with a programmatic client 119.

Turning specifically to the example enterprise application platform 112, a web server 124 and an Application Program Interface (API) server 125 may be coupled to an application server 126 and provide web and programming interfaces to the application server 126. The application server 126 may in turn be coupled to one or more database servers 128 that facilitate access to one or more databases 130. Cross-functional service 132 may include a relational database module to provide support services for access to database(s) 130, with database(s) 130 including user interface library 136. The Web server 124, API server 125, application server 126, and database server 128 may host a cross-function service 132. Application servers 126 may further host domain applications 134.

The cross-functionality services 132 provide services to users and processes that utilize the enterprise application platform 112. For example, cross-function services 132 may provide portal services (e.g., web services), database services, and connectivity to domain applications 134 for users operating client machine 116, client/server machine 117, and small device client machine 122. Further, the cross-function service 132 may provide an environment for delivering enhancements to existing applications and for integrating third party and legacy applications with existing cross-function services 132 and domain applications 134. Furthermore, while the system 100 shown in fig. 1 employs a client-server architecture, embodiments of the present disclosure are of course not limited to such an architecture, and may equally well find application in distributed or peer-to-peer architecture systems.

The enterprise application platform 112 can improve (e.g., increase) accessibility of data across different environments of the computer system architecture. For example, the enterprise application platform 112 may effectively and efficiently enable users to use real-world data created from the use of one or more end-users of deployed instances of software solutions in a development environment when testing the instances of software solutions in the development environment. The enterprise application platform 112 is described in more detail below in conjunction with fig. 2-8.

Fig. 2 is a block diagram illustrating enterprise applications and services in the enterprise application platform 112 according to an example embodiment. Enterprise application platform 112 may include cross-functionality services 132 and domain applications 134. Cross-functionality service 132 may include portal module 140, relational database module 142, connector and messaging module 144, API module 146, and development module 148.

Portal module 140 may enable a single point of access to other cross-functional services 132 and domain applications 134 for client machine 116, small device client machine 122, and client/server machine 117. The portal module 140 may be used to process, author, and maintain web pages that present content (e.g., user interface elements and navigation controls) to a user. In addition, the portal module 140 can enable user roles, i.e., constructs that associate roles with specialized environments used by users to run tasks, utilize services, and exchange information with other users within a defined scope. For example, a role may determine the content available to a user and the activities that the user may perform. The portal module 140 includes a generation module, a communication module, a reception module, and a regeneration module. In addition, the portal module 140 can conform to web service standards and/or utilize various internet technologies, includingJ2EE, SAP advanced Business application Programming languageAnd Web Dynpro, XML, JCA, JAAS, X.509, LDAP, WSDL, WSRR, SOAP, UDDI, and

relational database module 142 may provide support services for access to database(s) 130, including user interface library 136. Relational database module 142 may provide support for object relational mapping, database independence, and distributed computing. Relational database module 142 may be used to add, delete, update, and manage database elements. Further, the relational database module 142 may conform to database standards and/or utilize various database technologies, including SQL, SQLDBC, Oracle, MySQL, Unicode, JDBC, and the like.

Connector and messaging module 144 may enable different types of communications across messaging systems utilized across functional services 132 and domain applications 134 by providing a common messaging application processing interface. The connector and messaging module 144 may enable asynchronous communications on the enterprise application platform 112.

The API module 146 may enable development of service-based applications by exposing interfaces to existing and new applications as services. The repository may be included in the platform as a central location for finding available services when building an application.

Development module 148 may provide a development environment for adding, integrating, updating, and extending software components on enterprise application platform 112 without impacting existing cross-functionality services 132 and domain applications 134.

Turning to the domain application 134, the customer relationship management application 150 may enable access to multiple data sources and business processes and may facilitate the collection and storage of relevant personalization information from the multiple data sources and business processes. Business personnel responsible for developing buyers into long-term customers may utilize the customer relationship management application 150 to provide assistance to the buyers throughout the customer participation cycle.

The financial applications 152 and business processes may be utilized by enterprise personnel to track and control financial transactions within the enterprise application platform 112. The financial application 152 may facilitate the execution of operations, analysis, and collaboration tasks associated with financial management. In particular, the financial application 152 may enable performance of tasks related to financial responsibility, planning, forecasting, and managing financial costs.

The human resources application 154 may be used by enterprise personnel and business processes to manage, deploy, and track enterprise personnel. In particular, the human resources application 154 may enable analysis of human resources issues and facilitate human resources decisions based on real-time information.

The product lifecycle management application 156 can enable management of the product throughout its lifecycle. For example, the product lifecycle management application 156 can enable collaborative engineering, customized product development, project management, asset management, and quality management among business partners.

The supply chain management application 158 may enable monitoring of observed performance in the supply chain. The supply chain management application 158 may facilitate compliance with production schedules and on-time delivery of products and services.

Third party applications 160 and legacy applications 162 may be integrated with domain applications 134 and utilize cross-functionality services 132 on enterprise application platform 112.

Fig. 3 is a block diagram illustrating a computer vision system 300, according to some example embodiments. In some example embodiments, the computer vision system 300 is configured to apply a transformation that simulates real-world distortion to a reference dataset of an image and generate a new image dataset for verifying the quality of the computer vision system 300. In some embodiments, the computer vision system 300 includes any combination of one or more of the transformation module 310, the classification module 320, the verification module 330, the function module 340, and the one or more databases 350. The modules 310, 320, 330, and 340 and the database(s) 350 may reside on a computer system or other machine having memory and at least one processor (not shown). In some embodiments, modules 310, 320, 330, and 340 and database(s) 350 may be incorporated into application server(s) 126 in fig. 1. However, other configurations of the modules 310, 320, 330, and 340 and the database(s) 350 are contemplated to be within the scope of the present disclosure.

In some example embodiments, one or more of modules 310, 320, 330, and 340 are configured to provide various user interface functionality, such as generating a user interface, interactively presenting a user interface to a user, receiving information from a user (e.g., interactions with a user interface), and so forth. Presenting information to a user may include causing information to be presented to the user (e.g., communicating information to a device with instructions to present information to the user). Information can be presented using a variety of means, including visually displaying the information and outputting using other devices (e.g., audio, tactile, etc.). Similarly, information may be received via various means, including alphanumeric input or other device input (e.g., one or more touch screens, cameras, tactile sensors, light sensors, infrared sensors, biosensors, microphones, gyroscopes, accelerometers, other sensors, etc.). In some example embodiments, one or more of modules 310, 320, 330, and 340 are configured to receive user input. For example, one or more of modules 310, 320, 330, and 340 may present one or more GUI elements (e.g., drop down menus, selectable buttons, text fields) that a user may use to submit input. In some example embodiments, one or more of modules 310, 320, 330, and 340 are configured to perform various communication functions to facilitate the functionality described herein, such as by communicating with computing device 305 via network 114 using a wired or wireless connection.

In some example embodiments, the transformation module 310 accesses the database 350 to obtain the reference image. For example, multiple reference images may be stored in database(s) 350, and transformation module 310 may access and retrieve the reference images stored in database(s) 350. Each reference image may include one or more objects captured in the reference image.

In some example embodiments, transformation module 310 is configured to generate a plurality of new images based on the reference image using a plurality of transformations. Each of the plurality of transforms may be configured to change a corresponding visual property of the reference image in order to simulate real-world distortion of the image. Fig. 4A-4C illustrate the application of different types of transforms 410A, 410B, and 410C to the reference image 400 in generating new images 420A, 420B, and 420C, respectively, according to some example embodiments. Although the examples shown in fig. 4A-4C illustrate an image 400 that includes a single object 405, it is contemplated that the image 400 may include multiple objects.

In fig. 4A, the transform 410A applied to the reference image 400 includes rotating the reference image 400. By applying transform 410A, transform module 310 generates a new image 420A in which object 405 has been rotated from its original orientation in reference image 400. Although FIG. 4A shows rotation about a single particular axis, it is contemplated that transformation 410A may also include rotation about a different axis or multiple axes.

In fig. 4B, the transformation 410B applied to the reference image 400 includes scaling (e.g., resizing) the visual size of the reference image 400. By applying transform 410B, transform module 310 generates a new image 420B in which object 405 has been resized from its original visual size in reference image 400. Although fig. 4B illustrates transformation 410B including resizing of reference image 400 with a visual effect of zooming in on object 405, it is contemplated that transformation 410B may also include resizing of reference image 400 with a visual effect of zooming out on object 405.

In fig. 4C, the transform 410C applied to the reference picture 400 includes changing the compression quality level of the reference picture 400. Image compression involves reducing the size in bytes of the graphics file, which will reduce the resolution of the graphics file. By applying transform 410C, transform module 310 generates a new image 420C in which the compression quality level of object 405 has changed from its original compression quality level in reference image 400. Fig. 4C shows that the transformation 410C includes a reduction in the compression quality level of the reference image 400, which results in a reduction in resolution, making the object 405 appear blurry or pixilated in the new image 420C.

4A-4C illustrate that single transforms 410A, 410B, and 410C are each applied to reference image 400, in some example embodiments, transform module 310 applies multiple transforms to reference image 400 in generating new images. Fig. 5 illustrates the application of a plurality of transforms 510 to a reference image 400 in generating a new image 520, according to some example embodiments. In the example shown in fig. 5, the plurality of transforms 510 includes rotating the reference image 400, scaling (e.g., resizing) the visual size of the reference image 400, and changing the compression quality level of the reference image 400. It is contemplated that in generating new image 520, transform module 310 may apply other types of transforms and other combinations of transform types to reference image 400.

In some example embodiments, the transformation module 310 is configured to generate a plurality of transformations to be applied to the reference image based on the transformation parameters. For example, the transformation module 310 may receive corresponding transformation parameters for each of a plurality of transformation types (e.g., rotation, scaling, reduction in compression quality levels), and then generate a plurality of transformations based on the received transformation parameters. In some example embodiments, each transformation parameter includes or otherwise indicates a metric or some other detail of how the corresponding transformation is applied to the reference image in generating the new image. For example, one transformation parameter may indicate a rotation of 10 degrees about the x-axis of the reference image, while another transformation parameter may indicate a magnification of 10% of the visual size of the reference image, and yet another transformation parameter may indicate a reduction of 15% of the compression quality level of the reference image. In some example embodiments, each transformation parameter includes or otherwise indicates a range of metrics for how the corresponding transformation is applied to the reference image when generating the new image. For example, one transformation parameter may indicate a rotation range of ± 15 degrees about the x-axis of the reference image, while another transformation parameter may indicate a range of 50% to 150% of the visual size of the reference image in the scaling of the visual size of the reference image, and yet another transformation parameter may indicate a range of reduction in the compression quality level of the reference image of 0% to 40%.

In some example embodiments, the transformation module 310 is configured to receive user input indicating transformation parameters via a user interface. Fig. 6 illustrates a GUI 600 configured to receive transformation parameters, according to some example embodiments. In fig. 6, GUI 600 displays a plurality of user interface elements 610, each user interface element 610 configured to enable a user to input a configuration of a corresponding transformation parameter. For example, in fig. 6, user interface elements 610A, 610B, 610C, and 610D each include a control element, such as a slider, that a user can manipulate to configure a corresponding transformation parameter (e.g., rotation about an x grid, rotation about a y grid, rotation about a z grid, translation along an x axis). The plurality of user interface elements 610 may correspond to a particular category, which may be selected by a user from a list of transformation categories or types 620, such that selection of a particular transformation category or type from the list 620 causes display of the corresponding user interface element 610 for transformation parameters of the selected transformation category or type. In some example embodiments, in response to a user configuring the transformation parameters, corresponding code for the transformation parameters may be generated and displayed in portion 640 of GUI 600. The code may be for implementation of a transformation of the configuration of the reference image in generating the new image.

In some example embodiments, the plurality of transforms used by transform module 310 to generate one or more new images for the reference image includes different combinations of transform types and different combinations of transform values corresponding to the transform types for each of the different combinations of transform types. For example, the transformation module 310 may employ three types of transformations, such as rotation, scaling, and reduction of compression quality levels, and generate different sets of one or more of the three types of transformations. In some example embodiments, the transformation module 310 generates all possible sets of one or more of the three types of transformations:

1) rotate

2) Rotation + zoom

3) Rotation + scaling + reduction in compression quality level

4) Reduction in rotation + compression quality level

5) Zoom

6) Scaling + reduction in compression quality level

7) Reduction in compression quality level

For each of these sets, the transformation module 310 may generate a different transformation value based on the corresponding transformation parameter. The different transformation values may be based on the range indicated by the transformation parameters. In one example, for the above second set consisting of rotation and scaling, the transformation module 310 may generate different combinations of transformation values for rotation and scaling, such as:

1) rotating: -15%; zooming: reduced by 50%

2) Rotating: -14%; zooming: reduced by 50%

3) Rotating: -13%; zooming: reduced by 50%

.

.

.

29) Rotating: + 14%: zooming: reduced by 50%

30) Rotating: + 15%; zooming: reduced by 50%

31) Rotating: -15%; zooming: reduce by 49 percent

32) Rotating: -15%; zooming: reduce by 48 percent

.

.

.

In some example embodiments, the transformation module 310 is configured to select a set of transformations by selecting points in a multi-dimensional transformation space created by cartesian products of transformation parameters (e.g., ranges of values), and then generate a new image test set by applying each transformation from the selected set of transformations to each reference image of a plurality of reference images. In some example embodiments, the transformation module 310 is configured to divide the transformation types into unions (unions), and then construct a transformation space for each union. The transform module 310 may then select points from each transform space, each point representing a certain transform, resulting in a different set of transforms, where each set corresponds to a union of its transforms.

In some example embodiments, the classification module 320 is configured to feed a plurality of new images into the image classifier to generate a corresponding classification result for each of the plurality of new images. Each classification result may include a corresponding prediction class for one or more objects in a corresponding new image. For example, the new image includes a picture of two dogs, and the image classifier may process the new image and generate a classification result indicating that the new image includes two dogs, an indication of a particular breed of dog, or some other type of indication corresponding to a classification of two dogs. In some example embodiments, the image classifier includes a neural network model, such as a convolutional neural network. However, other types of image classifiers are also within the scope of the present disclosure.

In some example embodiments, the validation module 330 is configured to determine that the image classifier does not satisfy one or more accuracy criteria based on the classification results generated for the plurality of new images. The accuracy criterion may comprise that at least some part of the generated classification result represents a requirement for an accurate prediction of the corresponding new image. For example, the accuracy criterion may include a requirement that at least a certain portion (e.g., at least 75%) of the generated classification results match a comparison result or value representing a correct prediction, such as a ground truth (ground true) value or a classification result of a reference image. In some example embodiments, the validation module 330 is configured to determine that the image classifier does not satisfy the one or more accuracy criteria by receiving a corresponding base fact value for a plurality of new images (e.g., an actual true classification of the new images), and then comparing the generated classification result to the corresponding base fact value to determine whether the generated classification result represents an accurate prediction of the corresponding new images. In some example embodiments, the validation module 330 is configured to determine that the image classifier does not meet the one or more accuracy criteria by feeding the reference image into the image classifier to generate corresponding classification results for the reference image, and then comparing the classification results for the plurality of new images to the classification results for the reference image to determine whether the generated classification results represent an accurate prediction of the corresponding new images.

In some example embodiments, the determination of whether the image classifier satisfies one or more accuracy criteria includes calculating one or more accuracy values of the classification results generated for the plurality of new images, and the accuracy criteria includes one or more requirements for the one or more accuracy values (e.g., a minimum threshold to satisfy). Such accuracy values may include, but are not limited to, any combination of one or more of a mean recall (recall) value, an accuracy value, a mean accuracy value, and an accuracy value. These accuracy values may be based on one or more of the number of True (TPs), the number of True Negatives (TNs), the number of False Positives (FPs), and the number of False Negatives (FNs) found in generating classification results for a plurality of new images. The terms true, true negative, false positive, and false negative compare the generated classification results of the image classifier under test with a trusted external judgment (e.g., the classification results of the underlying fact value or the reference image). The terms positive and negative refer to a prediction by the image classifier (e.g., classifying an object as positive for a particular thing, classifying an object as negative for not a particular thing), and the terms true and false refer to whether the prediction corresponds to an external judgment (e.g., whether the classification is correct or true, and whether the classification is incorrect or false).

The average recall value or true rate is a measure of how well the image classifier finds all of the new images to be actually positive, and can be calculated using the following formula:

the accuracy value or positive prediction value is a measure of how accurate the prediction is (e.g., a percentage of correct predictions), and may be calculated using the following formula:

the average precision value is the average of the maximum precision at different recall values. The accuracy value is a measure of the degree to which the image classifier correctly identifies or excludes the condition, and may be calculated based on the proportion of true results (both true and true negative) among the total number of cases examined:

in some example embodiments, the determination of whether the image classifier satisfies one or more accuracy criteria is implemented using the following pseudo-code:

1. the sum of all false positive detections (FP) at all images.

2. The sum of all true detections (TP) at all images.

3. The sum of all false negative detections (FN) at all images.

4. The sum of all true negative detections (TN) at all images.

5. Recalls were calculated based on the results of steps 1-4 (FP, TP, FN and TN).

6. The accuracy is calculated based on the results of steps 1-4 (FP, TP, FN, and TN).

7. It is checked whether the recall is above a minimum threshold.

8. It is checked whether the accuracy is above a minimum threshold.

9. If steps 7 and 8 return true (e.g., recall and precision meet respective thresholds), then it is determined that the image classifier meets one or more accuracy criteria; if steps 7 and 8 return false (e.g., a recall or a precision that does not meet the corresponding threshold), then it is determined that the image classifier does not meet one or more accuracy criteria.

In some example embodiments, the function module 340 is configured to perform a function in response to or otherwise based on a determination that the image classifier does not satisfy one or more accuracy criteria. In some example embodiments, the functions include displaying a notification on a user interface of the computing device that the image classifier does not meet the one or more accuracy criteria. In some example embodiments, the functions include training the image classifier using the plurality of new images as training data in one or more machine learning operations. Other types of functions are also within the scope of the present disclosure.

Fig. 7 is a flow diagram illustrating a method 700 of verifying the quality of a computer vision system using image dataset transformations, according to some example embodiments. Method 700 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In an example embodiment, method 700 is performed by the computer vision system of FIG. 3 or any combination of one or more of its modules 310, 320, 330, and 340 as described above.

At operation 710, the computer vision system 300 accesses a database to obtain a reference image. At operation 720, the computer vision system 300 generates a plurality of new images based on the reference image using a plurality of transforms. Each of the plurality of transforms may be configured to change a corresponding visual property of the reference image. In some example embodiments, the plurality of transforms includes any combination of one or more of rotating the reference image, scaling the visual size of the reference image, and changing the level of compression quality of the reference image. The generating of the plurality of new images may include: corresponding transformation parameters for each of a plurality of transformation types are received, and a plurality of transformations are generated based on the transformation parameters. In some example embodiments, the plurality of transforms includes different combinations of transform types and different combinations of transform values corresponding to the transform types for each of the different combinations of transform types. The receiving of the corresponding transformation parameters for each of the plurality of transformation types may include receiving user input via a user interface. In some example embodiments, the user input indicates, via the user interface, a transformation value for each of a plurality of transformation types.

At operation 730, the computer vision system 300 feeds the plurality of new images into the image classifier to generate a corresponding classification result for each of the plurality of new images. In some example embodiments, the image classifier includes a neural network model. For example, the neural network model may comprise a convolutional neural network model. However, other types of neural network models and other types of image classifiers are also within the scope of the present disclosure.

At operation 740, the computer vision system 300 determines, based on the generated classification results for the plurality of new images, that the image classifier does not satisfy one or more accuracy criteria. In some example embodiments, determining that the image classifier does not satisfy the one or more accuracy criteria includes receiving corresponding base fact values for the plurality of new images and comparing the generated classification results to the corresponding base fact values. In some example embodiments, determining that the image classifier does not satisfy the one or more accuracy criteria comprises feeding the reference image into the image classifier to generate corresponding classification results for the reference image and comparing the classification results for the plurality of new images to the classification results for the reference image.

At operation 750, the computer vision system 300 selectively performs a function in response to or otherwise based on a determination that the image classifier does not meet one or more accuracy criteria. In some example embodiments, the functions include displaying a notification on a user interface of the computing device that the image classifier does not meet the one or more accuracy criteria. In some example embodiments, the functions include training the image classifier using the plurality of new images as training data in one or more machine learning operations. Other types of functions are also within the scope of the present disclosure.

It is contemplated that any of the other features described within this disclosure may be incorporated into method 700.

Certain embodiments are described herein as comprising logic or multiple components, modules, or mechanisms. The modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In an example embodiment, one or more computer systems (e.g., a stand-alone client or server computer system) or one or more hardware modules (e.g., processors or groups of processors) of a computer system may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

In various embodiments, the hardware modules may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC)) to perform certain operations. A hardware module may also include programmable logic or circuitry (e.g., contained within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It should be understood that the decision to implement a hardware module mechanically in a dedicated and permanently configured circuit or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations.

Thus, the term "hardware module" should be understood to encompass a tangible entity, either a physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) entity to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each hardware module need not be configured or instantiated at any one time. For example, in the case where the hardware modules include a general-purpose processor configured using software, the general-purpose processor may be configured as corresponding different hardware modules at different times. The software may thus configure the processor, e.g., to constitute a particular hardware module at one time and to constitute a different hardware module at a different time.

A hardware module may provide information to and receive information from other hardware modules. Thus, the described hardware modules may be considered to be communicatively coupled. In the case where a plurality of such hardware modules are present at the same time, communication may be achieved by signal transmission (e.g., over appropriate circuits and buses connecting the hardware modules). In embodiments in which multiple hardware modules are configured or instantiated at different times, communication between the hardware modules may be accomplished, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware modules. For example, one hardware module may perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Further hardware modules may then access the memory device to retrieve and process the stored output. The hardware modules may also initiate communication with input or output devices and may operate on resources (e.g., collection of information).

Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., via software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. In some example embodiments, the modules referred to herein may comprise processor-implemented modules.

Similarly, the methods described herein may be implemented, at least in part, by processing. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of some of the operations may be distributed among one or more processors, which may exist not only within a single machine, but also deployed across multiple machines. In some example embodiments, one or more processors may be located in a single location (e.g., within a home environment, an office environment, or as a server farm), while in other embodiments, processors may be distributed across multiple locations.

The one or more processors may also operate to support the performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as an example of a machine including a processor), which are accessible via a network (e.g., network 114 of fig. 1) and via one or more appropriate interfaces (e.g., APIs).

The illustrative embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Example embodiments may be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be run on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

In an example embodiment, the operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations may also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry, e.g., an FPGA or an ASIC.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments where a programmable computing system is deployed, it should be understood that both hardware and software architectures are contemplated. In particular, it should be understood that the choice of whether to implement certain functions in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or in a combination of permanently and temporarily configured hardware may be a design choice. In various example embodiments, hardware (e.g., machine) and software architectures that may be deployed are listed below.

Fig. 8 is a block diagram of a machine in the example form of a computer system 800 in which instructions 824 for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a Personal Computer (PC), a tablet PC, a Set-Top Box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Example computer system 800 includes a processor 802 (e.g., a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or both), a main memory 804 and a static memory 806, which communicate with each other via a bus 808. The computer system 800 may also include a graphics or video Display unit 810 (e.g., a Liquid Crystal Display (LCD) or Cathode Ray Tube (CRT)). Computer system 800 also includes an alphanumeric input device 812 (e.g., a keyboard), a User Interface (UI) navigation (or cursor control) device 814 (e.g., a mouse), a storage unit (e.g., a disk drive unit) 816, an audio or signal generation device 818 (e.g., a speaker), and a network interface device 820.

The storage unit 816 includes a machine-readable medium 822 on which is stored one or more sets (e.g., software) of data structures and instructions 824 embodying or used by any one or more of the methodologies or functions described herein. The instructions 824 may also reside, completely or at least partially, within the main memory 804 and/or within the processor 802 during execution thereof by the computer system 800, the main memory 804 and the processor 202 also constituting machine-readable media. The instructions 824 may also reside, completely or at least partially, within the static memory 806.

While the machine-readable medium 822 is shown in an example embodiment to be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 824 or data structures. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present embodiments, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of the machine-readable medium include non-volatile memories including, for example, semiconductor Memory devices (e.g., Erasable Programmable Read-Only memories (EPROMs), Electrically Erasable Programmable Read-Only memories (EEPROMs), and flash Memory devices); magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and Compact disk-Read-Only-Memory (CD-ROM) and Digital Versatile disk (or Digital Video disk) Read-Only-Memory (DVD-ROM) disks.

The instructions 824 may further be transmitted or received over a communication network 826 using a transmission medium. The instructions 824 may be sent using the network interface device 820 and any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a LAN, a WAN, the internet, a mobile telephone network, a POTS network, and a wireless data network (e.g., WiFi and WiMax networks). The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

Each of the features and teachings disclosed herein may be utilized separately or in conjunction with other features and teachings to provide systems and methods for blind spot implementation in neural networks. Representative examples utilizing many of these additional features and teachings, both separately and in combination, are described in further detail with reference to the accompanying drawings. This detailed description is merely intended to teach a person of skill in the art further details for practicing certain aspects of the present teachings and is not intended to limit the scope of the claims. Thus, combinations of features disclosed in the above detailed description may not be necessary to practice the present teachings in the broadest sense, and are instead taught merely to describe certain representative examples of the present teachings.

Some portions of the detailed descriptions herein are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing" or "computing" or "calculating" or "determining" or "displaying" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other physical quantities similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

The present disclosure also relates to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, Read-Only memories (ROMs), Random Access Memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.

The example methods or algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems, computer servers, or personal computers may be used with programs in accordance with the teachings herein, or it may be convenient to construct more specialized apparatus to perform the method steps disclosed herein. The structure for a variety of these systems will appear from the description herein. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure as described herein.

Furthermore, various features of the representative examples and the dependent claims may be combined in ways that are not specifically and explicitly enumerated in order to provide additional useful embodiments of the present teachings. It is also expressly noted that all value ranges or indications of groups of entities disclose every possible intermediate value or intermediate entity for the purpose of original disclosure, as well as for the purpose of limiting the claimed subject matter. It is also expressly noted that the sizes and shapes of the components shown in the figures are designed to facilitate an understanding of how the present teachings are implemented, and are not intended to be limiting of the sizes and shapes shown in the examples.

Although embodiments have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The described embodiments are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This detailed description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred to herein, individually and/or collectively, by the term "invention" merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Examples of the invention

1. A computer-implemented method, comprising:

accessing a database to obtain a reference image;

generating, by at least one hardware processor, a plurality of new images based on the reference image using a plurality of transforms, each of the plurality of transforms configured to change a corresponding visual property of the reference image;

feeding, by at least one hardware processor, a plurality of new images into an image classifier to generate a corresponding classification result for each of the plurality of new images;

determining, by the at least one hardware processor, that the image classifier does not satisfy the one or more accuracy criteria based on the generated classification results for the plurality of new images; and

in response to or otherwise based on determining that the image classifier does not satisfy the one or more accuracy criteria, a function is selectively performed by the at least one hardware processor.

2. The computer-implemented method of example 1, wherein the plurality of transforms includes rotating the reference image.

3. The computer-implemented method of example 1 or example 2, wherein the plurality of transforms includes scaling a visual size of the reference image.

4. The computer-implemented method of any of examples 1 to 3, wherein the plurality of transforms includes changing a compression quality level of the reference image.

5. The computer-implemented method of any of examples 1 to 4, wherein generating a plurality of new images comprises:

receiving a corresponding transformation parameter for each of a plurality of transformation types; and

a plurality of transforms is generated based on the transform parameters, the plurality of transforms including different combinations of transform types and different combinations of transform values corresponding to the transform types for each of the different combinations of transform types.

6. The computer-implemented method of example 5, wherein receiving corresponding transformation parameters for each of a plurality of transformation types comprises receiving user input via a user interface, the user input indicating transformation values for each of the plurality of transformation types via the user interface.

7. The computer-implemented method of any of examples 1 to 6, wherein the image classifier comprises a neural network model.

8. The computer-implemented method of example 7, wherein the neural network model comprises a convolutional neural network model.

9. The computer-implemented method of any of examples 1 to 8, wherein the one or more accuracy criteria include a requirement that at least a particular portion of the generated classification results match a comparison value representing a correct prediction.

10. The computer-implemented method of any of examples 1 to 9, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

receiving corresponding base fact values for a plurality of new images; and

the generated classification result is compared with the corresponding base fact value.

11. The computer-implemented method of any of examples 1 to 10, wherein determining that the image classifier does not satisfy the one or more accuracy criteria comprises:

feeding the reference image into an image classifier to generate a corresponding classification result for the reference image; and

the classification results of the plurality of new images are compared to the classification results of the reference image.

12. The computer-implemented method of any of examples 1 to 11, wherein the function comprises displaying a notification on a user interface of the computing device that the image classifier does not meet the one or more accuracy criteria.

13. A computer-implemented method as any one of examples 1-12 recites, wherein the functions include training an image classifier using a plurality of new images as training data in one or more machine learning operations.

14. A system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that, when executed, cause at least one processor to perform the method of any one of examples 1-13.

15. A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any of examples 1-13.

16. A machine-readable medium carrying a set of instructions which, when executed by at least one processor, causes the at least one processor to carry out the method of any one of examples 1 to 13.

The Abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.