阅读说明：本技术 识别信息赋予装置、识别信息赋予方法以及程序 (Identification information providing device, identification information providing method, and program ) 是由 伊藤智祥 秦秀彦 于 2019-12-25 设计创作，主要内容包括：削减制作学习模型的学习数据的劳力和时间。识别信息赋予装置(1)具备：获取部(11),获取多个图像数据；赋予部(12),使用学习完毕的学习模型,对从多个图像数据选择的图像数据赋予识别信息；和更新部(13),使用被赋予了识别信息的图像数据来更新学习模型,赋予部使用被更新的学习模型,对获取部获取到的剩余的图像数据赋予识别信息。(The labor and time required for creating learning data of a learning model are reduced. The identification information providing device (1) is provided with: an acquisition unit (11) that acquires a plurality of image data; an assigning unit (12) that assigns identification information to image data selected from the plurality of image data using the learned learning model; and an updating unit (13) that updates the learning model using the image data to which the identification information has been given, wherein the adding unit adds the identification information to the remaining image data acquired by the acquiring unit using the updated learning model.)

1. An identification information providing device is provided with:

an acquisition unit that acquires a plurality of image data;

an assigning unit that assigns identification information to image data selected from the plurality of image data using a learned learning model; and

an updating unit that updates the learning model using the image data to which the identification information is given,

the adding unit adds identification information to the remaining image data acquired by the acquiring unit, using the updated learning model.

2. The identification information imparting device according to claim 1, wherein,

the identification information providing device further includes: a correction unit that displays the image data and the identification information added to the image data by the addition unit on a display unit, receives a request for correcting the identification information, and corrects the identification information in accordance with the request,

the updating unit updates the learning model using the image data in which the identification information is corrected by the correcting unit.

3. The identification information imparting device according to claim 2, wherein,

the correction unit displays the image data and the identification information on a display unit in a manner such that the identification information added by the addition unit and the identification information corrected by the correction unit are distinguished from each other.

4. The identification information imparting device according to claim 3, wherein,

the correction unit may correct the identification information a plurality of times, and when the image data and the identification information are displayed on the display unit, the correction unit may display the identification information corrected by the correction unit in the past and the identification information newly corrected this time in a separated manner.

5. The identification information imparting device according to any one of claims 1 to 4, wherein,

the adding unit may use a plurality of the learning models including feature information of an image, and may use a learning model including feature information of the image data acquired by the acquiring unit and feature information of a predetermined range.

6. The identification information imparting device according to claim 5, wherein,

the feature information contains a photographing condition of the image data,

the adding unit uses a learning model associated with the same imaging condition as that of the image data.

7. The identification information imparting device according to claim 5 or 6, wherein,

the feature information includes a color histogram of the image data,

the adding unit uses a learning model associated with a color histogram in which a difference from the color histogram of the image data is within a predetermined range.

8. The identification information imparting device according to any one of claims 2 to 4, wherein,

the identification information providing device further includes: a selection unit that detects a moving subject from the image data selected from the plurality of image data, selects a region of the subject,

the correction unit displays the image data and the area selected by the selection unit in the image data as identification information, receives a request for correcting the identification information, and corrects the identification information in accordance with the request.

9. The identification information imparting device according to any one of claims 1 to 8, wherein,

the plurality of image data are image data having continuity.

10. An identification information providing method, comprising:

a step of acquiring a plurality of image data;

assigning identification information to image data selected from the plurality of image data using a learned learning model;

updating the learning model using the image data to which the identification information is given; and

and assigning identification information to the remaining acquired image data using the updated learning model.

11. A program for causing a computer to execute the method of claim 10.

Technical Field

The present disclosure relates to an identification information providing device, an identification information providing method, and a program for providing identification information to data used for machine learning.

Background

In recent years, machine learning has been used in various fields. In machine learning, the number of learning data is important, and learning with a large amount of learning data can provide a result with high accuracy. In this case, information related to data needs to be given in advance. The operation is called Annotation (Annotation), and for example, when a person appears in the photograph data, information such as position information of a region in which the person exists in the photograph data and information of the type of "person" is given.

Since the amount of learning data is enormous, the annotation operation is manually performed, which requires a lot of labor and time. Patent document 1 describes a technique for reducing manual operations. Patent document 1 describes the following technique: first, reference data is generated by manual operation, and learning data is generated using the reference data.

Prior art documents

Patent document

Patent document 1: japanese patent laid-open publication No. 2018-200531

Disclosure of Invention

Problems to be solved by the invention

Provided are an identification information providing device, an identification information providing method, and a program, which enable learning data to be easily created.

Means for solving the problem

The disclosed identification information providing device is provided with: an acquisition unit that acquires a plurality of image data; an assigning unit that assigns identification information to image data selected from the plurality of image data using the learned learning model; and an updating unit that updates the learning model using the image data to which the identification information is given, and the giving unit gives the identification information to the remaining image data acquired by the acquiring unit using the updated learning model.

These general and specific approaches may also be implemented by systems, methods, and computer programs and combinations of these.

Effect of invention

With the identification information providing device, the identification information providing method, and the program according to the present disclosure, when learning data for machine learning is generated, it is possible to automatically and easily provide identification information and generate the learning data.

Drawings

Fig. 1 is a block diagram illustrating a configuration of a travel route analysis system including an identification information providing device according to the present disclosure.

Fig. 2 is a conceptual diagram illustrating a configuration of image data used by the identification information providing apparatus according to the present disclosure.

Fig. 3 is a block diagram illustrating a configuration of an identification information providing apparatus according to the present disclosure.

Fig. 4 is a conceptual diagram illustrating the addition of identification information in the identification information adding device of fig. 3.

Fig. 5 is an example of image data to be subjected to the addition of identification information by the identification information adding device of fig. 3.

Fig. 6 is an example of identification information given to the image data of fig. 5.

Fig. 7 is another example of identification information given to the image data of fig. 5.

Fig. 8 is a conceptual diagram illustrating correction of the identification information corresponding to fig. 4.

Fig. 9 is a flowchart for explaining the identification information providing method according to the present disclosure.

Fig. 10 is a block diagram illustrating a configuration of an identification information providing apparatus according to a modification.

Fig. 11 is a conceptual diagram illustrating the addition and correction of the identification information in the identification information adding device of fig. 10.

Fig. 12 is a conceptual diagram illustrating a modification of the addition and correction of the identification information in the identification information addition device of fig. 3.

Detailed Description

[ knowledge as a basis for the present disclosure ]

In recent years, the use of machine learning has been widespread in a wide range of fields. Further, with the advancement of computing devices such as high-speed gpu (graphics Processing unit) servers, the Processing speed of image data has been increasing. Thus, by analyzing information used for moving image data obtained by machine learning, it is possible to analyze in detail only contents that are difficult to analyze by humans. For example, even when analyzing motions of people and objects in various places such as factories, warehouses, shops, and offices, high-precision analysis that is difficult to perform only by human analysis can be performed by using a large amount of data by machine learning.

However, in the case of machine learning, the results of generation of learning data and construction of a learning model are also significant. Therefore, when learning data is generated, it is important to provide identification information, i.e., to annotate the learning data. However, the provision of the identification information still requires a lot of labor and time.

The present disclosure provides an identification information providing device, an identification information providing method, and a program, which automatically and easily provide identification information for data required for generating learning data for machine learning. This enables generation of learning data that can provide highly accurate results.

[ embodiment ]

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings as appropriate. However, in the detailed description, unnecessary portions of the description relating to the related art and the substantially same configuration may be omitted. This is for simplicity of illustration. Furthermore, the following description and drawings are disclosed for a sufficient understanding of the present disclosure by those skilled in the art and are not intended to limit the subject matter of the claims.

The identification information providing device, the identification information providing method, and the program according to the present disclosure automatically provide identification information when generating learning data learned by a machine. Hereinafter, an example in which the identification information providing device expands the learning data used for the travel route analysis will be described. In the following description, the object of the identification information provision by the identification information provision device is image data including a person and a shopping cart, and description will be given of an example in which identification information about the person and the shopping cart is provided.

In the present disclosure, the "identification information" refers to information such as a tag and metadata that is given to image data that is data for learning in machine learning. Note that "assignment of identification information" is assignment of a tag or metadata to image data, and is synonymous with "annotation".

In the present disclosure, the term "movement path" refers to a path or a trajectory along which a person or an object moves, and the term "movement path analysis" refers to recording the movement path of the person or the object, and analyzing and outputting the recorded movement path as statistical data.

< moving route analyzing System >

As shown in fig. 1, the identification information providing device 1 according to the present disclosure is used in, for example, a travel route analysis system 100 that analyzes the movement of a person or the like. The travel route analysis system 100 includes, in addition to the identification information providing device 1, an imaging device 2, a sensor 3, a posture detection device 4, an operation estimation device 5, and a data integration/travel route generation device 6.

The photographing device 2 is a camera that photographs a space that is a target of movement route analysis. The imaging device 2 is not necessarily a device for capturing a moving image, but since the movement path analysis system 100 analyzes the movement of a person or the like, it is necessary to capture images of a plurality of continuous frames. In fig. 1, only one imaging device 2 is shown, but since it is preferable to image the entire space as a target, the travel route analysis system 100 may include a plurality of imaging devices 2. For example, when the space S shown in fig. 2 is photographed, 4 photographing devices 2 for photographing the 1 st area a1, the 2 nd area a2, the 3 rd area A3, and the 4 th area a4 may be provided. In addition, the travel route analysis system 100 can combine the image data simultaneously captured by the plurality of imaging devices 2 into one image data and use the image data. In the following description, the image data captured at one timing is described as one piece of image data or as image data already combined, and the description of the combining process is omitted.

The sensor 3 is a human detection sensor for detecting the presence of a human being by infrared rays or ultrasonic waves, for example. The sensor 3 may be a sound sensor that inputs a sound to detect a motion of a person or the like. The sensor 3 may receive a radio wave transmitted from a transmitter provided in a person, a shopping cart, an object, or the like. In fig. 1, only one sensor 3 is illustrated, but the travel route analysis system 100 may include a plurality of sensors 3, or may include a plurality of types of sensors. In the moving route analysis, by using the detection result by the sensor 3, the accuracy of person detection can be improved compared to the case where a person is detected from only image data. For example, the present invention can be used for determining whether or not an object that uses the position information obtained from the sensor 3 together with the image data and provides the image data with identification information is a shopping cart. Specifically, this is because, when a transmitter is provided as the sensor 3 for a shopping cart, the radio wave of the sensor 3 can accurately map where the shopping cart is included in the image data.

The identification information providing device 1 provides identification information to image data captured by the imaging device 2. The specific configuration of the identification information providing apparatus 1, the processing in the identification information providing apparatus 1, and the like will be described later with reference to fig. 3 to 9.

The posture detecting device 4 detects the posture of a person existing in the space as a target of the movement route analysis by the movement route analyzing system 100, using the image data captured by the imaging device 2 and the detection data of the sensor 3. The posture detecting device 4 detects, for example, whether the person is standing or sitting, or the like. Specifically, when the identification information includes either the "sitting" state or the "standing" state, the result of the posture detection device 4 can be used. The specific configuration and processing of the posture detection device 4 will not be described.

The operation estimation device 5 estimates an operation performed by a person existing in the space to be analyzed by the movement path analysis system 100, using the image data captured by the imaging device 2 and the detection data of the sensor 3. The operation estimation device 5 estimates, for example, whether the person is stopping, walking, running, or carrying an object. Specifically, when the operation type is included in the identification information, the result of the operation estimation device 5 can be used. The specific configuration and processing of the operation estimation device 5 will not be described.

The data integration/movement route generation device 6 generates movement route analysis data of a person or the like in the target space using image data generated by being given identification information by the identification information giving device 1, detection data of the posture by the posture detection device 4, and estimation data of the operation by the operation estimation device 5. By using the generated movement route analysis data, it is possible to effectively improve the arrangement of objects in the target space and the operation content.

< identification information providing device >

As shown in fig. 3, the identification information providing device 1 is an information processing device including a control unit 10, a storage unit 20, a communication interface (I/F)21, an input unit 22, an output unit 23, and the like.

The control unit 10 is a controller that controls the entire identification information providing apparatus 1. For example, the control unit 10 reads and executes the identification information adding program P stored in the storage unit 20, thereby realizing the processing as the acquisition unit 11, the adding unit 12, the correction unit 14, and the update unit 13. The control unit 10 is not limited to the implementation of a predetermined function by the cooperation of hardware and software, and may be a specially designed hardware circuit that implements the predetermined function. That is, the control unit 10 can be realized by various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, and an ASIC.

The storage unit 20 is a recording medium on which various kinds of information are recorded. The storage unit 20 is implemented by, for example, a RAM, a ROM, a flash memory, an ssd (solid State device), a hard disk, other storage devices, or a suitable combination thereof. The storage unit 20 stores information used for identification information, various information acquired for identification information addition, and the like, in addition to the identification information addition program P executed by the control unit 10. For example, the storage unit 20 stores the learning model 200, the image data 210, the sensor data 220, and the identification information 230.

The communication I/F21 is an interface circuit (module) for enabling data communication with an external device (not shown). The input unit 22 is an input means such as an operation button, a keyboard, a mouse, a touch panel, and a microphone used for operation and data input. The output unit 23 is output means such as a display and a speaker for outputting the processing result and data.

The identification information providing apparatus 1 may be realized by a plurality of information processing apparatuses. A part of the data stored in the storage unit 20 may be stored in an external storage device or read from an external storage device and used. For example, the identification information providing device 1 may use the learning model 200, or may be configured to be readable and usable from an external server device or the like.

The acquisition unit 11 acquires a plurality of image data 210 captured by the imaging device 2 ((1) of fig. 4). The acquisition unit 11 also acquires sensor data 220 detected by the sensor 3. Further, the acquisition unit 11 causes the storage unit 20 to store the acquired image data 210 and the sensor data 220. At this time, for example, the image data 210 and the sensor data 220 are associated by including the photographing time of the image data 210 and the detection time of the sensor data 220, respectively.

The plurality of image data 210 acquired by the acquisition unit 11 are continuous data such as moving image data and still image data of a plurality of frames captured continuously. That is, the image data 210 includes data of a plurality of frames continuously photographed. For example, as shown in fig. 5, the acquisition unit 11 acquires image data 210 of a person 320 such as a store clerk or a customer, or a shopping cart 330 in a store in which a plurality of product shelves 310 are arranged. In this case, the image data 210 may be associated with data detected by the sensor 3 provided in the shopping cart 330 as the sensor data 220, or may be associated with data detected by the sensor 3 provided in the product shelf 310, for example.

The assigning unit 12 assigns the identification information to a part of the image data selected from the plurality of image data 210 using the learning model 200 generated in advance. For example, the identification information is information associated with "coordinates" of the object to be extracted, a "width" which is a length in the x-axis direction and a "height" which is a length in the y-axis direction of the region to be extracted, and a "category" which specifies a type of the object in the image data 210. Further, since the image data 210 may include a plurality of objects, a plurality of regions may be extracted from one image data 210 and given a plurality of pieces of identification information. The adding unit 12 associates the identification information with the image data 210 and stores the associated identification information in the storage unit 20.

For example, as a method for the assigning unit 12 to select the image data 210 to which the identification information is assigned from the plurality of image data 210, there is "method 1: method selected at certain intervals "," method 2: method of random selection "," method 3: a method of calculating the feature amount of each image data 210 using image processing and selecting image data 210 having a large difference in feature. For example, in the case of method 1, the calculation of the processing is simple, and the processing time can be shortened compared to other methods. In the case of method 3, since the images have different appearances, the change in the learning data can correspond to a plurality of changed images.

Specifically, in the case of method 1, the assigning unit 12 selects a part of the image data 210 from the image data 210 stored in the storage unit 20 as "for annotation 1" (fig. 4 (2)), and designates the remaining image data 210 as "for annotation 2" (fig. 4 (3)). Assuming that there are 10 ten thousand frames of image data 210, for example, by selecting 1 frame of image data every 200 frames, 500 frames of image data can be selected in total.

The assigning unit 12 performs "1 st annotation processing" using the learning model 200 created in advance, and assigns identification information to each selected image data 210 ((4) of fig. 4). Then, when the learning model 200 is updated by the updating unit 13 using the image data 210 to which the identification information is given by the 1 st annotation process (5) of fig. 4), the giving unit 12 gives the identification information to the image data 210 for the 2 nd annotation to which the identification information is not given in the 1 st annotation process by the updated learning model 200 as the "2 nd annotation process" (fig. 4 ((6)). Then, the learning model 200 is updated by the updating unit 13 using the image data 210 for the 2 nd annotation to which the identification information is given ((7) of fig. 4).

Here, the learning model 200 used by the assigning unit 12 is preferably a model generated from image data having a relationship or similarity with the image data 210. That is, the storage unit 20 stores a plurality of learning models 200 associated with feature information of an image, and the adding unit 12 selects and uses the learning models 200 associated with feature information of the image data 210 to which identification information is added and feature information in a predetermined range. In this way, the conventional learning model 200 can provide the image data 210 with the identification information.

For example, the feature information associated with the image data 210 is the imaging condition of the image data 210. The "shooting conditions" are information such as "shooting location", "purpose of shooting location", "type of shooting location", "mounting position of the shooting device 2", "feature of person included in the image data 210", and the like. Here, the "shooting condition" of the image data 210 is input by the operator via the input unit 22, for example, when the acquisition unit 11 starts acquisition of the image data 210. The adding unit 12 thus uses the learning model 200 associated with the same imaging conditions as those of the image data. The terms "identical" and "not exactly identical" may include, for example, the same or similar general concepts. Further, the learning model 200 having the same and similar combination of a plurality of associated imaging conditions may be used.

Specifically, the same example as the "shooting location" refers to an example of the learning model 200 generated using image data shot at the same location as the image data 210. If the present invention is used for the travel route analysis, for example, if the imaging locations are the same, the accuracy of the travel route analysis can be improved by using the learning model 200 generated using the image data captured at the same location.

The same example as "the purpose of the shooting location" is an example in which the learning model 200 generated using image data shot at a location having the same purpose as the image data 210 is used. The image data 210 is assumed to be captured at a factory, and is an example of using the learning model 200 generated from image data captured at a factory of the same manufactured product, for example. If the present invention is used for the travel route analysis, for example, if the purpose of the imaging location is the same, and in many cases, people move along a similar travel route in the same-purpose location, the accuracy of the travel route analysis can be improved by using the learning model 200 generated using image data captured in the same-purpose location.

The same example as the "type of shooting location" is an example in which the learning model 200 generated using image data captured in a store of the same series as the image data 210 and a store of the same processed product is used. The image data 210 is assumed to be captured in a convenience store, and is an example of using the learning model 200 generated from image data captured in another convenience store, for example. If the present invention is used for the travel route analysis, for example, when the types of the imaging locations are the same, and in many cases, people move along similar travel routes in the same type of location, the accuracy of the travel route analysis can be improved by using the learning model 200 generated using image data captured in the same type of location.

The same example of "the mounting position of the imaging device 2" is an example of using a learning model generated using image data set at the same height. For example, other similar conditions can be combined with the mounting position of the imaging position. For example, by combining the installation position of the imaging device 2 with the purpose and type of the imaging location of the location, the accuracy of the addition of the identification information by the addition unit 12 can be improved.

Note that the same example as "the feature of the person included in the image data 210" refers to an example using the learning model 200 generated from image data including a person having a similar feature to the feature of the person included in the image data 210. The image data 210 is an example of the learning model 200 generated from the image data captured in the shop with many female customers, when the shop with many female customers is captured. In this case, the same conditions can be combined with other characteristics of the person. For example, by combining the characteristics of the person with the purpose and type of the shooting location of the location, the accuracy of the addition of the identification information by the addition unit 12 can be improved. If the image data 210 is used for the movement route analysis, for example, if the characteristics of the persons included in the image data 210 are the same, the movement routes of the persons are often similar, and therefore, the accuracy of the movement route analysis can be improved by using the learning model 200 generated using the image data 210 in which the characteristics of the persons included in the image data 210 are the same.

Further, for example, the feature information associated with the image data 210 may be a "color histogram" of the image data 210. The assigning unit 12 uses the learning model 200 in which the difference between the color histogram of the image data 210 and the color histogram associated with the learning model 200 is within a predetermined range. Here, as the color histogram, a distance function is used in which the hue and saturation values are quantized and the histograms thereof are compared. For example, the color histogram associated with the learning model 200 is obtained by averaging the image data used in the generation of the learning model 200. The "color histogram" can be obtained from the image data 210 by the control unit 10 and stored in the storage unit 20, for example. Further, it is not necessary to obtain all the continuous image data 210, and for example, a color histogram obtained for a certain image data 210 can be used in association with a predetermined frame including the image data 210. Further, instead of the histogram itself, the class of the color histogram may be set according to the tendency of the color histogram, and the learning model 200 associated with the class determined according to the color histogram of the image data 210 may be used.

The learning model 200 to be used may be selected by a combination of the "color histogram" and the "imaging condition" described above. For example, the learning model 200 may be used in which the types of the imaging locations are the same and the difference between the color histograms is within a predetermined range. For example, the learning model 200 may be used in which the purpose of the shooting location and the feature of the person are the same and the difference between the color histograms is within a predetermined range. This can improve the accuracy of the addition of the identification information by the addition unit 12.

For example, as shown in fig. 6, the adding unit 12 adds a plurality of pieces of identification information to the image shown in fig. 5. Specifically, an area of the recognition target is selected from the image data 210, and the recognition information of the recognition target included in the selected area is given. In the example shown in fig. 5, the assigning unit 12 assigns the identification information of "person" to the area of the person 320 (the rectangle shown by the dotted line) and the identification information of "shopping cart" to the area of the shopping cart 330 (the rectangle shown by the dotted line). In the example shown in fig. 6, the area of the person 320 is indicated by a dotted rectangle, and the area of the shopping cart is indicated by a one-dot chain line rectangle. Further, as in the example of fig. 6 in which the areas of the "person" and the "shopping cart" overlap, when the recognition objects are close, the areas of the plurality of recognition objects may overlap.

As shown in fig. 7, the providing unit 12 may provide more detailed identification information to the person. Specifically, in the example shown in fig. 7, more specific identification information such as "male/clerk", "female/customer", "child/customer", and "male/customer" is given to the person.

The updating unit 13 updates the learning model 200 using the image data 210 to which the identification information is given. Specifically, the updating unit 13 performs learning using the image data 210 to which the identification information is given by the "1 st annotation processing" to update the learning model 200. Then, the assigning unit 12 executes "annotation 2 nd processing" using the updated learning model 200, and assigns identification information to the remaining image data 210.

The correction unit 14 receives a request to correct the identification information by displaying the image data 210 including the identification information given by the giving unit 12 on the output unit 23, and corrects the identification information in accordance with the request. That is, since the learning model 200 may not be generated exclusively for the image data 210 captured by the imaging device 2, the accuracy of the learning model 200 and the analysis accuracy of the movement path analysis in the movement path analysis system 100 can be improved by performing the correction by the correction unit 14 as necessary.

For example, as described above in fig. 7, when the identification information is given to the image data 210, for example, when the image data is originally a "male customer" and is erroneously recognized as a "male clerk", or when the image data is originally a "male clerk" and is erroneously recognized as a "shopping cart", the request for correcting the erroneously given identification information is received.

The updating unit 13 updates the learning model 200 using the image data 210 whose identification information is corrected by the correcting unit 14. Therefore, the timing at which the updating unit 13 updates the learning model 200 is: the correction unit 14 confirms to the operator whether or not correction is performed and confirms a timing at which correction is not necessary ((5) of fig. 4), a timing at which update is requested from the correction unit 14 ((5-4) of fig. 8), or a timing at which correction of the information is completed is recognized by the correction unit 14 ((5-6) of fig. 8).

Fig. 8 corresponds to the drawing of fig. 4, and a case (one-dot chain line portion of fig. 8) where the identification information is corrected with respect to the image data to which the identification information is given by the 1 st annotation process will be specifically described. Specifically, the correction unit 14 selects at least a part of the image data 210 to which the identification information is given as correction data ((5-1) of fig. 8), sets the remaining image data 210 as re-annotation data ((5-2) of fig. 8), displays the correction image data on the output unit 23, and checks the presence or absence of correction to the operator. For example, when the identification information is given to the image data 210 of 500 frames by the execution of the 1 st annotation process in the giving unit 12, the correction unit 14 can select, for example, any image data 210 from which the operator arbitrarily selects, as the correction target. As another method, when the identification information is given to the 500-frame image data 210, the correction unit 14 can select 25-frame image data as the correction target by extracting 1-frame image data every 20 frames, for example.

In addition, the correction unit 14 may select the image data 210 to be corrected, in addition to a method in which the operator arbitrarily selects the image data 210 to be corrected. Examples of the method by which the correction unit 14 selects the image data 210 to be corrected include "a method of selecting at regular intervals", "a method of randomly selecting", "a method of calculating the feature amount of each image data 210 using image processing and selecting image data 210 having a large difference in feature".

When a request for correction is input from the operator via the input unit 22, the correction unit 14 corrects the identification information in accordance with the input request, and updates the image data 210 in the storage unit 20 ((5-3) of fig. 8). The correction unit 14 requests the update unit 13 to update the learning model 200 based on the image data 210 associated with the identification information for correction, with respect to the image data 210 for correction (fig. 8 (5-4)). When the learning model 200 is updated by the updating unit 13, the correcting unit 14 requests the adding unit 12 to perform the annotation process again on the image data 210 whose correction of the identification information 230 is completed by the correcting unit 14 by the updated learning model 200, and adds new identification information to correct the identification information ((5-5) of fig. 8). Then, the correcting unit 14 requests the updating unit 13 to update the learning model 200 based on the image data 210 associated with the corrected identification information (fig. 8 (5-6)).

Fig. 8 shows an example in which the identification information providing device 1 performs processing by dividing the data for correction in which the identification information is corrected by the operator and the data for annotation in which the identification information is corrected by the annotation processing by the providing unit 12 when correcting the identification information, or may correct the identification information 230 by the input of the operator for all of the 1 st annotation images.

When the correction unit 14 displays the image data 210 including the identification information on the output unit 23, the identification information added by the addition unit 12 and the identification information corrected by the correction unit 14 can be displayed separately. The correction unit 14 can correct the identification information a plurality of times, and when the image data 210 including the identification information is displayed on the display unit, the identification information that was corrected by the correction unit 14 in the past and the identification information that was newly corrected this time can be displayed separately. Specifically, the regions may be represented by different colors, or by different means (broken lines, one-dot chain lines, double lines, wavy lines, etc.). The characters as the identification information are represented by different colors and different forms (font, character size, underline, and the like). In this way, the correction unit 14 can determine whether or not the selection of the correction target is most appropriate even when the operator selects the image data 210 of the correction target of the identification information by displaying the identification information given at different timings in different modes.

In this way, the identification information providing device 1 can provide identification information by using the existing learning model 200 from the acquired image data 210, and also update the learning model 200 to generate a target learning model. Thus, the assignment of the identification information by the manual operation of the operator is not required, and the operation of assigning the identification information can be simplified and the accuracy of the assigned identification information can be improved.

< identification information providing method >

Next, an identification information providing method performed by the identification information providing apparatus 1 will be described with reference to a flowchart shown in fig. 9.

The identification information providing device 1 acquires the image data 210 captured by the imaging device 2 (S1). The image data 210 is image data 210 of a plurality of frames having continuity.

The identification information providing apparatus 1 selects the image data 210 for the 1 st annotation from the image data 210 acquired in step S1 (S2). The image data 210 not selected in step S2 among the image data acquired in step S1 is the image data 210 for annotation 2.

The identification information providing device 1 executes the 1 st annotation process to provide identification information to the image data 210 selected in step S2 using the learning model 200 stored in the storage unit 20 (S3).

The identification information providing device 1 selects image data for correction from the image data 210 to which identification information has been provided in step S3 (S4).

The identification information providing apparatus 1 displays the image data 210 selected in step S4 on the output unit 23 together with the identification information, and checks whether or not correction is necessary (S5).

When the correction is necessary (yes in S5), the identification information providing apparatus 1 inputs the content of the correction of the identification information through the input unit 22 (S6).

The identification information providing apparatus 1 corrects the identification information of the image data 210 based on the content input in step S6 (S7).

The identification information providing apparatus 1 performs the learning process using the image data 210 whose identification information is corrected in step S7, and updates the learning model 200 (S8).

The identification information adding device 1 performs the 1 st annotation process again using the learning model 200 updated in step S8, and adds identification information to the image data 210 whose identification information has not been corrected in step S7, thereby correcting the identification information (S9).

The identification information providing apparatus 1 performs the learning process using the image data 210 whose identification information is corrected in step S9, and updates the learning model 200 (S10). Here, for example, the evaluation data, which is an image for evaluation that is not used in learning, may be stored in the storage unit 20 in advance, and the processing in steps S4 to S10 may be repeated until the detection rate using the evaluation data becomes equal to or greater than a predetermined threshold value.

If it is determined that the identification information added in step S3 does not require correction (no in S5), the identification information adding device 1 performs learning processing using the image data 210 to which the identification information is added in step S3, and updates the learning model 200 (S11).

The identification information providing apparatus 1 performs the 2 nd annotation process using the learning model 200 updated in step S10 or step S11, and provides identification information to the 2 nd annotation image data 210 (S12).

The identification information providing device 1 executes the learning process using the image data 210 to which the identification information is provided in step S12, and updates the learning model 200 (S13).

In this way, the identification information providing device 1 can provide identification information using the existing learning model 200 from the acquired image data 210, and can generate a target learning model by updating the learning model 200. Thus, the assignment of the identification information by the manual operation of the operator is not required, and the operation of assigning the identification information can be simplified and the accuracy of the assigned identification information can be improved.

[ Effect and supplement ]

As described above, the above-described embodiments have been described as technical examples disclosed in the present application. However, the technique in the present disclosure is not limited to this, and can be applied to an embodiment in which changes, substitutions, additions, omissions, and the like are appropriately made. Accordingly, other embodiments are exemplified below.

Modifications of the examples

(1) Application of motion vectors

As shown in fig. 10, the identification information providing apparatus 1A may further include a selection unit 15. For example, the selection unit 15 detects a moving object from image data selected from a plurality of image data, and selects a region in which the moving object exists.

For example, the selection unit 15 selects the 1 st annotation image data ((2A) of fig. 11). The selection unit 15 also uses the other images as the 2 nd annotation image ((3A) of fig. 11). The selection unit 15 also obtains a motion vector from the selected 2 frames (fig. 4A). The motion vector is a vector that compares pixel information from one frame as a reference to another frame and expresses the difference thereof, and is also called a stream vector. By determining the motion vector, it can be determined that a moving subject exists in the image data. Specifically, the selection unit 15 selects, for example, a rectangular region as a region including a portion where the motion vector is detected. The selection unit 15 stores the "coordinates", "width", and "height" that specify the selected region in the storage unit 20 in association with the image data 210 as the identification information 230.

The correction unit 14 selects the image data 210 to be added of the "type" to be added to the identification information 230 from the image data 210 associated with the region as the identification information 230 ((5A-1) of fig. 11). The correction unit 14 uses the unselected image data 210 as annotation data ((5A-2) of fig. 11). The correction unit 14 displays the area as the identification information 230 together with the image data 210, and corrects the identification information 230 in response to a request for receiving a request for correcting the identification information (fig. 11 (5A-3)). Specifically, a "category" is added to the identification information 230. Then, the correcting unit 14 requests the updating unit 13 to update the learning model 200 based on the image data 210 associated with the corrected identification information ((5A-4) of fig. 11). As in the above case, the learning model 200 preferably learns image data that is the same as or similar to the image data 210 as learning data.

When the learning model 200 is updated by the updating unit 13, the correcting unit 14 requests the adding unit 12 to perform annotation processing on the image data 210 whose correction of the identification information 230 by the correcting unit 14 is not completed by the updated learning model 200, and adds new identification information 230 to correct the identification information 230 ((5A-5) of fig. 8). Then, the correcting unit 14 requests the updating unit 13 to update the learning model based on the image data 210 associated with the corrected identification information 230 (fig. 11 (5A-6)). Next, the assigning unit 12 assigns the identification information to the image data 210 for the 2 nd annotation to which the identification information is not assigned in the 1 st annotation process, as the "2 nd annotation process", by using the updated learning model 200 ((6) of fig. 11). Then, the learning model 200 is updated by the updating unit 13 using the image data 210 for the 2 nd annotation to which the identification information is added ((7) of fig. 11).

In addition, although fig. 11 shows an example in which the identification information providing device 1A performs processing by dividing the data for addition to which the identification information is added by the operator and the data for annotation to which the identification information is corrected by the annotation processing by the providing unit 12 when correcting the identification information, the identification information 230 may be added to all the 1 st annotation images by the input of the operator.

Therefore, even when it is assumed that it is difficult to assign identification information by the 1 st annotation process using the conventional learning model 200, since it is possible to identify a target region by detecting a motion vector and assign identification information using the region, it is possible to improve the accuracy of the assigned identification information by the operation of assigning identification information by the operator.

(2) Annotating only containment categories

In the above example, the example of correcting the identification information 230 that is erroneously added has been described, but the present invention is not limited to this, and the same is true for the case of correcting the higher-order identification information 230 to the lower-order identification information 230. Specifically, in the 1 st annotation process, the identification information 230 of the upper category is assigned, and the identification information 230 of the lower category is corrected by the process of correcting the identification information, whereby the 2 nd annotation process may be executed using the updated learning model 200. For example, first, the learning model 200 may assign the identification information of "person" as the upper category and correct the identification information to the identification information of "male", "female", "adult", "child" and the like as the lower category.

For example, as shown in fig. 12, when the image data 210 for the 1 st annotation and the image data 210 for the 2 nd annotation are selected from the acquired image data 210, the assigning unit 12 assigns the identification information by the learning model 200 ((4B) of fig. 12). Here, the given identification information 230 includes a higher-level category. Then, the correction unit 14 selects the image data 210 for correction from the image data 210 for annotation 1 ((5B-1) of fig. 12), and sets the remaining image data as the image data for annotation 2 ((5B-2) of fig. 12). The correction unit 14 corrects the identification information in accordance with a request input from the operator, and updates the image data 210 in the storage unit 20 ((5B-3) of fig. 12). Here, the corrected identification information 230 includes the lower category. The correction unit 14 requests the update unit 13 to update the learning model 200 based on the image data 210 associated with the identification information for correction, with respect to the image data 210 for correction (fig. 12 (5B-4)).

When the learning model 200 is updated by the updating unit 13, the correcting unit 14 requests the adding unit 12 to perform the annotation process again on the image data 210 for which the correction of the identification information 230 by the correcting unit 14 is not completed by the updated learning model 200, and adds new identification information to correct the identification information ((5B-5) of fig. 12). Here, the corrected identification information 230 includes the lower category. Then, the correcting unit 14 requests the updating unit 13 to update the learning model 200 based on the image data 210 associated with the corrected identification information (fig. 12 (5B-6)). Next, the assigning unit 12 assigns the identification information to the image data 210 for the 2 nd annotation to which the identification information is not assigned in the 1 st annotation process, as the "2 nd annotation process", by using the updated learning model 200 ((6) of fig. 12). Then, the learning model 200 is updated by the updating unit 13 using the image data 210 for the 2 nd annotation to which the identification information is given ((7) of fig. 12).

In addition, although fig. 12 shows an example in which the identification information providing device 1 performs processing by dividing the data for correction in which the identification information is corrected by the operator and the data for annotation in which the identification information is corrected by the annotation processing by the providing unit 12 when the identification information is corrected, the identification information 230 may be corrected by the input of the operator for all of the 1 st annotation images.

When annotating a plurality of categories, the annotation is corrected on the basis of automatically annotating the included categories and finally annotated, so that the operation efficiency is improved.

Brief description of the embodiments

(1) The disclosed identification information providing device is provided with: an acquisition unit that acquires a plurality of image data; an assigning unit that assigns identification information to image data selected from the plurality of image data using the learned learning model; and an updating unit that updates the learning model using the image data to which the identification information is given, and the adding unit adds the identification information to the remaining image data acquired by the acquiring unit using the updated learning model.

Accordingly, it is possible to provide identification information from the acquired image data by using an existing learning model, and to generate a target learning model by updating the learning model, and it is not necessary to provide identification information by a manual operation of an operator, thereby simplifying the operation of providing identification information and improving the accuracy of the provided identification information.

(2) The identification information providing device of (1) may further include: the correction unit displays the image data and the identification information added to the image data by the addition unit on the display unit, receives a request for correcting the identification information, and corrects the identification information in accordance with the request, and the update unit updates the learning model using the image data whose identification information is corrected by the correction unit.

This makes it possible to correct the identification information given by the existing learning model as needed, and to improve the accuracy of the identification information.

(3) In the identification information providing device of (2), the correction unit may display the identification information provided by the providing unit and the identification information corrected by the correction unit separately when the image data and the identification information are displayed on the display unit.

This makes it possible to evaluate the correction operation of the identification information and to improve the accuracy of the correction operation.

(4) In the identification information providing device of (2), the correction unit may correct the identification information a plurality of times, and when the image data and the identification information are displayed on the display unit, the identification information corrected by the correction unit in the past and the identification information newly corrected this time may be displayed separately.

This makes it possible to evaluate the correction operation of the identification information and to improve the accuracy of the correction operation.

(5) In the identification information providing device of (1), the providing unit may use a plurality of learning models including feature information of the image, and may use a learning model including feature information of the image data acquired by the acquiring unit and feature information of a predetermined range.

This makes it possible to select and use the most appropriate learning model from among a plurality of existing learning models, and thus to improve the accuracy of providing identification information.

(6) In the identification information providing device of (5), the feature information may include a photographing condition of the image data, and the providing unit may use a learning model associated with the same photographing condition as the photographing condition of the image data.

This makes it possible to select and use the most appropriate learning model from among a plurality of existing learning models, and thus to improve the accuracy of providing identification information.

(7) In the identification information providing device of (5), the feature information may include a color histogram of the image data, and the providing unit may use a learning model associated with the color histogram in which a difference from the color histogram of the image data is within a predetermined range.

This makes it possible to select and use the most appropriate learning model from among a plurality of existing learning models, and thus to improve the accuracy of providing identification information.

(8) The identification information providing device of (2) may further include: the image processing apparatus includes a selection unit that detects a moving object from image data selected from a plurality of image data, and selects a region of the object, and a correction unit that displays the image data and the region selected by the selection unit for the image data as identification information, receives a request for correcting the identification information, and corrects the identification information in accordance with the request.

This makes it possible to select a region where motion exists from the acquired image data, and to facilitate the addition of identification information by manual operation of the operator.

(9) In the identification information providing apparatus of (1), the plurality of image data may be image data having continuity.

This makes it possible to provide identification information from the acquired image data having continuity by using an existing learning model, and also to generate a target learning model by updating the learning model.

(10) The identification information providing method of the present disclosure includes: a step of acquiring a plurality of image data; assigning identification information to image data selected from the plurality of image data using the learned learning model; updating the learning model using the image data to which the identification information is given; and a step of giving identification information to the acquired remaining image data using the updated learning model.

This makes it possible to add identification information from the acquired image data using an existing learning model, to generate a target learning model by updating the learning model, and to simplify the operation of adding identification information and improve the accuracy of the added identification information without adding identification information by manual operation of an operator

(11) The program of the present disclosure causes a computer to execute (10) the method.

Accordingly, it is possible to provide identification information from the acquired image data by using an existing learning model, and it is possible to generate a target learning model by updating the learning model, and it is not necessary to provide identification information by a manual operation of an operator, thereby simplifying the operation of providing identification information and improving the accuracy of the provided identification information.

The identification information providing device and the identification information providing method according to all claims of the present disclosure can be realized by cooperation with hardware resources, such as a processor, a memory, and a program.

Industrial applicability

The identification information providing device and the identification information providing method of the present disclosure are useful for creating teacher data for machine learning.

-description of symbols-

1. 1A identification information providing device

10 control part

11 acquisition part

12 applying part

13 update part

14 correcting part

20 storage part

200 learning model

210 image data

220 sensor data

230 identification information

P-ID information providing program

21 communication I/F

22 input unit

23 output unit (display unit).