阅读说明：本技术 元件图像识别用学习完成模型生成系统及元件图像识别用学习完成模型生成方法 (Learning completion model generation system for component image recognition and learning completion model generation method for component image recognition ) 是由 小林贵纮 鬼头秀一郎 横井勇太 于 2018-02-09 设计创作，主要内容包括：元件图像识别用学习完成模型生成系统以吸附于元件安装机(12)的吸嘴(31)的元件或安装于电路基板(11)的元件为拍摄对象,通过相机(18、22)来拍摄该拍摄对象并生成在进行图像识别时使用的学习完成模型,元件图像识别用学习完成模型生成系统具备取得在成为基准的元件的图像识别中使用的基准学习完成模型的计算机(23)。该计算机针对成为基准的元件具有预定的类似关系的元件的每个种类收集样本元件图像,并针对该元件的每个种类追加该样本元件图像作为上述基准学习完成模型的教师数据而进行再学习,从而针对该元件的每个种类生成用于该元件的图像识别的按元件类别学习完成模型。(A learning completion model generation system for component image recognition is provided with a computer (23) for acquiring a reference learning completion model used for image recognition of a component to be used as a reference, the learning completion model generation system being configured to take an image of a component attached to a suction nozzle (31) of a component mounting machine (12) or a component mounted on a circuit board (11) as an image pickup object and to generate a learning completion model used for image recognition by cameras (18, 22). The computer collects a sample element image for each type of element having a predetermined similarity relationship with respect to a reference element, adds the sample element image to each type of element as teacher data of the reference learning completion model, and re-learns the sample element image, thereby generating a learning completion model for each element type for image recognition of the element.)

1. A learning model generation system for recognizing component images, which takes a component adsorbed on a suction nozzle of a component mounting machine or a component mounted on a circuit board as an imaging object, images the imaging object by a camera and generates a learning model used for recognizing images,

the learning model generation system for component image recognition includes a computer for acquiring a reference learning model used for image recognition of a component to be a reference,

the computer collects a sample element image for each type of an element having a predetermined similarity relationship with the element serving as the reference, adds the sample element image to each type of the element as teacher data of the reference learning completion model, and re-learns the element, thereby generating a learning completion model for each element type for image recognition of the element.

2. The learning completion model generation system for component image recognition according to claim 1,

the element-by-element type learning completion model generated for each type of the element by the computer is included in the element shape data for image processing prepared for each type of the element.

3. The learning completion model generation system for component image recognition according to claim 1 or 2, wherein,

the element having a predetermined similar relationship with the element serving as the reference means an element having the same or similar shape although any one of the size, color, raw material, manufacturing company, and manufacturing lot is different from the element serving as the reference.

4. The learning completion model generation system for component image recognition according to any one of claims 1 to 3,

the computer collects, as the sample component image, an image obtained by photographing the photographic subject by a camera of a component mounter or a camera of an inspection machine in production.

5. The learning completion model generation system for component image recognition according to any one of claims 1 to 4,

the reference learning completion model and the component type learning completion model are learning completion models for determining whether the suction posture of the component sucked to the suction nozzle is normal suction or abnormal suction.

6. The learning completion model generation system for component image recognition according to any one of claims 1 to 4,

the reference learning completion model and the component-by-component learning completion model are learning completion models for determining whether or not a component attached to the suction nozzle is present.

7. The learning completion model generation system for component image recognition according to any one of claims 1 to 4,

the reference learning completion model and the learning completion model for each component type are learning completion models for determining whether or not a component mounted on the circuit board is present.

8. The learning completion model generation system for component image recognition according to any one of claims 1 to 7,

the computer transfers the generated component-by-component classification learning completion model to a component mounting machine or inspection machine using the component-by-component classification learning completion model.

9. A learning completion model generation method for component image recognition, which takes a component adsorbed on a suction nozzle of a component mounting machine or a component mounted on a circuit board as an imaging object, images the imaging object by a camera and generates a learning completion model used for image recognition,

the method for generating a learning completion model for component image recognition includes the steps of:

acquiring a reference learning completion model used for image recognition of a reference element;

collecting a sample element image for each kind of element having a predetermined similar relationship with the element that becomes the reference; and

the acquired sample element image is added to the teacher data of the reference learning completion model for each type of the element and relearning is performed, and a learning completion model for each element type for image recognition of the element is generated for each type of the element.

Technical Field

The present specification discloses a technique for a learning completion model generation system for component image recognition and a learning completion model generation method for component image recognition, in which a component attached to a suction nozzle of a component mounting machine or a component mounted on a circuit board is taken as an imaging target, and the imaging target is imaged by a camera to generate a learning completion model used for image recognition.

Background

As for the suction posture of the component sucked to the suction nozzle of the component mounting machine, the component is sucked horizontally if it is normally sucked, but there is a case where abnormal suction in which the component is sucked in an abnormal posture such as inclination occurs due to some cause. Since such abnormal suction causes a defective component mounting, a camera for picking up an image of a component sucked by a suction nozzle is mounted in a component mounting machine, and an image picked up by the camera is processed to determine whether the suction posture of the component is normal suction or abnormal suction, and the component determined to be abnormally sucked is discarded, and only the component determined to be normally sucked is mounted on a circuit board.

In the conventional general image processing, the normal suction/abnormal suction is discriminated using the image processing component shape data including the size of the component, but in the case where the component to be sucked to the suction nozzle is a minute component, it is sometimes difficult to discriminate the normal suction/abnormal suction by the image processing using the conventional image processing component shape data.

Therefore, as described in patent document 1 (japanese patent application laid-open No. 2008-130865), a learning completion model for discriminating normal suction/abnormal suction is generated in advance by using a mechanical learning method such as a neural network, a component image captured by a camera of a component mounter is processed during production, and normal suction/abnormal suction is discriminated by using the learning completion model.

Disclosure of Invention

Problems to be solved by the invention

For example, even if the elements have the same electrical specification, there may be a difference in size, color, material, manufacturing company, manufacturing lot, and the like, and the image recognition result may be different depending on the difference. However, in the case of elements having the same electrical specifications before the start of production, if the types of elements are refined in accordance with the size, color, material, manufacturing company, manufacturing lot, and the like and a learning completion model is to be generated by a mechanical learning method for all the types, a very large number of learning completion models must be generated, and the learning completion model generation operation takes a large number of man-hours.

Therefore, for elements of a type similar to the elements for which the learning completion model has already been generated, such as the shape, there are cases where the normal adsorption/abnormal adsorption is discriminated using the existing learning completion model, but in this case, the discrimination accuracy expected in production cannot be obtained in some cases. In this case, it is necessary to quickly generate a learning completion model dedicated to the element, but it takes a long time to generate the learning completion model from the beginning by a conventional method.

Means for solving the problems

In order to solve the above problems, a learning model creation system for component image recognition takes a component attached to a suction nozzle of a component mounter or a component mounted on a circuit board as an imaging target, the photographic subject is photographed by a camera and a learning completion model used in performing image recognition is generated, the learning model generation system for component image recognition includes a computer for acquiring a reference learning model used for image recognition of a component to be a reference, the computer collects a sample element image for each kind of element having a predetermined similar relationship with the element that becomes the reference, and the sample element image is added for each type of the element as teacher data of the reference learning completion model for relearning, thereby generating a learning completion model by component category for image recognition of the component for each category of the component.

In short, for an element having a predetermined similarity relationship with an element serving as a reference for generating a reference learning completion model, a sample element image is collected for each type of the element, and the sample element image is added as teacher data of the reference learning completion model for each type of the element and relearning is performed, thereby generating a learning completion model for each element type for image recognition of the element. In this way, it is possible to relatively easily generate a learning completion model for each component type for image recognition of a component having a predetermined similarity relationship with a component serving as a reference from the reference learning completion model.

Drawings

Fig. 1 is a block diagram showing a configuration example of a component mounting line according to an embodiment.

Fig. 2 is a front view illustrating normal adsorption.

Fig. 3 is a front view illustrating the inclined adsorption.

Fig. 4 is a flowchart showing a flow of processing of the component suction posture determination program.

Fig. 5 is a flowchart showing a flow of processing of the learning completion model generation program for each component type.

Detailed Description

One embodiment is explained below.

First, the structure of the component mounting line 10 will be described with reference to fig. 1.

The component mounting line 10 is configured by arranging one or more mounting-related lines such as a component mounter 12, a solder printer 13, and a flux applying device (not shown) along the conveying direction of the circuit board 11. An inspection machine 14 for inspecting whether or not the mounting state of each component mounted on the circuit board 11 is good is provided on the board carrying-out side of the component mounting line 10.

Each of the component mounting machines 12, the solder printers 13, and the inspection machines 14 of the component mounting line 10 are connected to a production management computer 21 via a network 16 so as to be able to communicate with each other, and the production of the component mounting line 10 is managed by the production management computer 21. The control device 17 of each component mounting machine 12 is mainly configured by one or more Computers (CPUs), moves a mounting head (not shown) along a path of a component suction position → a component imaging position → a component mounting position in accordance with a production job (production program) sent thereto from the production management computer 21, sucks a component supplied from a feeder 19 (see fig. 2 and 3) by a suction nozzle 31 (see fig. 2 and 3) of the mounting head, images the component from below the component by the component imaging camera 18, processes the picked-up image by an image processing function of the control device 17 of the component mounting machine 12, and determines whether the suction posture of the component is normal suction (see fig. 2) or abnormal suction (see fig. 3) by using a learning completion model described later. As a result, if abnormal suction is determined, the component is discarded in a predetermined discard box (not shown), and if normal suction is determined, the suction position X, Y and the angle θ of the component are repeatedly measured, and the deviation between the suction position X, Y and the angle θ of the component is corrected, and the component is mounted on the circuit board 11, so that a predetermined number of components are mounted on the circuit board 11.

The control device 20 of the inspection machine 14 is mainly configured by one or more Computers (CPUs), and the inspection camera 22 captures the mounting state of each component on the circuit board 11 carried in from above, processes the captured image, recognizes the presence or absence of each component on the circuit board 11 and the mounting state such as a mounting position deviation, and inspects the presence or absence of a mounting failure (inspection failure) of each component based on the recognition result. In this case, the presence or absence of each element on the circuit board 11 may be determined using a learning completion model described later.

A learning computer 23 is connected to the network 16 of the component mounting line 10, and the learning computer 23 collects and learns teacher data (sample component images) used for generating a reference learning completion model and a learning completion model for each component type, which will be described later.

The control device 17 of each component mounting machine 12 executes a component suction posture determination program of fig. 4 described later during production, selects a learning completion model corresponding to the type of the component sucked by the suction nozzle 31, determines whether the suction posture of the component is normal suction or abnormal suction based on the processing result of the captured image of the component, transmits the captured image determined to be normal suction to the learning computer 23 as a sample component image for normal suction, and transmits the captured image determined to be abnormal suction to the learning computer 23 as a sample component image for abnormal suction.

On the other hand, the learning computer 23 executes a component-by-component learning completion model generation program shown in fig. 5, which will be described later, to classify and collect sample component images of normal suction/abnormal suction delivered from the control device 17 of each component mounter 12 for each component type, and acquires information on the inspection result of the inspection machine 14, calculates the mounting failure occurrence rate for each type of component, when there is a component whose mounting failure occurrence rate exceeds a determination threshold, a sample component image of normal adsorption/abnormal adsorption collected for the component is added as teacher data of a reference learning completion model and relearning is performed, the component-by-component learning completion model for image recognition of the component is generated, and the component-by-component learning completion model is transmitted to the control device 17 of each component mounter 12. As the method of relearning, a mechanical learning method such as a neural network or a support vector machine may be used.

Here, the reference learning completion model is a learning completion model used for image recognition of a reference element, and may be generated by the learning computer 23 collecting a sample element image of normal adsorption/abnormal adsorption of the reference element as teacher data and performing learning by mechanical learning such as a neural network or a support vector machine, or may be a learning completion model generated by an external computer being taken into the learning computer 23. The reference element is not limited to a specific element, and an element in which a learning completion model is generated in advance may be referred to as a "reference element".

The controller 17 of each component mounting machine 12 stores the reference learning completion model and the learning completion model for each component type, which are sent from the learning computer 23, in a storage device (not shown) in association with the type of component to be image-recognized using the models. In this case, the image processing element shape data prepared for each type of element is stored while including the reference learning completion model or the element type learning completion model. In the following description, the term "learning completion model" is simply used to include both the reference learning completion model and the learning completion model for each component type. The image processing device shape data is data indicating external features such as the size of the main body of the device, the position, size, pitch, number of terminals such as bumps and pins, and is used for discriminating the type of the device recognized by the image, or measuring the suction position and angle of the device. The process of including the learning completion model generated for each type of component in the image processing component shape data prepared for each type of component may be performed by the control device 17 of each component mounting apparatus 12, or may be performed by the learning computer 23. Alternatively, the learning model may be transferred from the learning computer 23 to the production management computer 21, the learning model may be included in the image processing component shape data by the production management computer 21, and the image processing component shape data including the learning model may be transferred from the production management computer 21 to the control device 17 of each component mounting apparatus 12.

The control device 17 of each component mounting machine 12 selects the learning completion model for the component to perform the image recognition of the component when the learning completion model for the image recognition of the component attached to the suction nozzle 31 exists among the learning completion models stored for each type of component, but when the learning completion model for the component does not exist, regards a component having a predetermined similarity relationship with the component attached to the suction nozzle 31 among the components having the learning completion model as a "reference component", and uses the learning completion model for the reference component as the "reference learning completion model" to perform the image recognition of the component attached to the suction nozzle 31. In this case, the learning completion model for the reference element may be a learning completion model for each element type generated from a reference learning completion model for another element, and in this case, the learning completion model for each element type generated from the reference learning completion model for another element may be used as the reference learning completion model.

In this case, the elements having a predetermined similar relationship are, for example, elements whose shapes are the same or similar although any one of the size, color, raw material, manufacturing company, manufacturing lot, and the like of the elements is different. If the elements have a predetermined similarity relationship with each other, even if the image recognition of one element is performed using the learning completion model for the other element, the image recognition can be performed with a certain degree of accuracy (generally, at least the minimum accuracy required for production). In other words, if the image recognition of one element can be performed with a certain degree of accuracy using the learning completion model for the other element, it can be said that the two elements have a predetermined similarity relationship.

Next, the flow of processing of the component suction posture determination program of fig. 4 and the component type learning completion model generation program of fig. 5 will be described.

[ Process for determining the orientation of component suction ]

The component suction posture determination program of fig. 4 is executed by the control device 17 of each component mounting machine 12 every time when the component suction camera 18 picks up an image of a component sucked to the suction nozzle 31 of each component mounting machine 12 during production.

When the control device 17 of each component mounter 12 starts the program, first, in step 101, the component attached to the suction nozzle 31 is picked up by the component pickup camera 18, and the picked-up image is taken in. Then, the process proceeds to step 102, where it is determined whether or not there is a learning completion model for the captured component among learning completion models stored for each type of component in a storage device (not shown), and if there is a learning completion model for the captured component, the process proceeds to step 103, where the learning completion model for the captured component is selected as the learning completion model for the current image recognition.

On the other hand, if there is no learning completion model for the captured component among the learning completion models stored in the storage device for each type of component, the process proceeds to step 104, and a learning completion model for a component having a predetermined similarity relationship with the captured component is selected as the learning completion model for the current image recognition from among the learning completion models stored in the storage device for each type of component.

As described above, after the learning completion model for the current image recognition is selected, the process proceeds to step 105, the current captured image is processed by the image processing function of the control device 17, and whether the picked-up component is normally picked up (see fig. 2) or abnormally picked up (see fig. 3) is determined using the selected learning completion model.

Then, the routine proceeds to step 106, where it is determined whether or not the determination result of the suction posture is normal suction, and if normal suction is performed, the routine proceeds to step 107, where the current captured image is transmitted to the learning computer 23 as a normally-sucked sample element image, and the routine is ended. On the other hand, if the determination result of the suction posture is not normal suction but abnormal suction, the routine proceeds to step 108, and the present captured image is transferred to the learning computer 23 as a sample element image of abnormal suction, and the routine is ended. Thus, the learning computer 23 collects sample component images of normal suction/abnormal suction from the control device 17 of each component mounting apparatus 12.

The control device 17 of each component mounter 12 may temporarily collect the image of the normally sucked/abnormally sucked sample component. In this case, the control device 17 of each component mounting machine 12 collectively transmits the sample component images collected up to that time to the learning computer 23 every time a predetermined number of sample component images are collected (or every time a predetermined period of time is collected), or the control device 17 of each component mounting machine 12 collectively transmits the sample component images collected up to that time to the learning computer 23 every time a sample component image transmission request is output from the learning computer 23. Alternatively, the production management computer 21 may collect the sample component images from the control devices 17 of the respective component mounting machines 12, and transfer the sample component images from the production management computer 21 to the learning computer 23. In either way, the learning computer 23 can ultimately collect the sample element images.

[ learning by component class to complete model creation program ]

The learning computer 23 repeatedly executes the learning completion model generation program for each component type shown in fig. 5 at predetermined cycles. When the learning computer 23 starts the program, first, in step 201, a sample component image of normal suction/abnormal suction is collected for each component type from the control device 17 of each component mounting machine 12 or the production management computer 21. Then, in the next step 202, the inspection machine 14 acquires information of the inspection result.

Then, the process proceeds to step 203, and a sample component image obtained by imaging a component determined to be improperly mounted by the inspection machine 14 is discarded from among the collected normally-adsorbed sample component images. The reason for this is that even if it is determined that the component is normally sucked, the component determined to be improperly mounted by the inspection machine 14 may actually be abnormally sucked. The processing in step 203 may be performed by the control device 17 of each component mounting machine 12 or the production management computer 21, and only images obtained by capturing images of components that have not been determined as being mounted poorly by the inspection machine 14 may be collected by the learning computer 23 as sample component images that are normally picked up.

Then, the process proceeds to step 204, where a mounting failure occurrence rate is calculated for each component type based on information of the inspection result obtained from the inspection machine 14. Then, the routine proceeds to step 205, where it is determined whether or not there is a component whose calculated mounting failure occurrence rate exceeds a predetermined determination threshold, and if there is no component whose mounting failure occurrence rate exceeds the determination threshold, it is determined that the accuracy of image recognition using the selected learning completion model is ensured (it is not necessary to generate a learning completion model for each component type), and the routine is ended.

On the other hand, if there is a component whose mounting failure occurrence rate exceeds the determination threshold, it is determined that the accuracy of image recognition is not secured for the component (generation of a learning completion model for each component type is necessary), and the process proceeds to step 206, where a sample component image collected for the component and normally adsorbed/abnormally adsorbed is added as teacher data of a reference learning completion model used for image recognition of the component and is relearned, thereby generating a learning completion model for each component type for the component. Then, the routine proceeds to step 207, and the generated learning completion model for each component type is transferred to the control device 17 of each component mounter 12, and the routine is ended. Thus, the controller 17 of each component mounter 12 is in a state capable of performing image recognition using the component-by-component learning completion model delivered from the learning computer 23.

According to the present embodiment described above, since the element-by-element type learning completion model for image recognition of the element is generated for each type of the element by collecting the sample element image for each type of the element and adding the sample element image for each type of the element as the teacher data of the reference learning completion model and performing relearning, with respect to the element having a predetermined similarity relationship with respect to the element serving as the reference for which the reference learning completion model is generated, the element-by-element type learning completion model for image recognition of the element having the predetermined similarity relationship with respect to the element serving as the reference can be relatively easily generated from the reference learning completion model, and the number of steps for generating the learning completion model can be reduced.

In addition, in the present embodiment, since the image processing component shape data prepared for each type of the component includes the component-by-component type learning completion model generated for each type of the component, the same image recognition using the component-by-component type learning completion model can be performed even in a component mounter of another component mounting line that can use the image processing component shape data, and there is an advantage that the production quality is improved and stabilized.

However, the learning completion model for each component type may not be associated with the image processing component shape data and may be managed separately.

In the present embodiment, the component held by the suction nozzle 31 of each component mounting apparatus 12 is picked up by the component pickup camera 18 during production, and the picked-up image is processed to determine whether the component is normally picked up or abnormally picked up, and the picked-up image determined to be normally picked up is collected as a sample component image for normal pickup, and the picked-up image determined to be abnormally picked up is collected as a sample component image for abnormal pickup, so that the image picked up by the component pickup camera 18 during production can be collected as a sample component image for normal pickup/abnormal pickup, and the number of steps for collecting the sample component image can be reduced.

However, the method of collecting the sample component images is not limited to the method of collecting the sample component images during production, and for example, the component image pickup camera 18 may capture images of a normally-suctioned component and an abnormally-suctioned component respectively which are suctioned to the suction nozzle 31 of the component mounter 12 before the start of production, and collect the captured images as sample component images of normal suction/abnormal suction. Alternatively, a dedicated imaging device for imaging a sample element image may be used, and the sample element images of normal adsorption/abnormal adsorption imaged by the imaging device may be collected. In the case where a dedicated imaging device is used, the image of the normally adsorbed/abnormally adsorbed sample element can be collected before the start of production, during production, or after the end of production.

In the present embodiment, when a component having a mounting failure occurrence rate exceeding a predetermined determination threshold value is produced during production, a sample component image of normal suction/abnormal suction collected with respect to the component is added to teacher data of a reference learning completion model used for image recognition of the component and is relearned, and a component-by-component type learning completion model for the component is generated and transmitted to the control device 17 of each component mounter 12.

However, the generation of the learning completion model for each component type may be performed before the start of production or after the end of production. Alternatively, the learning completion model by element type may be generated at a point in time when the number of collected normal adsorption/abnormal adsorption sample element images exceeds a predetermined number.

The learning completion model of the present embodiment is a learning completion model for determining whether the suction posture of the component sucked by the suction nozzle 31 is normal suction or abnormal suction, but may be a learning completion model for determining whether or not the component sucked by the suction nozzle 31 is present. In this case, the image captured by the component imaging camera 18 in a state where the component adsorbed to the suction nozzle 31 is present may be collected as a sample component image with the component, the image captured by the component imaging camera 18 in a state where the component adsorbed to the suction nozzle 31 is not present may be collected as a sample component image without the component, and the sample component image with/without the component classified for each type of the component may be added as teacher data of the reference learning completion model for image recognition of the component to be relearned, thereby generating a component-by-component learning completion model for the component for each type of the component. In this case, the sample element image may be collected by using a dedicated imaging device.

The inspection machine 14 may inspect whether or not a component is present on the circuit board 11 using a learning completion model for determining whether or not a component mounted on the circuit board 11 is present. In this case, the control device 20 of the inspection machine 14 may capture an image of the mounted state of each component on the circuit board 11 carried in by the inspection camera 22, process the captured image, inspect whether or not each component is present on the circuit board 11 using a learning completion model, collect the captured image determined as the component-present sample component image and the captured image determined as the component-absent sample component image, add the component-present/component-absent sample component images classified for each type of the component as teacher data of a reference learning completion model for image recognition of the component, and re-learn the component-by-component type learning completion model for each type of the component. In this case, the sample element image may be collected by using a dedicated imaging device.

It is needless to say that the present invention can be implemented by changing the configuration of the component mounting line 10, appropriately changing the processing contents and the processing order of the respective programs shown in fig. 4 and 5, and by making various changes without departing from the scope of the invention.

Description of the reference numerals

10. A component mounting production line; 11. a circuit substrate; 12. a component mounting machine; 14. an inspection machine; 17. a control device of the component mounting machine; 18. a camera for shooting the component; 19. a feeder; 20. a control device of the inspection machine; 21. a computer for production management; 22. an inspection camera; 23. a learning computer; 31. a suction nozzle.