阅读说明：本技术 面向在线语义分割机器视觉检测的多任务协同调度方法 (Multi-task cooperative scheduling method for online semantic segmentation machine vision detection ) 是由 刘桂雄 黄坚 于 2021-05-17 设计创作，主要内容包括：本发明公开了一种面向在线语义分割机器视觉检测的多任务协同调度方法,包括建立检测鉴别协同调度任务模型；高分辨率图像分块语义分割任务调度,将图像分为N-(sub)个分辨率U-(sub)×V-(sub),调度目标为得到高分辨率图像语义分割时间T-(High)最小值下的U-(sub)、V-(sub)、N-(sub)值；多网络多图像分批语义分割任务调度,将多图像打包为N-(batch)个图像每组、GPU上加载N-(CNN-GPU)个语义分割网络模型进行分批并行处理,调度目标为得到多网络多图像分批语义分割时间T-(Low)最小值下的N-(batch)、N-(CNN-GPU)值；调度检测鉴别协同调度任务模型,得到检测鉴别总时间T-(inspect)最小值下执行顺序,完成检测鉴别协同调度。(The invention discloses a multitask cooperative scheduling method for online semantic segmentation machine vision detection, which comprises the steps of establishing a detection and identification cooperative scheduling task model; high resolution image blocking semantic segmentation task scheduling, dividing image intoN sub Resolution U sub ×V sub Scheduling the target to obtain the high resolution image semantic segmentation time T High U at minimum sub 、V sub 、N sub A value; scheduling multi-network multi-image batch semantic segmentation task, packaging multi-image into N batch Each group of images and GPU are loaded with N CNN‑GPU The multiple semantic segmentation network models are subjected to batch parallel processing, and the scheduling objective is to obtain multiple network multiple image batch semantic segmentation time T Low N at minimum batch 、N CNN‑GPU A value; scheduling detection and identification are carried out on the cooperative scheduling task model to obtain the total detection and identification time T inspect And executing the sequence under the minimum value to finish the detection and identification cooperative scheduling.)

1. The multitask collaborative scheduling method for online semantic segmentation machine vision detection is characterized by comprising the following steps:

a, establishing a detection and identification cooperative scheduling task model;

b, high-resolution image blocking semantic segmentation task scheduling, namely dividing the image into N_subResolution U_sub×V_subScheduling the target to obtain the high resolution image semantic segmentation time T_HighU at minimum_sub、V_sub、N_subA value;

c multi-network multi-image batch semantic segmentation task scheduling, and packaging multi-image into N_batchEach group of images and GPU are loaded with N_CNN-GPUThe semantic segmentation network model carries out batch parallel processing, and the dispatching aim is to obtain multiple networksMulti-image batch semantic segmentation time T_LowN at minimum_batch、N_CNN-GPUA value;

d, scheduling, detecting and identifying the cooperative scheduling task model to obtain the total detection and identification time T_inspectAnd executing the sequence under the minimum value to finish the detection and identification cooperative scheduling.

2. The multi-task cooperative scheduling method for online semantic segmentation machine vision inspection according to claim 1, wherein the step a of inspecting and identifying the cooperative scheduling model task comprises: start Activity S, locate Activity P_{Mov_i_j}Light source control activity P_{Light_i_j}Image acquisition activity P_{Acq_i_j}Semantic segmentation activity G_{Seg_i_j}(ii) a The semantic segmentation activity G_{Seg_i_j}The method comprises the following steps: block scheduling partition subactivity P_{High_i_j_m}Batch scheduling partitioning sub-activity P_{Low_i_j}(ii) a Wherein, the subscripts i and j represent the j imaging condition of the i detection object, and the number of the detection objects is set as N_itemDetecting the condition number N_condI is 1, … N_itemJ has a value range of 1, … N_cond(ii) a Subscript m represents the mth chunking scheduled segmentation activity, set to the high resolution image number N_highWhen m is 1, … N_high。

3. The on-line semantic segmentation machine vision inspection-oriented multitask cooperative scheduling method according to claim 1, wherein in the step B, the image is divided into N_subResolution U_sub×V_subAspect ratio of not more than gamma_scaleThe sub-image and high-resolution image blocking scheduling optimization mathematical model is as follows:

the physical significance of the mathematical model is: sub-image segmentation video memory occupation under constraint conditionThe length-width ratio of the subimages is less than or equal to gamma_scaleCalculating the total division timeU at minimum_sub、V_sub、N_subThe value is obtained.

4. The on-line semantic segmentation machine vision inspection-oriented multitask cooperative scheduling method according to claim 1, wherein in the step B, the high-resolution image blocking semantic segmentation task scheduling solving method comprises the following steps: for the online semantic segmentation machine vision detection system, a semantic segmentation network model f is used_CNNVideo memory occupation M of (·) segmented image_segVideo memory occupation M including deep learning framework_frameworkModel memory occupation M_CNNImage display memory occupation M_imgM is occupied by characteristic mapping video memory_tensor(ii) a Through f_CNN(. cut) various image pixel numbers p_allImage, calibrationIs fitted out Then:

thus, video memory occupancy estimationEstimation of upper limit of pixel quantityRespectively as follows:

substituting the formula (3) into the formula (1) to obtain a high-resolution image block scheduling optimization mathematical model expression:

。

5. the on-line semantic segmentation machine vision inspection-oriented multitask cooperative scheduling method as claimed in claim 1, wherein in the step C, the multi-network multi-image batch semantic segmentation task scheduling is performed to obtain semantic segmentation timeAt minimum valueA value;

packing multiple images into N_batchEach group of images and GPU are loaded with N_CNN-GPUThe semantic segmentation network models perform batch parallel processing, and the multi-network multi-image batch scheduling mathematical model is as follows:

the physical significance of the mathematical model is as follows: image segmentation video memory occupation under constraint conditions To findN at minimum_batch、N_CNN-GPUThe value is obtained.

6. The on-line semantic segmentation machine vision inspection-oriented multitask cooperative scheduling method according to claim 1, wherein the multi-network multi-image batch semantic segmentation task scheduling solving method in the step C is as follows:

by different N_CNN-GPUDifferent N_batchMultiple networks f_CNN(. 2) segmenting the multiple images, scalingIs fitted outThen:

thus, video memory occupancy estimationComprises the following steps:

M_GPUlower number of pixels p_allUpper limit of number of image batchesUpper limit of number of semantically partitioned networksRespectively as follows:

substituting the formula (7), the formula (8) and the formula (9) into the formula (5) to obtain a mathematical model expression for multi-network multi-image batch scheduling optimization:

。

7. the on-line semantic segmentation machine vision inspection-oriented multitask cooperative scheduling method according to claim 1, wherein the step D specifically comprises the following steps:

total detection and identification time T after detection and identification cooperative scheduling for online semantic segmentation machine vision detection_inspectComprises the following steps:

T_{Mov_i_j}、T_{Light_i_j}、T_{Acq_i_j}、T_{Seg_i_j}respectively as nodes P of a cooperative scheduling DAG model_{Mov_i_j}、P_{Light_i_j}、P_{Acq_i_j}、G_{Seg_i_j}And (4) solving the collaborative scheduling DAG model by adopting a longest path priority algorithm according to the weight value to obtain an execution sequence, thereby realizing scheduling.

Technical Field

The invention relates to the technical field of line machine visual detection based on deep learning, in particular to a multitask cooperative scheduling method based on deep learning semantic segmentation.

Background

The machine vision system based on deep learning semantic segmentation has the problems of long segmentation time, large video memory occupation and the like under the conditions of high-resolution images, multi-network multi-images, detection and identification and the like, and has negative influence on the online real-time performance of the machine vision system. The multi-task parallel scheduling method is beneficial to improving indexes such as segmentation time, video memory occupation and the like of online machine vision detection and identification. The bottom layer parallel scheduling method can optimize the calculation efficiency of equipment, reduce the idle rate and improve the parallel rate, and a proper bottom layer parallel scheduling method and hardware are required to be selected in the design stage; the high-level scheduling method can be used for optimizing the segmentation time and video memory occupation when processing a large amount of image data and a complex CNN model; the multi-level scheduling method can realize parallel and cooperative processing of various steps of the detection process, and shorten the total segmentation time.

The invention provides a multitask collaborative scheduling method for online semantic segmentation machine vision detection. The core of the method is as follows: firstly, a high-resolution image processing process is intelligently scheduled, and the storage cost is reduced; secondly, a large number of image processing processes are intelligently scheduled, and the computing resource efficiency and real-time capability are improved; and thirdly, all key modules of the machine vision system work cooperatively.

Disclosure of Invention

In order to solve the technical problems, the invention aims to provide a multitask collaborative scheduling method for online semantic segmentation machine visual inspection.

The purpose of the invention is realized by the following technical scheme:

a multitask cooperative scheduling method for online semantic segmentation machine vision detection comprises the following steps:

a, establishing a detection and identification cooperative scheduling task model;

b, high-resolution image blocking semantic segmentation task scheduling, namely dividing the image into N_subResolution U_sub×V_subScheduling the target to obtain the high resolution image semantic segmentation time T_HighU at minimum_sub、 V_sub、N_subA value;

c multi-network multi-image batch semantic segmentation task scheduling, and packaging multi-image into N_batchEach group of images and GPU are loaded with N_CNN-GPUThe multiple semantic segmentation network models are subjected to batch parallel processing, and the scheduling objective is to obtain multiple network multiple image batch semantic segmentation time T_LowN at minimum_batch、N_CNN-GPUA value;

d, scheduling, detecting and identifying the cooperative scheduling task model to obtain the total detection and identification time T_inspectAnd executing the sequence under the minimum value to finish the detection and identification cooperative scheduling.

One or more embodiments of the present invention may have the following advantages over the prior art:

the method provided by the invention reasonably schedules and processes a large amount of calculation overhead and storage overhead in the semantic segmentation process, intelligently improves the real-time capability of machine vision detection and identification, and has the core functions of: firstly, a high-resolution image processing process is intelligently scheduled, and the storage overhead is reduced; secondly, a large number of image processing processes are intelligently scheduled, and the computing resource efficiency and the real-time capability are improved; and thirdly, all key modules of the machine vision system work cooperatively.

Drawings

FIG. 1 is a flow chart of a multitask cooperative scheduling method for online semantic segmentation machine vision inspection;

FIGS. 2a and 2b are example 1 detection of an identified co-scheduled DAG model;

FIGS. 3a and 3b are graphs of optimal demodulation degrees of detection discrimination co-scheduling in embodiment 1;

FIG. 4 is a ticket anti-counterfeiting authentication coordinated scheduling DAG model of example 2;

fig. 5a and 5b are gantt charts of the best solution execution of the bill anti-counterfeiting authentication scheduling in embodiment 2.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings.

As shown in fig. 1, the method is a multitask cooperative scheduling method flow oriented to online semantic segmentation machine vision detection, and includes the following steps:

step 10, establishing a detection and identification cooperative scheduling task model, wherein the tasks comprise positioning activities, light source control activities, camera imaging activities, semantic segmentation activities and the like, and the semantic segmentation activities comprise: partitioning, scheduling and partitioning sub-activities in blocks, and scheduling sub-activities in batches and partitioning;

step 20, high resolution image block semantic segmentation task scheduling, dividing the image into N_subIndividual resolution U_sub×V_subScheduling the target to obtain the high resolution image semantic segmentation time T_HighU at minimum_sub、V_sub、N_subA value;

step 30, scheduling multi-network multi-image batch semantic segmentation tasks, and packaging the multi-images into N_batchEach group of images and GPU are loaded with N_CNN-GPUThe semantic segmentation network models are subjected to batch parallel processing, and the scheduling objective is to obtain the multi-network multi-image batch semantic segmentation time T_LowN at minimum_batch、 N_CNN-GPUA value;

step 40, scheduling, detecting and identifying the cooperative scheduling task model to obtain the total detection and identification time T_inspectAnd executing the sequence under the minimum value to finish the detection and identification cooperative scheduling.

The step 10 specifically includes: establishing a detection and identification cooperative scheduling task model, wherein the activity tasks of the model comprise: start Activity S, locate Activity P_{Mov_i_j}Light source control activity P_{Light_i_j}Image acquisition activity P_{Acq_i_j}Semantic segmentation activity G_{Seg_i_j}. Semantic segmentation activity G_{Seg_i_j}The method comprises the following steps: block scheduling partition subactivity P_{High_i_j_m}Batch scheduling sub-activity partition P_{Low_i_j}。

In the event and sub-event, the subscripts i and j represent the j imaging condition of the i-th detection object, and the number of the detection objects is set to be N_itemDetecting the condition number N_condI is 1, … N_itemJ has a value range of 1, … N_cond. Subscript m represents the mth chunking scheduled segmentation activity, set to the high resolution image number N_highWhen m is 1, … N_high。

Table 1 is an activity parameter table of semantic segmentation machine vision inspection identification cooperative scheduling model, and it can be seen that the semantic segmentation machine vision inspection identification cooperative scheduling model can enable G_{Seg_i_j}With the next P_{Mov_i_j+1}、 P_{Light_i_j+1}、P_{Acq_i_j}And (5) performing cooperative parallel processing.

TABLE 1 semantic segmentation machine vision inspection identification cooperative scheduling model activity parameter table

The step 20 specifically includes: dividing an image into N_subResolution U_sub×V_subAspect ratio not exceeding gamma_scaleThe sub-image and high-resolution image blocking scheduling optimization mathematical model is as follows:

the physical significance of the mathematical model is: sub-image segmentation video memory occupation under constraint condition The length-width ratio of the subimages is less than or equal to gamma_scaleCalculating the total division timeU at minimum_sub、V_sub、N_subThe value is obtained.

The high-resolution image blocking semantic segmentation task scheduling solving method comprises the following steps:

for the online semantic segmentation machine vision detection system, a semantic segmentation network model f is used_CNNVideo memory occupation M of (DEG) divided image_segVideo memory occupation M including deep learning framework_frameworkModel display occupation M_CNNImage display memory occupation M_imgM is occupied by characteristic mapping video memory_tensorAnd the like. Through f_CNN(. cut) various image pixel numbers p_allImage, calibrationIs fitted outThen:

thus, video memory occupancy estimationEstimation of upper limit of pixel quantityRespectively as follows:

substituting the formula (3) into the formula (1) to obtain a high-resolution image block scheduling optimization mathematical model expression:

the step 30 specifically includes:

packing multiple images into N_batchEach group of images and GPU are loaded with N_CNN-GPUThe semantic segmentation network model performs batch parallel processing, and the multi-network multi-image batch scheduling mathematical model is as follows:

the physical significance of the mathematical model is as follows: image segmentation video memory occupation under constraint conditions To findN at minimum_batch、N_CNN-GPUThe value is obtained.

Scheduling a multi-network multi-image batch semantic segmentation task, wherein the solving method comprises the following steps:

by different N_CNN-GPUDifferent N_batchMultiple networks f_CNN(. 2) segmenting the multiple images, scalingIs fitted outThen:

thus, video memory occupancy estimationComprises the following steps:

M_GPUlower number of pixels p_allUpper limit of number of image batchesUpper limit of number of semantically partitioned networksRespectively as follows:

substituting the formula (7), the formula (8) and the formula (9) into the formula (5) to obtain a multi-network multi-image batch scheduling optimization mathematical model expression:

the step 40 specifically includes:

detection and identification total time T after detection and identification cooperative scheduling for online semantic segmentation machine vision detection_inspectComprises the following steps:

T_{Mov_i_j}、T_{Light_i_j}、T_{Acq_i_j}、T_{Seg_i_j}respectively as nodes P of a cooperative scheduling DAG model_{Mov_i_j}、P_{Light_i_j}、P_{Acq_i_j}、G_{Seg_i_j}And (4) solving the collaborative scheduling DAG model by adopting a longest path priority algorithm according to the weight value to obtain an execution sequence, thereby realizing scheduling.

Example 1

Example 1 is implemented in detection discrimination co-scheduling for online semantic segmentation machine vision inspection. The automatic detection ATX standard machine case comprises a full-automatic detection device, a full-automatic detection system and a full-automatic detection system. The concrete requirements are as follows: detecting the whole front panel and the whole back panel (the width is 185 mm multiplied by 420mm) of the chassis, wherein the positioning error of the interface and the key is less than or equal to 0.5mm, and the positioning error of the standard component is less than or equal to 0.2 mm; identifying more than 20 types of assembly parts, wherein the detection time of more than 50 cases with detection parts is less than or equal to 8 s; and thirdly, outputting the position and boundary information of the assembly points with the conditions of missing assembly and error assembly.

The length-width ratio of the front panel and the rear panel of the case is 420/185 ≈ 2.27, the scheme of the machine vision detection system is that 3 industrial cameras are matched with a pneumatic mechanism, images are collected at different positions in the long side direction of the front panel and the rear panel, and the detection of the whole front panel and the rear panel is completed. The camera imaging visual field is 280 multiplied by 200mm, and images are respectively formed on the upper side and the lower side of the panel for 1 time by matching with a pneumatic mechanism, so that images of the whole front panel and the whole rear panel can be obtained; by adopting a Haikang MV-CH250-90GM area array industrial camera, the resolution U multiplied by V is 5120 multiplied by 3840, the pixel precision is approximately equal to 0.0547mm, and the positioning error of the algorithm can meet the requirement that the positioning error is less than or equal to 0.2mm without exceeding 3.65 pixels; and considering that the type of the identified assembly parts is more, the positioning precision is high, and the online real-time requirement is met, and a Pythrch deep learning frame and a Mask R-CNN segmentation model are adopted. The server adopts an AMD64 architecture server which comprises 2 GeForce GTX 1080Ti GPUs, 1 8-core 16 threads Intel i7-7820X to a strong CPU. The hardware parameter function of the case standard component assembly quality detection system is shown in table 2.

TABLE 2 hardware parameter function table of online case standard component assembly quality detection system

The detection system meets the precondition of detecting and identifying the cooperative scheduling model and can be abstracted into the imaging condition number N_cond2, high resolution image number N_highThe 2 test identifies the co-scheduling model, with the activity or function parameters as shown in table 2. Video memory occupancy estimation for 5120 × 3840 image segmentation in resolution U × V in table The method has the problem of high-resolution image segmentation, and the segmentation is recorded as P by adopting high-resolution image blocking scheduling_{High_i_j_1、}P_{High_i_j_2}Time of day8 frames of 640 multiplied by 480 images are subjected to semantic segmentation, the problem of multi-network multi-image batch scheduling exists, and the segmentation of the multi-network multi-image batch scheduling is marked as P by adopting an algorithm 5-2_{Low_i_j}Dividing timeSemantic segmentation activity G_{Seg_i_j}Execution time

TABLE 3 Online chassis standard component assembly quality detection cooperative scheduling model activity or function parameter table

Machine vision using online-oriented semantic segmentationThe detection and identification cooperative scheduling implementation algorithm for detection can respectively implement positioning-light source control cooperation and semantic segmentation-positioning-light source control-image acquisition cooperation. FIG. 2a, FIG. 2b, FIG. 3a, and FIG. 3b are example 5-6 DAG model and optimal demodulation degree Gantt graph, respectively, wherein FIG. 2a is a single detection object DAG model (N)_itemFig. 2b is a small lot detection DAG model (N) ═ 1)_item5); FIG. 3a is N_item1, 8-Activity Co-scheduling optimal solution (T)_inspect4.478s), fig. 3b is N_item5, 40 Activity Co-Schedule optimal solution (T)_inspect24.478 s); in the DAG model, the edge weights are the activity execution times. It can be seen that, using the present invention, a single test object N is detected_itemTotal time T for detection and discrimination of 1_inspec4.478s, 1.467s (obtained from 5.945s-4.478 s) less than the basic scheduling model, 24.67% shorterSmall batch detection of N_itemTotal time T for detection and identification of 5_inspectCompared with the basic scheduling model, the method reduces 13.247s (obtained from 37.725s-24.478 s) and shortens 35.11 percentThe scheduling optimization effect is obvious, and the requirement that the detection time of a single case is less than or equal to 8s can be met.

Example 2

Detection and discrimination cooperative scheduling implementation example 2 of online semantic segmentation machine vision detection. The development of bill anti-counterfeiting discriminator requires the realization of intelligent key anti-counterfeiting characteristic recognition of legal bills such as checks, drafts and the like. The concrete requirements are as follows: detecting 22 key anti-counterfeiting features such as watermarks, fluorescent main patterns, logos and the like on the complete ticket under 4 illumination excitation conditions such as white light, backlight, infrared light, ultraviolet light and the like; detecting micro anti-counterfeiting characteristics such as micro characters (1mm multiplied by 1mm) on the money column, anti-Stokes luminescence of ticket number positions (the line width is about 0.1mm) and the like; comprehensively identifying the consistency of the bill type and the miniature characters, the bill number, the watermark, the size and the main pattern, and the consistency of the bank-bank logo and the bill number; and fourthly, the detection time of the single check and the money order is less than or equal to 1 s.

Considering that the anti-counterfeiting feature on the bill has larger scale difference, setting 1 panoramic camera and 2 microscopic cameras for visual sensing, wherein the visual field (100 multiplied by 180mm) of the panoramic cameras collects images of complete checks and bills of drafts; 2 micro visual senses respectively identify the anti-counterfeiting characteristics of the miniature characters (1mm multiplied by 1mm) and the anti-Stokes luminescence (the line width is about 0.1 mm). The illumination excitation conditions include 4 (N) of white light, backlight, infrared light, ultraviolet light, etc_cond4). The specific hardware parameter functions are shown in table 4.

TABLE 4 hardware parameter function table for bill anti-fake discriminator

Based on the basic structure block diagram, the detection system meets the precondition of detecting and identifying the cooperative scheduling model and can be abstracted into the imaging condition number N_condNumber of detected objects N4_condThe test 1 identifies the co-scheduling model, with the activity or function parameters as shown in table 5.

In table P_{Low_i_j}For 3 frames 640 x 480 image semantic segmentation, multi-network multi-image batch scheduling is carried out, and the time is segmentedSemantic segmentation activity G_{Seg_i_j}Execution timePosition P_{Mov_i_j}Light source control P_{Light_i_j}Image acquisition P_{Acq_i_j}Activity execution time T_{Mov_i_j}、T_{Light_i_j}、T_{Acq_i_j}Obtained by measuring the actual value. The method can respectively realize ticket entrance-light source control, semantic segmentation-light source control cooperation and semantic segmentation-ticket output cooperation. Available table 5 is a table of activity function parameters of the DAG model for detecting and identifying the coordinated dispatching by the bill anti-counterfeiting identifier.

TABLE 5 Bill anti-counterfeit discriminator table for detecting and discriminating activity function of coordinated dispatching DAG model

FIG. 4, FIG. 5a, and FIG. 5b are respectively a DAG model and an optimal demodulation Gantt chart for the ticket anti-counterfeit authentication coordinated scheduling in application example 2, and T in FIG. 5a_inspect1.159s, T in FIG. 5b_inspect0.850 s. After the technology of the invention is used, a single bill N_item＝1、N_condTotal time T of detection and identification under the condition of 4 light source excitation_inspect0.850s, a reduction of 0.309s (from 1.159s to 0.850 s) from the basic scheduling model, a 26.65% reduction in T_inspectThe scheduling optimization effect is obvious, and the requirement that the detection time of a single bill is less than or equal to 1s can be met.

Although the embodiments of the present invention have been described above, the above descriptions are only for the convenience of understanding the present invention, and are not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.