阅读说明：本技术 一种整体与部件目标检测相结合的非约束车牌精准定位方法 (Whole and component target detection combined non-constrained license plate accurate positioning method ) 是由 徐光柱 刘高飞 匡婉 万秋波 刘鸣 雷帮军 石勇涛 于 2021-07-21 设计创作，主要内容包括：一种整体与部件目标检测相结合的非约束车牌精准定位方法,首先利用YOLOv3算法同步检测以车牌顶点为中心的4类左上、右上、右下、左下顶点目标区域和车牌区域,然后通过定位车牌顶点区域间接实现对车牌顶点位置的预测,并结合忽略类别的非极大值抑制算法CF-NMS、顶点区域归类、单一缺失顶点预测的后处理操作获取准确的车牌区域。最后,为进一步提升整体与部件目标检测相结合的车牌定位算法的性能,将多模型融合、粗定位与精定位结合等策略融入定位算法中,并经过实验验证了策略的有效性。本发明设计的整体与部件目标检测相结合的车牌定位算法,在不改变YOLOv3模型结构,同时不增加额外计算量的基础上实现了车牌顶点目标的检测,实现车牌精准定位。(A method for accurately positioning an unconstrained license plate by integrally and integrally detecting a component target comprises the steps of synchronously detecting 4 types of target areas with top left, top right, bottom right and bottom left vertexes and license plate areas by using a YOLOv3 algorithm, indirectly predicting the top point position of the license plate by positioning the top point area of the license plate, and acquiring an accurate license plate area by combining post-processing operations of a non-maximum suppression algorithm CF _ NMS for neglecting categories, top point area classification and single missing top point prediction. Finally, in order to further improve the performance of the license plate positioning algorithm combining the whole and the part target detection, strategies such as multi-model fusion, combination of coarse positioning and fine positioning and the like are fused into the positioning algorithm, and the effectiveness of the strategies is verified through experiments. The license plate positioning algorithm combining the whole detection and the component target detection realizes the detection of the top target of the license plate and the accurate positioning of the license plate on the basis of not changing the structure of a YOLOv3 model and not increasing extra calculation amount.)

1. A method for accurately positioning an unconstrained license plate by integrally and integrally detecting a component target is characterized by comprising the following steps of: synchronously detecting a target area taking the top point of the license plate as the center and an integral license plate area by using a YOLOv3 target detection algorithm; the predication of the top point position of the license plate is indirectly realized by positioning the top point area of the license plate; then, combining with post-processing operations such as a non-maximum suppression algorithm CF _ NMS for neglecting categories, license plate vertex region classification, single missing vertex prediction and the like, realizing accurate positioning of a license plate region; and for license plates which cannot be accurately positioned, directly reserving an external rectangular area of the license plates.

2. An accurate positioning method for an unconstrained license plate by integrally combining part target detection is characterized by comprising the following steps:

step 1: synchronously detecting a license plate vertex region and an integral license plate region by using a YOLOv3 target detection algorithm model;

step 2: and processing the model detection result by using CF _ NMS, license plate vertex region classification and single missing vertex prediction so as to obtain an accurate license plate region.

3. The method for accurately positioning the unconstrained license plate of the integrated and component target detection system as claimed in claim 2, wherein: the method also comprises a fusion strategy for fusing the multi-model positioning result, and the method comprises the following specific steps:

step 1): the method comprises the steps that target boundary frame information of license plate vertex areas detected by two YOLOv3 target detection models with different input sizes is reserved in a set D, all target boundary frames only contain central points, width and height and category information of rectangular frames, and when model output results are fused, all license plate vertex areas are treated as targets of the same type;

step 2): establishing an empty set B, randomly taking out a license plate vertex region boundary box in the set D and putting the license plate vertex region boundary box into the set B because confidence information does not exist at the moment, solving IoU between the rest vertex region boundary boxes in the set D and the rest vertex region boundary boxes in the set D, and deleting the vertex region boundary box information of IoU >0.45 from the set D as a formula (1);

step 3): repeating the step 2) until the number of the vertex areas contained in the set D is zero, and finally obtaining all the vertex area bounding boxes in the set B, namely the result of vertex area fusion;

and 4) fusing license plate region detection frames detected by two YOLOv3 target detection models with different input sizes in the same manner, and reserving the finally obtained license plate vertex region and license plate region fusion results.

4. The method for accurately positioning the unconstrained license plate of the integrated and component target detection system as claimed in claim 2, wherein: in the step 1, the sizes of 4 top target areas of the license plate are set in a self-adaptive mode:

A. b, C, D are respectively a license plate right lower vertex, a license plate left upper vertex and a license plate right upper vertex, and the 4 square frames are respectively license plate vertex target areas corresponding to the 4 vertexes;

the side length of the target area of the top point of the license plate is related to the size of the license plate, the side length of the square area corresponding to the upper left top point C and the lower left top point B is the same, and the length is 2h_leftThe side lengths of the areas corresponding to the upper right vertex D and the lower right vertex A are the same, and the length is 2h_rightWherein: h is_left、h_rightThe difference between the heights of BC and AD,

the 4 vertex areas of the 4 license plates are intercepted by using the method, each type of vertex areas of the license plates have similarity, the areas containing the license plates are all positioned in the same right-angle direction, and the rest directions are background information; and finally, taking the external rectangular region of the license plate as a class and taking the external rectangular region and the vertex region of the license plate as a target region for model training.

5. The method for accurately positioning the unconstrained license plate of the integrated and component target detection system as claimed in claim 2, wherein: in step 2, the ignoring category non-maximum suppression algorithm CF _ NMS comprises the following steps:

step 2.1: setting all the license plate vertex region types detected in the image as uniform values;

step 2.2: putting the bounding box information of all license plate vertex areas into a set B, and establishing an empty set D for storing bounding boxes to be reserved;

step 2.3: taking out the boundary box with the maximum confidence from the set B, adding the boundary box with the maximum confidence into an empty set D, deleting the information of the boundary box from the set B, performing IoU operation shown in formula (1) on the taken-out boundary box and all other boundary boxes in the set B, and then deleting all target boundary boxes of which the IoU is greater than a specified threshold of 0.7 from the set B;

step 2.4: and (4) repeating the step 2.3 until the set B is empty, wherein the finally obtained set D contains all license plate vertex region bounding boxes needing to be reserved.

6. The method for accurately positioning the license plate of the non-constraint type integrated with the detection of the target of the component as claimed in claim 2 or 5, wherein the method comprises the following steps: in the step 2, the license plate vertex region classification method comprises the following steps:

step 2.5: respectively putting a license plate vertex region target and a license plate region target which are obtained after CF _ NMS processing into a set B1 and a set B2, and establishing an empty set D1;

step 2.6: establishing an empty set D0, taking any license plate region from B2, storing the information in D0 and deleting the information from B2, taking all vertex region bounding boxes intersected with the vertex region bounding boxes from B1, storing the information in D0 and deleting the information from B1;

step 2.7: judging whether the number of the bounding boxes in D0 is greater than 3, namely, keeping the condition that the number of the detected license plate vertex areas is 3, 4 or more, because when the number of the vertex areas is less than 3, accurate license plate areas cannot be obtained; if the number of the vertex areas is more than or equal to 3, adding D0 into D1, otherwise, not performing any treatment;

step 2.8: repeating steps 2.6 and 2.7 until either B1 or B2 is empty;

step 2.9: if the B2 is empty and the number of the remaining license plate vertex areas in the B1 is more than 3, the vertex area information is stored in an empty set D2, D2 is added into D1, and otherwise, no processing is carried out.

7. The method for accurately positioning the unconstrained license plate of the integrated and component target detection system according to claim 5, wherein the method comprises the following steps: after the license plate vertex region classification step, the corresponding conditions of the existing license plate region and the license plate vertex region are divided into 3 types: 1 license plate region and 3 corresponding vertex regions, 1 license plate region and 4 or more corresponding vertex regions, 0 license plate region and 4 or more vertex regions;

for the case of 4 vertex region determination, 4 bounding box centers can be directly obtained;

for the condition that the number of the vertex areas exceeds 4, because the vertex areas contain errors, the centers of the bounding boxes of the 4 license plate areas can be randomly selected from the vertex areas, and then the license plate areas are connected according to a certain sequence to obtain accurate license plate areas;

for the case of only 3 vertex regions, the missing vertex positions can be predicted in combination with the corresponding license plate regions.

8. The method for accurately positioning the license plate of the non-constraint combined with the detection of the target of the component as a whole according to claim 7, wherein the method comprises the following steps: obtaining 4 vertex coordinates of a corresponding license plate in the image, and connecting the 4 vertexes according to a certain sequence to obtain a license plate region, wherein the specific method comprises the following steps:

firstly, finding out a vertex A positioned at the top end, then calculating included angles alpha, beta and gamma between connecting lines of the vertex and other three vertexes and a horizontal rightward line segment of the vertex, wherein the vertexes with the minimum and the maximum corresponding included angles are adjacent to the vertex A, respectively recording the vertexes as a point B and a point D, recording the rest vertexes as a point C, and finally connecting the vertexes according to the sequence of A-B-C-D-A to obtain the license plate region.

Technical Field

The invention relates to the technical field of image processing, in particular to an accurate positioning method for an unconstrained license plate by integrally and integrally detecting a component target.

Background

In recent years, license plate recognition systems in fixed scenes such as high-speed toll stations, parking lots, cell entrances and exits have been in mature commercial application, but the effect of the license plate recognition systems on complex license plate images shot by various mobile devices such as mobile phones is still to be improved. An accurate license plate detection method is a premise of correctly identifying license plates, and the existing license plate detection method adopts a rectangular frame to mark license plates, as shown in fig. 1(a), a large number of background redundant areas are often introduced, so that difficulty is caused in license plate number identification. As shown in fig. 1(b), the license plate under the non-ideal environment is accurately selected, so that the accuracy of subsequent license plate identification can be effectively improved, and the method is the most critical step in the problem of license plate identification under the non-ideal environment. The existing license plate detection methods are mainly divided into two types, one is a traditional method based on image processing, and the other is a method based on deep learning.

The traditional method based on image processing generally utilizes information such as manually set image edges, colors, textures and the like to detect the license plate. The method comprises the following steps of document [1] but bin, Meiwenhao, Wuqian and the like, a distorted license plate positioning and correcting method research [ J ] manufacturing automation, a Hough transformation method recorded in 2019,41(3):7-11, four edges of a license plate are found by detecting straight lines in an image, and then four edge intersections are used for obtaining the vertexes of the license plate to achieve accurate license plate positioning. However, the method has poor detection effect under the conditions of small picture and fuzzy edge.

Document [2] license plate positioning method combining macro photo, thank-exing, luhain.color and edge texture [ J ]. computer hardware: the existing generation of electronic technology, 2018,41(21):67-70. the color characteristic method recorded in the method utilizes the characteristic that most license plates are blue bottom and yellow bottom, divides the license plate area according to the fixed color matching of the license plate background and characters, and then obtains the accurate license plate area by utilizing a mathematical morphology method. However, under the condition that the license plate reflects light or a traffic signboard similar to the license plate appears in the picture, the phenomenon of false detection is easy to occur.

The method is characterized in that a license plate positioning algorithm [ J ] based on MSER and edge projection is adopted in a document [3] ZhengGuilin, Wuhuang Zi Mulberry, a license plate positioning algorithm [ J ] based on MSER and edge projection, 2019,40(1): 241-.

From the current state of research, there are the following limitations based on the conventional image processing method: firstly, the characteristics set manually are relatively single, and good effects are difficult to realize under various conditions when the system is used for non-constrained scenes. Secondly, for scenes with small license plates, blurriness and large inclination degree, the traditional method is difficult to process. And thirdly, when areas such as traffic boards and advertising boards similar to license plates appear in the scene, the traditional method is easily interfered and false detection occurs.

In recent years, a Deep Convolutional Neural Network (DCNN) is developed at a high speed, a strong multi-level feature extraction capability of the DCNN is widely applied to the field of target detection, and the existing license plate detection method based on deep learning can be divided into a single-stage method and a multi-stage method according to detection steps. The single-stage method refers to directly predicting the license plate area through a network model.

Document [4] Tian Y, Lu X, Li W X. license plate detection and localization in complex scenes based on depletion learning [ A ].2018 Chinese Control and precision Conference (CCDC) [ A ]. Los Alamines: IEEE Computer Society Press,2018:6569 plus 6574. However, when the rotation angle is large or the distance between the license plate and the camera is long, the positioning effect is still to be improved.

The document [5] Xu Z B, Yang W, Meng A J, et al, Towards end-to-end license plate detection and recognition, A large database and base A, European conference on computer vision C, Heidelberg, Springer,2018: 261-. In order to verify the network performance, authors construct a Chinese city parking data set (CCPD) shot in multiple scenes, and the effect is superior to that of some common target detection networks.

Document [6] Xie L L, Ahmad T, Jin L W, et al.A new CNN-based method for multi-directional card license plate detection [ J ]. IEEE Transactions on Intelligent Transportation Systems,2018, 19(2): 507-.

Document [7] chinese patent application No.: 202010225652.0, designing a depth neural network license plate positioning method based on image enhancement in a complex environment, establishing a full convolution neural network as a license plate detection network, then performing image enhancement on license plates in fuzzy images such as early morning, dusk and fog in various complex environments, and improving the license plate detection accuracy of the whole model. However, the method only helps to roughly position the blurred license plate, and does not solve the problem of accurate positioning of the inclined license plate.

The multi-stage method is to locate the license plate through a plurality of network models, namely, a license plate candidate area is determined firstly, and then the license plate is located in the area. The document [8] He K M, Gkioxari G, Doll' R P, et al.Mask R-CNN [ J ]. IEEE Transactions on Pattern Analysis and Machine understanding, 2020,42(2):386-397 ] proposes a multi-stage license plate detection method based on a Mask-RCNN structure, firstly extracts image characteristics by using a convolution module similar to GoogleNet, and the module promotes the detection capability of fine-grained characteristics while ensuring the calculation speed; then inputting the extracted characteristic diagram into a Mask-RCNN network which does not comprise a segmentation step, acquiring a license plate candidate region in the image, setting 12 groups of anchors according to the size and length-width ratio of a license plate for adapting to license plate detection, and outputting two branched information by the network by using a full connection layer, namely license plate and non-license plate classification and bounding box regression; and finally, filtering license plate and non-license plate areas, using a RoI-Align layer in Mask-RCNN, and setting the size of the pooling layer to be 8 multiplied by 7, so as to reduce the false detection condition of the license plate as much as possible. The method achieves higher detection precision on a plurality of public license plate data sets, but the condition of missing detection exists on license plate images with over-bright or over-dark light rays and images containing a plurality of license plates.

Document [9] Han J, Yao J, ZHao J, et al, Multi-oriented and scale-invariant license plate detection based on volumetric neural networks [ J ]. Sensors,2019,19(5): 1175-. Its network consists of two sub-networks: (1) the RPN sub-network is used for acquiring a license plate candidate region; (2) and the detection sub-network is used for determining a positive sample candidate region and obtaining a license plate region in a regression mode, and the two sub-networks share a special diagnosis extraction network layer constructed based on the Faster R-CNN. The method utilizes a parallelogram to position the license plate, and realizes the detection of the license plate with large scale span by combining the multi-output layer candidate region extraction strategy and the feature fusion strategy of the anchor frame, which is superior to the prior method in the aspects of the detection of the license plate with different directions and multiple scales, but has false detection and missing detection on smaller license plate targets.

Document [10] chinese patent application No.: 202010223731.8, designing a license plate positioning and recognition method under an unconstrained condition, firstly performing data enhancement on data to be detected, then inputting the data into a trained license plate detection model YOLOv3-tiny to obtain the approximate position of a license plate, and finally inputting the approximate position into a regression network to obtain a vertex coordinate and perform perspective correction at the same time to obtain a license plate region without redundancy. However, the method has poor effect on the license plates with larger inclination degree and the license plates with more fuzziness.

Disclosure of Invention

The invention provides an unconstrained license plate accurate positioning method combining integral and component target detection, which realizes the detection of a license plate vertex target and accurately positions a license plate region by fully utilizing the excellent performance of a YOLOv3 network on the basis of not changing the structure of a YOLOv3 model and not increasing extra calculation amount.

The technical scheme adopted by the invention is as follows:

a method for accurately positioning an integral license plate by combining the detection of a component target is characterized in that a part target area taking a license plate vertex as a center, namely a license plate vertex area and an integral license plate area are synchronously detected by a YOLOv3 target detection algorithm; the predication of the top point position of the license plate is indirectly realized by positioning the top point area of the license plate; then, combining with post-processing operations such as a non-maximum suppression algorithm CF _ NMS for neglecting categories, license plate vertex region classification, single missing vertex prediction and the like, realizing accurate positioning of a license plate region; and for license plates which cannot be accurately positioned, directly reserving an external rectangular area of the license plates.

A method for accurately positioning an unconstrained license plate by integrally and integrally detecting a component target comprises the following steps:

step 1: synchronously detecting a target area and a license plate area which take the top point of the license plate as the center by using a YOLOv3 target detection algorithm model;

step 2: and processing the model detection result by using CF _ NMS, license plate vertex region classification and single missing vertex prediction so as to obtain an accurate license plate region.

The method also comprises a fusion strategy for fusing the multi-model positioning result, and the method comprises the following specific steps:

step 1): and (3) reserving the target boundary frame information of the license plate vertex areas detected by two YOLOv3 target detection models with different input sizes into a set D, wherein all the target boundary frames only contain the central point, width and height and category information of a rectangular frame, and when the output results of the models are fused, all the license plate vertex areas are treated as the same type of targets.

Step 2): and (4) establishing an empty set B, randomly taking one license plate vertex region boundary box in the set D and putting the license plate vertex region boundary box into the set B because confidence coefficient information does not exist at the moment, solving IoU between the rest vertex region boundary boxes in the set D and deleting IoU >0.45 of vertex region boundary box information from the set D.

Step 3): and repeating the step 2) until the number of the vertex areas contained in the set D is zero, and finally obtaining all the vertex area bounding boxes in the set B, namely the result of vertex area fusion.

And 4) fusing the license plate region detection frames detected by the two YOLOv3 target detection models with different input sizes in the same manner, and reserving the finally obtained license plate vertex region and license plate region fusion results.

A. B, C, D are respectively a license plate right lower vertex, a license plate left upper vertex and a license plate right upper vertex, and the 4 square frames are respectively vertex areas corresponding to the 4 vertices, namely target areas for model training;

the side length of the target area is related to the size of the license plate, the side length of the square area corresponding to the upper left vertex C and the lower left vertex B is the same, and the length is 2h_leftThe side lengths of the areas corresponding to the upper right vertex D and the lower right vertex A are the same, and the length is 2h_rightWherein: h is_left、 h_rightThe difference between the heights of BC and AD,

the 4 vertex areas of the license plate are intercepted by using the method, each type of vertex area of the license plate has similarity, the areas containing the license plate are all positioned at the same right-angle position, and the rest positions are background information; and finally, taking the external rectangular region of the license plate as a class and taking the external rectangular region and the vertex region of the license plate as a target region for model training.

In step 2, the ignoring category non-maximum suppression algorithm CF _ NMS comprises the following steps:

step 2.1: setting all the license plate vertex region types detected in the image as uniform values;

step 2.2: and putting the information of the bounding boxes of the vertex areas of all the license plates into the set B, and establishing an empty set D for storing the bounding boxes to be reserved.

Step 2.3: taking out the boundary box with the maximum confidence from the set B, adding the boundary box with the maximum confidence into an empty set D, deleting the information of the boundary box from the set B, performing IoU operation shown in formula (1) on the taken-out boundary box and all other boundary boxes in the set B, and then deleting all target boundary boxes of which the IoU is greater than a specified threshold of 0.7 from the set B;

step 2.4: and (4) repeating the step 2.3 until the set B is empty, and finally obtaining a set D, namely a bounding box containing all the license plate vertex areas needing to be reserved.

In the step 2, the license plate vertex region classification method comprises the following steps:

step 2.5: and respectively putting the vertex region target and the license plate region target obtained after the CF _ NMS processing into a set B1 and a set B2, and establishing an empty set D1.

Step 2.6: an empty set D0 is created, any license plate region is taken from B2, its information is saved in D0 and deleted from B2, all vertex region bounding boxes that intersect it are taken from B1, their information is saved in D0 and deleted from B1.

Step 2.7: and D0, judging whether the number of the bounding boxes is more than 3, namely, keeping the condition that the number of the detected license plate vertex areas is 3, 4 or more, because when the number of the vertex areas is less than 3, accurate license plate areas cannot be obtained. If the number of the vertex areas is more than or equal to 3, adding D0 into D1, otherwise, not performing any processing.

Step 2.8: repeat step 2.6 and step 2.7 until either B1 or B2 is empty.

Step 2.9: if the B2 is empty and the number of the remaining license plate vertex areas in the B1 is more than 3, the vertex area information is stored in an empty set D2, D2 is added into D1, and otherwise, no processing is performed.

In the step 2, after the license plate vertex region classification step, the corresponding conditions of the existing license plate region and the license plate vertex region are divided into 3 types: 1 license plate region and 3 corresponding vertex regions, 1 license plate region and 4 or more corresponding vertex regions, 0 license plate region and 4 or more vertex regions;

for the case of 4 vertex region determination, 4 bounding box centers can be directly obtained;

for the condition that the number of the vertex areas exceeds 4, because the vertex areas contain errors, the centers of the bounding boxes of the 4 license plate areas can be randomly selected from the vertex areas, and then the license plate areas are connected according to a certain sequence to obtain an accurate license plate area;

for the case of only 3 vertex regions, the missing vertex positions can be predicted in combination with the corresponding license plate regions.

In the step 2, 4 vertex coordinates of a corresponding license plate in the image are obtained, and the license plate region can be obtained by connecting the 4 vertices according to a certain sequence, wherein the specific method comprises the following steps:

the method comprises the steps of firstly finding out a vertex A positioned at the uppermost end, then calculating included angles alpha, beta and gamma between connecting lines of the vertex and other three vertexes and a horizontal rightward line segment of the vertex, wherein the vertex with the minimum and the maximum corresponding included angle degrees is adjacent to the vertex A and is respectively marked as a point B and a point D, the rest vertexes are marked as a point C, and finally connecting the vertexes according to the sequence of A-B-C-D-A to obtain the license plate area.

The invention discloses an accurate positioning method of an unconstrained license plate by integrally and integrally detecting a component target, which has the following technical effects: 1) The most common commercial license plate recognition system at present is sensitive to scene change, and the problems of license plate missing detection, false detection, inaccurate marking and the like exist when the license plate recognition problem in a real non-ideal environment is solved. The main reasons are that the license plate is small in size in a non-constrained environment and the conditions of inclination, blurring, damage and the like exist, and the current model is difficult to achieve a good positioning effect on the license plate under various conditions. The invention designs a single-stage license plate positioning algorithm combining integral and component target detection, and can acquire a minimum external quadrilateral region to accurately position a license plate.

2) Most license plate targets are regarded as an integral target in the existing license plate positioning method, and the target is positioned by using a rectangular frame. In order to obtain accurate positioning, a cascade regression network is generally adopted behind a target detection network to further detect the license plate vertex. The method is cascaded with more networks, so that the calculation load is increased on one hand, and the method is only suitable for detecting the serial mode of a single license plate target on the other hand. The invention integrates the whole license plate target positioning and the license plate vertex detection under a target detection frame, realizes the detection of the whole license plate region and the license plate vertex region, and realizes the accurate license plate positioning under various complex conditions through a subsequent fusion strategy. The license plate positioning method based on the combination of component target detection skillfully utilizes the rectangular detection frame to detect the vertex area (component target) of the license plate, and realizes the detection of the vertex target of the license plate on the basis of not changing the structure of the YOLOv3 model and not increasing extra calculation amount.

3) In the existing general target detection algorithm, YOLOv3 is taken as a typical end-to-end algorithm, high detection precision is realized while a high detection speed is maintained, but a small amount of missing detection phenomenon still exists in an extremely complex scene. Aiming at the problem of missing detection of a small number of vertexes in vertex detection, the invention designs a single missing vertex prediction method which can effectively solve the problem of single vertex missing.

4) In the invention, for the vehicle pictures shot in the non-ideal environment such as vehicles in an outdoor parking lot, moving vehicles under a high-position camera and the like shot by the handheld equipment, because the shooting direction has unfixed property, the license plate target in the images has different inclination angles, and the target area obtained by using a general target detection algorithm contains more redundant background information. In order to realize accurate identification of license plate numbers subsequently, the invention designs a license plate positioning method combining whole and part target detection. The method comprises the steps of synchronously detecting a region taking a license plate vertex as a center and a license plate region through a YOLOv3 target detection algorithm, indirectly obtaining the license plate vertex, and then combining post-processing operations such as CF _ NMS, license plate vertex region classification, single missing vertex prediction and the like to realize accurate license plate region positioning. The license plate positioning method combining whole and part target detection designed by the invention does not add extra calculation amount on the original YOLOv3 model, and can obtain the four vertex positions of the license plate on the basis of inheriting the excellent performance of YOLOv3, thereby realizing the accurate positioning of the license plate.

5) The invention provides a license plate single missing vertex prediction method which can effectively solve the problem of single missing detection of a target detection network in a complex scene. The position of the fourth vertex is deduced through the detected information of the 3 license plate vertexes of the license plate, the calculation process is simple, and the reliability is high.

Drawings

FIG. 1(a) is a schematic diagram of coarse positioning of a license plate;

fig. 1(b) is a schematic diagram of the precise positioning of the license plate.

Fig. 2 is a network structure diagram of YOLOv 3.

Fig. 3 is a flow chart of the accurate license plate positioning.

FIG. 4 is a diagram showing the relationship between 4 vertex regions of a license plate and an actual license plate region.

FIG. 5(a) is a top left vertex area view of a license plate;

FIG. 5(b) is a diagram of the left lower vertex region of the license plate;

FIG. 5(c) is a diagram of the top right vertex area of the license plate;

FIG. 5(d) is a diagram of the right lower vertex region of the license plate;

fig. 6 is a diagram of the effect of CF _ NMS processing.

Fig. 7 is a diagram showing a case where a plurality of license plates exist.

FIG. 8 is a diagram illustrating the effect of license plate vertex classification processing.

FIG. 9(a) is a schematic diagram of license plate single missing vertex prediction (taking AC as a diagonal to make a parallelogram);

FIG. 9(b) is a schematic diagram of license plate single missing vertex prediction (using BC as diagonal to make parallelogram);

FIG. 9(c) is a schematic diagram of the license plate single missing vertex prediction (parallelogram with AB as diagonal).

FIG. 10 is a schematic diagram of the license plate region determined from 4 vertices.

Fig. 11(a) is a diagram of the detection effect at 608 × 608 input sizes;

FIG. 11(b) is a diagram showing the effect of detection at 1024 × 1024 input sizes;

FIG. 11(c) is a graph showing the effect after fusion.

FIG. 12 is a diagram of the effect of fine license plate localization in an unconstrained scene.

Detailed Description

A method for accurately positioning an unconstrained license plate by combining whole and part target detection comprises the steps of firstly, synchronously detecting 4 types of target areas (target areas with upper left, upper right, lower right and lower left vertexes) and a whole license plate area by using a YOLOv3 algorithm, wherein the target areas are centered on the top of the license plate. And then, the predication of the vertex position of the license plate is indirectly realized by positioning the vertex region of the license plate, and an accurate license plate region is obtained by combining the post-processing operations of a non-maximum suppression algorithm (CF _ NMS) for neglecting the type, vertex region classification and single missing vertex predication. Finally, in order to further improve the performance of the license plate positioning algorithm combining the whole and the part target detection, strategies such as multi-model fusion, combination of coarse positioning and fine positioning and the like are fused into a positioning algorithm, and the effectiveness of the strategies is verified through experiments. The license plate positioning algorithm combining the whole and component target detection is designed, and accurate license plate positioning is realized on the basis of not additionally increasing the calculation amount of YOLOv 3.

When an image is acquired from a real scene, the distance between a camera and a license plate is long, the angle is large, the license plate in the image is small, and the existing target detection network has poor detection effect on small targets, so that how to accurately detect the small targets is very important. Yolov3 is one of the most popular single-stage target detection networks, and has the greatest characteristic that one-time prediction of target positions and types can be realized directly through a regression mode, so that the method has great advantages in operation speed. The YOLOv3 not only maintains the advantage of high detection speed of the previous version, but also improves the accuracy of target detection by fusing a plurality of model optimization strategies. Most importantly, the method enhances the detection capability of small targets, and is widely applied, and the network structure of the method is shown in FIG. 2. The YOLOv3 can be divided into a backbone network and a prediction network, the backbone network module is the front 52 layers of the Darknet-53 network, the module uses a large number of residual error structures to increase the network depth, and the extraction effect of the network on deep-level features is improved. The prediction network integrates the deep-level features after up-sampling with the shallow-level features, so that the target detection effect is improved, and 3 groups of anchor frames with different sizes are distributed to each feature map to adapt to detection of targets with different sizes. The invention selects a large-scale CCPD (China city parking data set) shot in the most widely used non-constrained environment as training data. The data set is selected as follows: the Base subset is randomly divided into two equal parts, 10 thousand are used as training sets, and the rest parts and the rest subsets except NP in CCPD are used as test sets in total of 25 thousand. During training, 20% of images in a training set are randomly selected as a verification set, and the manufacture of data set labels is based on 4 top point information of a license plate. Because a small amount of errors exist in the CCPD data set labels, the CCPD data set labels have negative influence on model training, and in order to improve the model effect, the labels in the CCPD data set labels are re-labeled, so that the detection effect of the whole license plate is improved.

The difficulty of accurate license plate positioning in the non-constrained environment is that the license plate image is affected by various factors such as bad weather (rain, snow, fog and the like), foreign matter shielding, random shooting angles and distances, and blurring caused by camera shake, so that the license plate in the image is in the forms of small target size, large inclination angle, blurring and the like, the difficulty of license plate detection is greatly increased, and the positioning effect of the existing technology on the license plate image in the non-ideal environment is still to be improved. Therefore, the invention designs a license plate positioning method combining the whole and the part target detection based on the idea of local area detection taking the vertex as the center. The invention comprises the following steps: firstly, a YOLOv3 target detection algorithm is used for synchronously detecting a license plate vertex region and an integral license plate region, and then a post-processing process of CF _ NMS (no class non-maximum suppression), license plate vertex classification and single missing vertex prediction is combined to obtain an accurate license plate region. For license plates which cannot be accurately positioned, the external rectangular area of the license plate can be directly reserved. In addition, in order to further improve the effect of accurate license plate positioning, the invention integrates a multi-model positioning result fusion strategy into target detection, and the specific flow is shown in fig. 3.

Because the license plate in the unconstrained environment occupies a relatively small area in the image, the vertex area of the selected license plate is relatively smallIf the size of the target is too small, the characteristics of the target are not obvious, so that the target detection network is not favorable for positioning; the size of the vertex area is set to be too large, the characteristics of the target area are not uniform, and the network learning is not facilitated, so that the detection effect is influenced due to the fact that the model cannot be converged. In order to make the characteristics of the top area of the license plate more consistent. The invention provides a self-adaptive method for setting the size of 4 vertex target areas of a license plate. As shown in fig. 4, a relationship diagram between 4 vertex regions of a license plate and an actual license plate region is shown, A, B, C, D in fig. 4 are respectively a right lower vertex, a left upper vertex and a right upper vertex of the license plate, and 4 square frames are respectively vertex regions corresponding to the 4 vertices, that is, a target region for model training. The side length of the target area is related to the size of the license plate, the side length of the square area corresponding to the upper left vertex C and the lower left vertex B is the same, and the length is 2h_leftThe side lengths of the areas corresponding to the upper right vertex D and the lower right vertex A are the same, and the length is 2h_rightWherein h is_left、h_rightThe difference in height between BC and AD. By cutting out the 4 vertex regions of the 4 license plates in the above manner, as shown in fig. 5(a), 5(b), 5(c) and 5(d), each type of vertex region of the license plates has similarity, the regions including the license plates are all located in the same right-angle direction, and the rest directions are background information. And finally, taking the external rectangular region of the license plate as one class, and taking 5 classes as a target region for model training.

After the synchronous detection of the license plate vertex area and the whole license plate area is finished, the model detection result is processed by using the post-processing operations of CF _ NMS, license plate vertex classification and single missing vertex prediction, so that an accurate license plate area is obtained. The CF _ NMS is different from a conventional NMS in that it treats the bounding boxes of multiple classes as the same class, and is suitable for handling the condition that the allowable overlapping area between classes is not large. The method comprises the following steps:

step 1: and setting all the license plate vertex region types detected in the image as uniform values.

Step 2: and putting all the vertex area bounding box information into the set B, and establishing an empty set D for storing the bounding boxes needing to be reserved.

And step 3: and (3) taking the bounding box with the maximum confidence from the B, adding the bounding box with the maximum confidence into the B, deleting the information of the bounding box from the B, performing IoU operation shown in formula (1) on the taken bounding box and all other bounding boxes in the B, and then deleting all target bounding boxes IoU larger than a specified threshold value of 0.7 from the set B.

And 4, step 4: and (4) repeating the step (3) until the set B is empty, and finally obtaining a set D which contains all license plate top point region bounding boxes needing to be reserved. The resulting effect is shown in fig. 6, where CF _ NMS can suppress the redundant detection blocks, leaving only 5 classes needed for detection.

As shown in fig. 7, when there are multiple license plates in fig. 7, the vertex regions need to be classified to obtain the correct license plate region. The invention designs a license plate vertex classifying method, when a certain license plate vertex area is intersected with a certain license plate area, the center point of the vertex area is a certain license plate vertex in the corresponding license plate area. The effect of the license plate vertex classification processing is shown in fig. 8, and the yellow rectangle in fig. 8 indicates the classification result.

The license plate vertex region classification steps are as follows:

step 1): and respectively putting the vertex region target and the license plate region target obtained after the CF _ NMS processing into a set B1 and a set B2, and establishing an empty set D1.

Step 2): an empty set D0 is created, any license plate region is taken from B2, its information is saved in D0 and deleted from B2, all vertex region bounding boxes that intersect it are taken from B1, their information is saved in D0 and deleted from B1.

Step 3): and D0, judging whether the number of the bounding boxes is more than 3, namely, keeping the condition that the number of the detected license plate vertex areas is 3, 4 or more, because when the number of the vertex areas is less than 3, accurate license plate areas cannot be obtained. If the number of the vertex areas is more than or equal to 3, adding D0 into D1, otherwise, not performing any processing.

Step 4): repeating step 2) and step 3) until either B1 or B2 is empty.

And 5) if the B2 is empty and the number of the remaining license plate vertex areas in the B1 is more than 3, storing the vertex area information in an empty set D2, and adding the D2 into the D1, otherwise, not performing any processing.

After all the false detection vertexes are suppressed, the region 1 and the region 2 in fig. 8 are two types of classification results, and it can be seen that the license plate vertex classification method in the invention can cope with the situations that a plurality of license plates exist in the image and false detection vertexes exist.

After the license plate vertex classification step, the corresponding conditions of the existing license plate region and the license plate vertex region are divided into 3 types: 1 license plate region and 3 corresponding vertex regions, 1 license plate region and 4 or more corresponding vertex regions, 0 license plate region and 4 or more vertex regions. For the case of 4 vertex region determination, 4 bounding box centers may be directly acquired. And for the condition that the number of the license plate vertex areas exceeds 4, the number of the license plate vertex areas is a small number because the license plate vertex areas contain wrong vertex areas, so that the centers of 4 license plate area bounding boxes can be randomly selected from the license plate area vertex areas, and then the license plate area can be accurately obtained by connecting the license plate area bounding boxes according to a certain sequence. For the case of only 3 vertex regions, the missing vertex positions can be predicted in combination with the corresponding license plate regions. As shown in fig. 9(a), 9(b), and 9(c), A, B, C represents the vertex positions corresponding to the three vertex regions, and the black rectangular frame represents the detected rectangular license plate region, and three parallelograms can be predicted from A, B, C, i.e., the parallelogram regions obtained with AC, BC, and AB as the diagonal lines, but only with BC as the diagonal line are the closest to the real license plate region. The judgment is based on that the parallelogram and rectangular license plate region IoU obtained by using BC as diagonal prediction is the largest, so that the license plate region with correct prediction can be reserved by setting a corresponding IoU threshold value.

After the post-processing process, 4 vertex coordinates of the corresponding license plate in the image can be obtained, and the license plate region can be obtained by connecting the 4 vertexes according to a certain sequence. However, there are many ways to connect the 4 vertices in order, and only a specific order can obtain the correct license plate detection frame. As shown in fig. 10, a schematic diagram of determining a license plate region according to 4 vertices is provided, where the specific method is as follows: firstly, finding out the vertex A positioned at the top end, then calculating included angles alpha, beta and gamma between connecting lines of the vertex and other three vertexes and a horizontal rightward line segment of the vertex, wherein the vertex with the minimum and the maximum corresponding included angle degrees is adjacent to the vertex A and is respectively marked as a point B and a point D, the rest vertexes are marked as a point C, and finally, connecting the vertexes according to the sequence of A-B-C-D-A to obtain the license plate region.

In order to further improve the detection effect of the license plate vertex region, particularly the detection effect of the license plate vertex region target with large scale span, the invention integrates a multi-model fusion strategy in a license plate region and vertex region synchronous detection algorithm based on YOLOv 3. The effect graphs are shown in fig. 11(a), fig. 11(b) and fig. 11(c), and the specific steps of fusing the detection results are as follows:

the method comprises the following steps: and (3) reserving target boundary box information of the license plate vertex area output by a YOLOv3 model under two input sizes of 608 multiplied by 608 and 1024 multiplied by 1024 into a set D, wherein all target boundary boxes only comprise the center point, width and height and category information of a rectangular box, and when model fusion is carried out, all license plate vertex areas are treated as targets of the same type.

Step two: and (4) establishing an empty set B, randomly taking one license plate vertex region bounding box in the D and putting the license plate vertex region bounding box in the B because confidence coefficient information does not exist at the moment, solving IoU between the remaining vertex region bounding boxes in the D and the rest vertex region bounding boxes, and deleting the vertex region bounding box information of IoU >0.45 from the D.

Step three: and (4) repeating the step (II) until the number of the vertex areas contained in the set D is zero, and finally obtaining all the vertex area bounding boxes in the set B, namely the result of vertex area fusion.

Step IV: and fusing the license plate region detection frames detected by the two YOLOv3 target detection models with different input sizes according to the same mode, and reserving the finally obtained license plate vertex region and license plate region fusion results. Compared with the effect of a single detection model, the effect after fusion is obviously improved.

Fig. 12 shows the accurate positioning effect and the license plate correction effect after determining 4 vertexes by using the license plate positioning method combining whole and component target detection, and it can be seen that: the method provided by the invention can obtain good accurate positioning effect on the license plate image acquired under non-ideal conditions.