阅读说明：本技术 鱼眼相机图片中目标检测矫正的方法与系统 (Method and system for detecting and correcting target in fisheye camera picture ) 是由 杨帆 王瀚洋 胡建国 白立群 陈凯琪 于 2020-06-26 设计创作，主要内容包括：本发明提供一种鱼眼相机图片中目标检测矫正的方法与系统,包括：接收鱼眼相机的视频流输入,通过帧处理获得检测用鱼眼图片；检测鱼眼中照片中的行人目标；以检测结果的目标框为基础,将目标框进行扩边抠取,作为感兴趣区域ROI；获取目标框中心点与鱼眼图片中心点的竖向夹角；根据所述竖向夹角和旋转中心确定旋转矩阵,其中以所述目标框中心点作为旋转中心；利用旋转矩阵对感兴趣区域ROI进行仿射变换获得新的感兴趣区域new_ROI；获取目标框在新的感兴趣区域new_ROI中位置；新的感兴趣区域new_ROI中的目标框再次抠取,输出新的图片作为目标识别的输入图片。本发明根据目标在鱼眼图片中的位置,将目标旋转正向将旋转的目标矫正正向,便于目标的识别。(The invention provides a method and a system for detecting and correcting a target in a fisheye camera picture, which comprises the following steps: receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing; detecting a pedestrian target in the picture in the fisheye; based on the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest (ROI); acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture; determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center; carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI; acquiring the position of the target frame in a new ROI (region of interest) _ ROI; and the target frame in the new interesting area new _ ROI is scratched again, and a new picture is output as an input picture for target recognition. According to the position of the target in the fisheye picture, the rotating target is rotated in the positive direction, and the rotated target is corrected in the positive direction, so that the target can be conveniently identified.)

1. A method for detecting and correcting a target in a fisheye camera picture is characterized by comprising the following steps:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

2. The method for detecting and correcting the object in the fish-eye camera picture as claimed in claim 1, wherein the coordinate of the center point is (x) for frame detection in step 3₁，y₁) And the width is w, the height is h, then expand the edge of the target frame and scratch, as the region of interest ROI, wherein the operation of expanding the edge and scratching comprises: keeping the central point of the target frame unchanged, and expanding the width w and the height h to N times of the original width w and the original height h respectively.

3. The method for detecting and correcting the object in the fish-eye camera picture according to claim 1, wherein in the step 3, the value of the expansion multiple N of the width w and the height h is between 3 and 5.

4. The method of claim 1, wherein in step 4, the target in the fisheye image is detected and corrected according to a fisheye image center point (x)₂，y₂) In calculating the target frameThe vertical included angle theta between the connecting line of the center point and the center point of the fish-eye picture and the vertical line is calculated, wherein the horizontal distance d between the two center points is calculated_xAnd a vertical distance d_y：

d_x＝x₂-x₁

d_y＝y₂-y₁

Then, according to the arctan formula, θ can be calculated:

θ＝arctan(d_x/d_y)。

5. the method for detecting and correcting the target in the fish-eye camera picture according to claim 1, wherein in the step 5, a rotation matrix M of the target frame is calculated according to the vertical included angle θ, wherein the rotation process takes the center point of the original target frame as a rotation center, the rotation center is taken as a new coordinate (0,0) point, and the calculation formula of the rotation matrix M is as follows:

6. the method according to claim 5, wherein in the step 6, for any pixel C coordinate (x, y) in the ROI, a new point C ' coordinate (x ', y ') is obtained after rotation, and the relationship between the two points before and after rotation is C ═ C × M, and the specific relationship is as follows:

x′＝xcosθ+ysinθ

y'＝-xsinθ+ycosθ

and obtaining a new interesting region new _ ROI by carrying out affine transformation on all pixel points in the interesting region ROI one by utilizing the rotation matrix M.

7. The method for object detection and rectification in a fish-eye camera picture as claimed in claim 4, wherein in the step 6, the position of the object frame in the new region of interest new _ ROI is determined according to the following manner:

and (3) taking the central point of the original target frame as a rotation center, rotating the ROI by an angle of theta, and then, correspondingly taking the coordinate point of the target in the rotated target frame as the coordinate point before rotation.

8. The method for object detection and correction in a fisheye camera picture as claimed in claim 5, wherein the step 7 matting operation comprises: the region pixels of the target frame in the new region of interest new ROI are fetched.

9. A system for correcting a target detection frame in a fisheye camera picture, comprising:

a module for receiving video stream input of a fisheye camera and obtaining a fisheye picture for detection through frame processing;

for detecting pedestrian objects in the photograph in the fish-eye;

a module for performing edge expansion and scratching on the target frame based on the target frame of the detection result to be used as the ROI;

the module is used for acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

a module for determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

a module for performing affine transformation on the region of interest ROI by using the rotation matrix to obtain a new region of interest new _ ROI;

a module for obtaining the position of the target frame in the new ROI; and

and a module for scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

10. A system for correcting a target detection frame in a fisheye camera picture, comprising:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

Technical Field

The invention relates to the technical field of image processing, in particular to a method and a system for detecting and correcting a target in a fisheye camera picture.

Background

In the current security field, because the range of the visual angle of the fisheye lens is wide, the fisheye lens is usually used as a video stream acquisition tool. But there is the problem of image distortion in the practical application of the fisheye lens, and the distortion is more serious the farther away from the center of the picture. Meanwhile, the rotation angles of the targets in the fisheye lens are different, and the rotated targets are not convenient to identify. Both of these problems present certain identification difficulties.

At present, the most common solution is to use a calibration method to obtain internal parameters of a fisheye lens camera, correct a fisheye distortion picture, and then use the corrected picture for processing. The method has the defects that the distorted picture is not corrected correctly completely by the correction result, and meanwhile, different camera lenses have difference in internal parameters due to inconsistent installation, so that different cameras cannot reuse the same set of calibration parameters. Meanwhile, the method for detecting and identifying the depth learning image of the corrected image is based on the pixel value of the image, and extra target pixel information is not added to the corrected image, so that gain is basically avoided even if the corrected image is used for detecting and identifying.

In the identification task, the identification passing rate of the target in the forward direction is generally higher than that of other rotation angles, and the target needs to be rotated to the forward direction in order to improve the identification precision.

Disclosure of Invention

The invention aims to provide a method and a system for detecting and correcting a target in a fisheye camera picture.

In order to achieve the above object, a first aspect of the present invention provides a method for detecting and correcting an object in a fisheye camera picture, including the following steps:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

According to the second aspect of the present invention, there is provided a system for correcting an object detection frame in a fisheye camera picture, including:

a module for receiving video stream input of a fisheye camera and obtaining a fisheye picture for detection through frame processing;

for detecting pedestrian objects in the photograph in the fish-eye;

a module for performing edge expansion and scratching on the target frame based on the target frame of the detection result to be used as the ROI;

the module is used for acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

a module for determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

a module for performing affine transformation on the region of interest ROI by using the rotation matrix to obtain a new region of interest new _ ROI;

a module for obtaining the position of the target frame in the new ROI; and

and a module for scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

According to a third aspect of the present invention, there is provided a system for correcting an object detection frame in a fisheye camera picture, including:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

According to the technical scheme, the rotation angle is calculated according to the position of the target in the picture, the rotated target is corrected in the positive direction, and the identification of the fisheye camera picture target is improved. According to the characteristic of the fisheye lens in the implementation process, the accurate target rotation angle can be obtained without adding extra angle information in a detection task; for targets with different rotation angles and distortion in the fisheye lens, the target boundary frame can be accurately segmented, so that the targets contain less interference information and are convenient to identify.

It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail below can be considered as part of the inventive subject matter of this disclosure unless such concepts are mutually inconsistent. In addition, all combinations of claimed subject matter are considered a part of the presently disclosed subject matter.

The foregoing and other aspects, embodiments and features of the present teachings can be more fully understood from the following description taken in conjunction with the accompanying drawings. Additional aspects of the present invention, such as features and/or advantages of exemplary embodiments, will be apparent from the description which follows, or may be learned by practice of specific embodiments in accordance with the teachings of the present invention.

Drawings

The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. Embodiments of various aspects of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which:

fig. 1 is a schematic flow chart of correcting a target detection frame in a fish-eye camera picture according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of a fisheye photo obtained by a fisheye camera, in which a first layer small frame represents a detected target frame, and a second layer large frame is extracted as a region of interest ROI after being enlarged.

Fig. 3a-3c are schematic views of a ROI according to three regions of interest.

Fig. 4a-4c are three schematic diagrams corresponding to fig. 3a-3c, showing the ROI affine transformed by the rotation matrix to obtain a new ROI new _ ROI, in which the object is corrected.

Fig. 5a-5c are the positions of the object frames of the three pedestrian objects in the new _ ROI corresponding to the object detection corresponding to fig. 3a-3 c.

Fig. 6a-6c are graphical representations of three pedestrian objects detected corresponding to the objects corresponding to fig. 3a-3 c.

Fig. 7a-7c are graphical representations of three pedestrian objects obtained corresponding to fig. 4a-4c after correction in accordance with the present invention.

Fig. 8 is a schematic diagram of a test result of the correction of the target detection frame in the picture of the fisheye camera according to the embodiment of the invention.

Detailed Description

In order to better understand the technical content of the present invention, specific embodiments are described below with reference to the accompanying drawings.

In this disclosure, aspects of the present invention are described with reference to the accompanying drawings, in which a number of illustrative embodiments are shown. Embodiments of the present disclosure are not necessarily intended to include all aspects of the invention. It should be appreciated that the various concepts and embodiments described above, as well as those described in greater detail below, may be implemented in any of numerous ways, as the disclosed concepts and embodiments are not limited to any one implementation. In addition, some aspects of the present disclosure may be used alone, or in any suitable combination with other aspects of the present disclosure.

With reference to fig. 1-7, a method for detecting and correcting an object in a fisheye camera picture according to an exemplary embodiment of the invention includes the following steps:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

An embodiment of the above process is described in more detail below in conjunction with the illustrations of fig. 2-7.

In some scenes, such as shopping malls, banks, museums and other places, a fisheye camera is used as a monitoring and collecting device (security camera), and videos in front of a lens are collected and uploaded to a system for video processing and analysis. For example, in some embodiments, the output video streams are provided locally on edge computing devices or platforms for object detection, such as pedestrian detection, or uploaded to a cloud server for object detection.

In the embodiment of the invention, the correction of the target detection in the fish-eye camera picture comprises three stages: target detection is carried out, a rotation angle is determined, angle transformation is carried out to obtain a new region of interest, detection input is obtained again, and therefore target detection is carried out as an input picture.

Object detection, especially pedestrian detection, can be realized based on some existing pedestrian detection algorithms, and human body position in picture is detected

And the rotation angle theta and the affine transformation are calculated for correcting the target, so that efficient identification is facilitated.

As shown in FIG. 1, in step 3, the coordinates of the center point are detected as (x) for the frame₁，y₁) And the width is w, the height is h, then expand the edge of the target frame and scratch, as the region of interest ROI, wherein the operation of expanding the edge and scratching comprises: keeping the central point of the target frame unchanged, and expanding the width w and the height h to N times of the original width w and the original height h respectively.

Fig. 2 is a schematic diagram of a fisheye photo obtained by a fisheye camera, in which a first-level small frame (a dark black frame) represents a detected target frame, and a second-level large frame (a light frame) is extracted as a region of interest ROI after being enlarged. Fig. 3a-3c exemplarily show a schematic according to three regions of interest ROI.

Wherein, the expansion multiple N of the width w and the height h is preferably between 3 and 5.

In step 4, according to the center point (x) of the fish-eye picture₂，y₂) Calculating a vertical included angle theta between a connecting line of the center point of the target frame and the center point of the fisheye picture and a vertical line, wherein the horizontal distance d between the two center points is calculated respectively_xAnd a vertical distance d_y：

d_x＝x₂-x₁

d_y＝y₂-y₁

Then, according to the arctan formula, θ can be calculated:

θ＝arctan(d_x/d_y)。

since the previously calculated rotation angle θ is the off-angle of the target from the vertical line, to rotate the target in the forward direction, it is necessary to rotate- θ to the forward direction. Therefore, in step 5 of the present invention, a rotation matrix M of the target frame is calculated according to the vertical included angle θ, wherein the rotation process takes the center point of the original target frame as a rotation center, the rotation center is taken as a new coordinate (0,0) point, and the calculation formula of the rotation matrix M is as follows:

preferably, in step 6, for any pixel point C coordinate (x, y) in the region of interest ROI, a new point C ' coordinate (x ', y ') is obtained after rotation, and a relationship between two points before and after rotation is C ═ C × M, where the specific relationship is as follows:

x′＝xcosθ+ysinθ

y′＝-xsinθ+ycosθ

and obtaining a new ROI new _ ROI by performing affine transformation on all pixel points in the ROI one by using the rotation matrix M.

Shown in connection with fig. 4a-4c are three illustrations corresponding to fig. 3a-3c and showing the object in a positive way, and affine transformation of the ROI using a rotation matrix to obtain a new region of interest new _ ROI.

In step 6, the position of the target frame in the new region of interest new _ ROI is determined according to the following:

and (3) taking the central point of the original target frame as a rotation center, rotating the ROI by an angle of theta, and then, correspondingly taking the coordinate point of the target in the rotated target frame as the coordinate point before rotation.

The matting operation at step 7 includes: the region pixels of the target frame in the new region of interest new ROI are fetched.

As shown in fig. 5a to 5c, 6a to 6c, and 7a to 7c, the results of the pedestrian matting obtained by the detection frame of the original pedestrian detection algorithm and the results of the pedestrian matting obtained by the detection frame corrected by the method of the present invention are shown, respectively, it is obvious that the image processed by the method of the present invention is more favorable for the later detection and recognition, and the recognition efficiency and accuracy are better.

The methods of the above-described embodiments of the present invention were tested and validated in conjunction with the following tests.

The test set in the experiment contains 5000 pictures, wherein 11471 pedestrians are contained in the test set, the pictures are collected under the fisheye lens, and the pedestrians at different positions have different shapes and angles. The method of the invention is used for respectively carrying out target detection, target correction and target identification. In the comparative experiment, the same detector and flow are used in the target detection stage, and the detector results are 9749 pedestrian frames and 203 false detection frames, so that the target identification is performed on 9952 frames. Meanwhile, in order to verify the influence of the recognition training mode, a variable of whether to use the rotating pedestrian patch for training is added.

It can be seen through the experimental result that because there are the pedestrian frame of various angles simultaneously in the pedestrian detection result of no target correction, if do not use rotatory pedestrian's patch training in the training, only use the pedestrian's patch training recognition model of forward promptly, the precision of discernment is minimum because training set and test set scene difference are too big. For the pedestrian frame without target correction, the identification model is added with the rotary pedestrian patch, the precision is obviously improved, and the training set and the test set are similar in scene at the moment.

The pedestrian detection result with target correction basically only has the pedestrian frames with the same positive angle, and the recognition model is trained by adding the rotating pedestrian patch, so that the precision can be continuously improved, and the test scene of the recognition model is simple at the moment, but the precision is not optimal due to the great difference between the training set and the test set; by using the method, the pedestrian frame for target correction is provided, the rotary pedestrian patch is not added in the recognition model training, and the training and the testing are kept consistent on the premise of simplifying the testing scene, so that the precision is highest.

In combination with the above embodiments, the present invention can also be implemented in the following configuration.

{ System for correcting target detection frame in fisheye camera picture }

System for target detection frame is corrected in fisheye camera picture includes:

a module for receiving video stream input of a fisheye camera and obtaining a fisheye picture for detection through frame processing;

for detecting pedestrian objects in the photograph in the fish-eye;

a module for performing edge expansion and scratching on the target frame based on the target frame of the detection result to be used as the ROI;

the module is used for acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

a module for determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

a module for performing affine transformation on the region of interest ROI by using the rotation matrix to obtain a new region of interest new _ ROI;

a module for obtaining the position of the target frame in the new ROI; and

and a module for scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

{ System for correcting target detection frame in fisheye camera picture }

System for target detection frame is corrected in fisheye camera picture includes:

one or more processors;

a memory storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:

step 1, receiving video stream input of a fisheye camera, and obtaining a fisheye picture for detection through frame processing;

step 2, detecting a pedestrian target in the picture in the fisheye;

step 3, on the basis of the target frame of the detection result, carrying out edge expansion and scratching on the target frame to be used as a region of interest ROI;

step 4, acquiring a vertical included angle between the center point of the target frame and the center point of the fisheye picture;

step 5, determining a rotation matrix according to the vertical included angle and the rotation center, wherein the center point of the target frame is used as the rotation center;

step 6, carrying out affine transformation on the ROI by utilizing the rotation matrix to obtain a new ROI new _ ROI;

step 7, acquiring the position of the target frame in a new ROI (region of interest) new _ ROI; and

and 8, scratching the target frame in the new ROI new _ ROI again and outputting a new picture as an input picture for target identification.

Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention should be determined by the appended claims.