阅读说明：本技术 一种基于相机和雷达融合的目标检测方法及系统 (Target detection method and system based on camera and radar fusion ) 是由 陈晓光 阎峰 王智新 刘震 史龙 吴穗宁 于 2019-10-18 设计创作，主要内容包括：本发明公开了一种基于相机和雷达融合的目标检测方法及系统,所述检测方法包括：首先,获取雷达探测到的雷达数据以及相机同步采集到的图像；其次,对雷达数据中的雷达目标进行信息相关性滤波和卡尔曼滤波,筛选出有效雷达目标；然后,通过深度学习方法检测图像中的相机目标并获取相机目标的目标信息；最后,利用交小比方法融合有效雷达目标与相机目标,并筛选、输出融合目标结果。采用所述检测方法有效的降低了目标的漏检率和误检率,且对融合目标结果进行筛选,保证了融合目标结果的精确性。(The invention discloses a target detection method and a system based on camera and radar fusion, wherein the detection method comprises the following steps: firstly, radar data detected by a radar and images synchronously acquired by a camera are acquired; secondly, performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target; then, detecting a camera target in the image by a deep learning method and acquiring target information of the camera target; and finally, fusing the effective radar target and the camera target by using a cross-over-small ratio method, and screening and outputting a fused target result. By adopting the detection method, the missing detection rate and the false detection rate of the target are effectively reduced, and the fused target result is screened, so that the accuracy of the fused target result is ensured.)

1. A target detection method based on camera and radar fusion is characterized in that the detection method comprises the following steps:

acquiring radar data detected by a radar and images synchronously acquired by a camera;

performing information correlation filtering and Kalman filtering on radar targets in the radar data to screen out effective radar targets;

detecting a camera target in the image by a deep learning method and acquiring target information of the camera target;

and fusing the effective radar target and the camera target by using a cross-over-small ratio method, and screening and outputting a fused target result.

2. The object detection method according to claim 1, wherein the radar data includes an id, a type, a position, a speed, a distance, an angle of an object;

the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target;

the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target.

3. The method of claim 2, wherein the acquiring radar data detected by the radar and the images synchronously acquired by the camera further comprises:

installing and adjusting the positions of the radar and the camera to enable the radar and the camera to have a common view field;

and establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, and calibrating external parameters between the radar coordinate system and the camera coordinate system through a translation vector t and a rotation matrix R between the radar and the camera.

4. The method of object detection according to any of claims 1-3, wherein the method further comprises:

arranging a calibration field within the field of view;

shooting a calibration field through a camera to extract a plurality of feature points;

calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system.

5. The method for detecting the target according to any one of claims 1 to 3, wherein the step of performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target specifically comprises the following steps:

performing information correlation filtering on radar targets in the radar data:

acquiring radar data of a current frame, and calling the recorded radar data of a previous frame of the current frame;

respectively acquiring radar targets in a previous frame of radar data and a current frame of radar data, judging whether the radar target in the current frame appears in the previous frame, if so, taking an average value of the speed of the radar target in the previous frame and the speed of the radar target in the current frame, and taking the average value as the average speed v of the radar target in two frames;

based on the average speed v, according to the sampling time T and the position x of the radar target in the last frame_i-1Calculating the predicted position of the target in the current frame

Calculating the predicted position

Based on the predicted distance

and performing Kalman filtering on the reserved effective radar target.

6. The object detection method according to claim 4, wherein the acquiring the object information of the camera object comprises acquiring a speed of the camera object, and specifically comprises the following steps:

calculating the coordinates of the camera target in the middle point of the bottom edge of the candidate frame based on the upper left coordinate and the lower right coordinate of the candidate frame of the camera target, and taking the middle point of the bottom edge as a mass point;

based on formula (1), projecting the particles into a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system;

and calculating the speed of the camera target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system.

7. The method for detecting the target according to any one of claims 1 to 3 or 6, wherein the fusing the effective radar target and the camera target by using the cross-over-fractional ratio method, and screening and outputting the result of the fused target specifically comprises the following steps:

acquiring the intersection minir of the effective radar target and the camera target by utilizing an intersection minir method;

judging whether the cross-to-small ratio is larger than a first preset value or not, wherein,

if the intersection ratio is larger than a first preset value, outputting a fused target result;

if the intersection ratio is smaller than or equal to a first preset value, temporarily storing the camera target and the effective radar target, respectively recording the times of the independent appearance of the camera target and the effective radar target, and judging whether the times of the independent appearance of the camera target or the effective radar target is larger than a second preset value or not, wherein,

if the times of the independent occurrence of the camera target or the effective radar target are larger than a second preset value, rejecting the camera target or the effective radar target;

if the times of the independent appearance of the camera target or the effective radar target are less than or equal to a second preset value, the camera target or the effective radar target is reserved, and the steps are executed again.

8. The method of claim 7, wherein the effective radar target to camera target intersection-to-minimality ratio is:

the ratio of the area of the effective radar target candidate area overlapping the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame area.

9. The method according to claim 8, further comprising obtaining a candidate region of the radar target, specifically comprising:

setting the distance between the radar target and the origin of the radar coordinate system as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the size of the candidate area of the radar target is lambda times of the standard size, namely (lambda w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

10. A target detection system based on camera and radar fusion comprises a radar and a camera, and is characterized by further comprising:

the acquisition unit is used for acquiring radar data detected by a radar and images synchronously acquired by a camera;

the filtering unit is used for performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target;

an image detection unit for detecting a camera target in an image by a deep learning method and acquiring target information of the camera target;

and the fusion unit is used for fusing the effective radar target and the camera target by using a cross-over-fraction method, and screening and outputting a fusion target result.

11. The detection system of claim 10, wherein the radar data includes an id, a type, a position, a speed, a distance, an angle of a target;

the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target;

the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target.

12. The detection system according to claim 11, further comprising a setting unit, configured to adjust positions of the radar and the camera to have a common field of view, establish a radar coordinate system, a camera coordinate system, and a ground coordinate system, and calibrate external parameters therebetween through a translation vector t and a rotation matrix R between the radar and the camera; wherein the content of the first and second substances,

arranging a calibration field within the field of view;

shooting a calibration field through a camera to extract a plurality of feature points;

calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system.

13. The detection system according to any one of claims 10 to 12, wherein the filtering unit is further configured to perform the steps of:

performing information correlation filtering on radar targets in the radar data:

acquiring radar data of a current frame, and calling the recorded radar data of a previous frame of the current frame;

respectively acquiring radar targets in a previous frame of radar data and a current frame of radar data, judging whether the radar target in the current frame appears in the previous frame, if so, taking an average value of the speed of the radar target in the previous frame and the speed of the radar target in the current frame, and taking the average value as the average speed v of the radar target in two frames;

based on the average speed v, according to the sampling time T and the position x of the radar target in the last frame_i-1Calculating the predicted position of the target in the current frame

Calculating the predicted position

Based on the predicted distance

and performing Kalman filtering on the reserved effective radar target.

14. The detection system according to claim 12, wherein the image detection unit is further configured to obtain target information of the camera target including a speed of the camera target, and specifically perform the following steps:

calculating the coordinates of the camera target in the middle point of the bottom edge of the candidate frame based on the upper left coordinate and the lower right coordinate of the candidate frame of the camera target, and taking the middle point of the bottom edge as a mass point;

based on formula (1), projecting the particles into a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system;

and calculating the speed of the camera target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system.

15. The detection system according to any one of claims 10-12, 14, wherein the fusion unit is further configured to perform the steps of:

acquiring the intersection minir of the effective radar target and the camera target by utilizing an intersection minir method;

judging whether the cross-to-small ratio is larger than a first preset value or not, wherein,

if the intersection ratio is larger than a first preset value, outputting a fused target result;

if the intersection ratio is smaller than or equal to a first preset value, temporarily storing the camera target and the effective radar target, respectively recording the times of the independent appearance of the camera target and the effective radar target, and judging whether the times of the independent appearance of the camera target or the effective radar target is larger than a second preset value or not, wherein,

if the times of the independent occurrence of the camera target or the effective radar target are larger than a second preset value, rejecting the camera target or the effective radar target;

if the times of the independent appearance of the camera target or the effective radar target are less than or equal to a second preset value, the camera target or the effective radar target is reserved, and the steps are executed again.

16. The object detection system of claim 15, wherein the effective radar target to camera target intersection-to-minimality ratio is:

the ratio of the area of the effective radar target candidate area overlapping the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame area.

17. The object detection system of claim 16, further comprising a processing unit for obtaining a candidate region of a radar target, wherein,

setting the distance between the radar target and the origin of the radar coordinate system as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the size of the candidate area of the radar target is lambda times of the standard size, namely (lambda w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

Technical Field

The invention belongs to the technical field of target detection, and particularly relates to a target detection method and system based on camera and radar fusion.

Background

At present, a method or a system for detecting obstacles is mainly based on vehicle-mounted detection and is used for detecting the obstacles in front, the detection is realized based on a motion coordinate system, the detection is realized only by using a region of interest (ROI) superposition method, and higher false detection rate exists. In the field of rail transit detection, how to realize the detection of obstacles based on a static coordinate system and by fully utilizing information fusion of two dimensions of a camera and a radar becomes a technical problem to be solved more and more urgently.

Disclosure of Invention

Aiming at the problems, the invention provides a target detection method and a target detection system based on camera and radar fusion, the method effectively reduces the missing detection rate and the false detection rate of the target, and ensures the accuracy of the result of the fused target.

The invention aims to provide a target detection method based on camera and radar fusion, which comprises the following steps:

acquiring radar data detected by a radar and images synchronously acquired by a camera;

performing information correlation filtering and Kalman filtering on radar targets in the radar data to screen out effective radar targets;

detecting a camera target in the image by a deep learning method and acquiring target information of the camera target;

and fusing the effective radar target and the camera target by using a cross-over-small ratio method, and screening and outputting a fused target result.

Further, the radar data includes id, type, location, speed, distance, angle of the target;

the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target;

the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target.

Further, before the acquiring the radar data detected by the radar and the image synchronously acquired by the camera, the method further comprises:

installing and adjusting the positions of the radar and the camera to enable the radar and the camera to have a common view field;

and establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, and calibrating external parameters between the radar coordinate system and the camera coordinate system through a translation vector t and a rotation matrix R between the radar and the camera.

Further, the method further comprises:

arranging a calibration field within the field of view;

shooting a calibration field through a camera to extract a plurality of feature points;

calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system.

Further, the information correlation filtering and kalman filtering are performed on the radar target in the radar data, and screening out an effective radar target specifically includes the following steps:

performing information correlation filtering on radar targets in the radar data:

acquiring radar data of a current frame, and calling the recorded radar data of a previous frame of the current frame;

respectively acquiring radar targets in a previous frame of radar data and a current frame of radar data, judging whether the radar target in the current frame appears in the previous frame, if so, taking an average value of the speed of the radar target in the previous frame and the speed of the radar target in the current frame, and taking the average value as the average speed v of the radar target in two frames;

based on the average speed v, according to the sampling time T and the position x of the radar target in the last frame_i-1Calculating the predicted position of the target in the current frame

Calculating the predicted position

Predicted distance to origin of radar coordinate system

Based on the predicted distance

The distance d from the radar target to the origin of the radar coordinate system in the current frame_iRemoving the invalid radar target and keeping the valid radar target;

and performing Kalman filtering on the reserved effective radar target.

Further, the acquiring the target information of the camera target includes acquiring a speed of the camera target, and specifically includes the following steps:

calculating the coordinates of the camera target in the middle point of the bottom edge of the candidate frame based on the upper left coordinate and the lower right coordinate of the candidate frame of the camera target, and taking the middle point of the bottom edge as a mass point;

based on formula (1), projecting the particles into a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system;

and calculating the speed of the camera target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system.

Further, the fusing the effective radar target and the camera target by using the cross-small ratio method, and screening and outputting a fused target result specifically comprises the following steps:

acquiring the intersection minir of the effective radar target and the camera target by utilizing an intersection minir method;

judging whether the cross-to-small ratio is larger than a first preset value or not, wherein,

if the intersection ratio is larger than a first preset value, outputting a fused target result;

if the intersection ratio is smaller than or equal to a first preset value, temporarily storing the camera target and the effective radar target, respectively recording the times of the independent appearance of the camera target and the effective radar target, and judging whether the times of the independent appearance of the camera target or the effective radar target is larger than a second preset value or not, wherein,

if the times of the independent occurrence of the camera target or the effective radar target are larger than a second preset value, rejecting the camera target or the effective radar target;

if the times of the independent appearance of the camera target or the effective radar target are less than or equal to a second preset value, the camera target or the effective radar target is reserved, and the steps are executed again.

Further, the effective radar target to camera target intersection-to-minimality ratio is:

the ratio of the area of the effective radar target candidate area overlapping the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame area.

Further, the method further includes acquiring a candidate region of the radar target, specifically including:

setting the distance between the radar target and the origin of the radar coordinate system as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the size of the candidate area of the radar target is lambda times of the standard size, namely (lambda w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

Another object of the present invention is to provide a target detection system based on camera and radar fusion, which includes a radar and a camera, and further includes:

the acquisition unit is used for acquiring radar data detected by a radar and images synchronously acquired by a camera;

the filtering unit is used for performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target;

an image detection unit for detecting a camera target in an image by a deep learning method and acquiring target information of the camera target;

and the fusion unit is used for fusing the effective radar target and the camera target by using a cross-over-fraction method, and screening and outputting a fusion target result.

Further, the radar data includes id, type, location, speed, distance, angle of the target;

the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target;

the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target.

Furthermore, the system also comprises a setting unit, a calibration unit and a control unit, wherein the setting unit is used for adjusting the positions of the radar and the camera to enable the radar and the camera to have a common view field, establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, and calibrating external parameters between the radar and the camera through a translation vector t and a rotation matrix R between the radar and the camera; wherein the content of the first and second substances,

arranging a calibration field within the field of view;

shooting a calibration field through a camera to extract a plurality of feature points;

calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system.

Further, the filtering unit is further configured to perform the following steps:

performing information correlation filtering on radar targets in the radar data:

acquiring radar data of a current frame, and calling the recorded radar data of a previous frame of the current frame;

respectively acquiring radar targets in a previous frame of radar data and a current frame of radar data, judging whether the radar target in the current frame appears in the previous frame, if so, taking an average value of the speed of the radar target in the previous frame and the speed of the radar target in the current frame, and taking the average value as the average speed v of the radar target in two frames;

based on the average speed v, according to the sampling time T and the position x of the radar target in the last frame_i-1Calculating the predicted position of the target in the current frame

Calculating the predicted positionPredicted distance to origin of radar coordinate system

Based on the predicted distance

The distance d from the radar target to the origin of the radar coordinate system in the current frame_iRemoving the invalid radar target and keeping the valid radar target;

and performing Kalman filtering on the reserved effective radar target.

Further, the image detection unit is further configured to acquire target information of the camera target, including acquiring a speed of the camera target, and specifically perform the following steps:

calculating the coordinates of the camera target in the middle point of the bottom edge of the candidate frame based on the upper left coordinate and the lower right coordinate of the candidate frame of the camera target, and taking the middle point of the bottom edge as a mass point;

based on formula (1), projecting the particles into a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system;

and calculating the speed of the camera target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system.

Further, the fusion unit is further configured to perform the following steps:

acquiring the intersection minir of the effective radar target and the camera target by utilizing an intersection minir method;

judging whether the cross-to-small ratio is larger than a first preset value or not, wherein,

if the intersection ratio is larger than a first preset value, outputting a fused target result;

if the intersection ratio is smaller than or equal to a first preset value, temporarily storing the camera target and the effective radar target, respectively recording the times of the independent appearance of the camera target and the effective radar target, and judging whether the times of the independent appearance of the camera target or the effective radar target is larger than a second preset value or not, wherein,

if the times of the independent occurrence of the camera target or the effective radar target are larger than a second preset value, rejecting the camera target or the effective radar target;

if the times of the independent appearance of the camera target or the effective radar target are less than or equal to a second preset value, the camera target or the effective radar target is reserved, and the steps are executed again.

Further, the effective radar target to camera target intersection-to-minimality ratio is:

the ratio of the area of the effective radar target candidate area overlapping the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame area.

Further, the system comprises a processing unit for obtaining a candidate region of the radar target, wherein,

setting the distance between the radar target and the origin of the radar coordinate system as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the size of the candidate area of the radar target is lambda times of the standard size, namely (lambda w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

Compared with an independent camera-based detection system or a radar-based detection system, the target detection method has stronger robustness, can realize all-weather detection, and has extremely low omission ratio and low false alarm rate; the target fusion result combines the advantages of the camera and the radar by adopting the object type and position identified by the camera (image) and the speed identified by the radar, so that the target detection result is more accurate by the target fusion result.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 shows a schematic flowchart of a target detection method based on camera and radar fusion in an embodiment of the present invention;

FIG. 2 is a diagram illustrating a relationship between a radar coordinate system, a camera coordinate system, and a ground coordinate system in an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a radar target information correlation filtering and Kalman filtering flow in an embodiment of the present invention;

FIG. 4 is a schematic diagram illustrating a process of fusing a radar target and a camera target according to an embodiment of the present invention;

fig. 5 shows a schematic structural diagram of a target detection system based on camera and radar fusion in an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, an embodiment of the present invention introduces a target detection method based on a camera and radar fusion, where the detection method includes first acquiring radar data detected by a radar and an image synchronously acquired by the camera; secondly, performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target; then detecting a camera target in the image by a deep learning method and acquiring target information of the camera target; and finally, fusing the effective radar target and the camera target by using a cross-over-small ratio method, and screening and outputting a fused target result. Compared with an independent camera-based detection system or a radar-based detection system, the method has stronger robustness, can realize all-weather detection, and has extremely low omission factor and low false alarm rate;

further, the radar data includes id (number), type, position, speed, distance, angle of the target; the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target; the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target. The camera target and the corresponding effective radar target in the fusion target result are the same target, and the fusion target result adopts the object type and position identified by the camera (image) and the radar identification speed, so that the advantages of the camera and the radar are combined, and the target detection result is more accurate by the fusion target result. It should be noted that: the type in the object information refers to the category of the object, for example: the types of objects may include humans, animals, and cars, among others.

In this embodiment, before the respectively obtaining the radar data detected by the radar and the image synchronously acquired by the camera, the method further includes:

installing and adjusting the positions of the radar and the camera to enable the radar and the camera to have a common view field; as shown in fig. 2, the radar and camera are rigidly connected by a bracket. Preferably, the radar is installed at a position 1m away from the ground and needs to be installed perpendicular to the ground, the camera is installed at a position about 6m away from the ground, and the camera and the radar are adjusted to have a common view field. Further, the camera can be a security monitor, and the radar can be a millimeter wave radar.

Establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, and calibrating external parameters between the radar coordinate system and the camera coordinate system through a translation vector t and a rotation matrix R between the radar and the camera; wherein, as shown in FIG. 2, the radar coordinate system is O_r-x_ry_rz_r,The camera coordinate system is O_c-x_cy_cz_cThe ground coordinate system is O_g-x_gy_gz_g. Further specifically, the establishing of the radar coordinate system and the camera coordinate system further includes arranging a calibration field in the field of view, specifically including the following steps:

shooting a calibration field through a camera to extract a plurality of feature points; in fig. 2, before shooting the calibration field, 9 light-emitting feature points are set on the ground, and preferably, in the embodiment of the present invention, at least 4 feature points are extracted by shooting the calibration field with a camera.

Based on the plurality of feature points, calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, specifically, measuring a placing angle of a calibration field relative to a radar coordinate system and a translation relation of an origin of the calibration field coordinate system relative to the radar coordinate system by a meter ruler, thereby obtaining the homography matrix H between the calibration field coordinate system and the radar coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system. After the homography matrix H is obtained through calculation, the one-to-one mapping relation between the camera coordinates of the target point and the ground coordinates is obtained, the target can be rapidly projected onto the ground from the camera coordinates, the speed of the target can be calculated conveniently, and the like. By establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, calibrating the radar coordinate system and the camera coordinate system, and calibrating the camera coordinate system and the groundThe relation of the coordinate system improves the speed of acquiring the target information and ensures the accuracy of detecting the target.

In this embodiment, since the radar data returns a moving target in the detection area, the acquiring of the radar data detected by the radar further includes performing information correlation filtering and kalman filtering combined filtering on the radar target in the radar data, so as to avoid a state where a false alarm target occurs in the radar due to reflection of the surrounding environment and interference of natural objects (for example, leaves blown by wind); in addition, the target volatility between radar data frames is reduced; as shown in fig. 3, the method specifically includes the following steps:

s11: acquiring radar data of a current frame, and calling recorded radar data of a previous frame;

s12: taking a radar target detected in a current frame, judging whether the radar target appears in a previous frame, if so, executing step S13, and if not, not performing filtering processing, that is, for a real signal, it is certain that valid data can be matched for multiple times, so as to keep the radar target as the previous frame data for cyclic detection of the next frame of radar data.

S13: averaging the speed of the radar target in the previous frame and the speed of the radar target in the current frame, and taking the average value as the average speed v of the radar target in two frames;

s14: based on the average velocity v, according to the sampling time T and the position x of the target in the previous frame_i-1Calculating the predicted position of the radar target in the current frame

In particular, the method comprises the following steps of,

s15: calculating the predicted position

Predicted distance to origin of radar coordinate system

Wherein the content of the first and second substances,

s16: based on the predicted distance

The distance d from the radar target to the origin of the radar coordinate system in the current frame_iD is calculated_iAnd

the absolute value of the difference between the two and d is determined_iAnd

whether the absolute value of the difference between is less than a preset threshold, i.e.

If the absolute value of the difference is smaller than a preset threshold, retaining the radar target, and executing step S17, if the absolute value of the difference is greater than or equal to the preset threshold, determining that the radar target is a false detection target, and rejecting the radar target (that is, the radar target is an invalid radar target);

s17: judging whether the current frame has an undetermined radar target, if so, executing the step S12; if there is no undetermined radar target in the current frame, go to step S18;

s18: and performing Kalman filtering on each reserved radar target to obtain a smooth output result. Preferably, each kalman filtered radar target is placed in a corresponding output list, and each kalman filtered radar target is a valid radar target. Further, finally, each frame of radar data returns all the detected valid radar targets after being filtered.

In this embodiment, the steps S11 to S17 are information correlation filtering, so that when the radar is used to collect a target, the speed of the false alarm target affected by reflection is utilized, and the reflected target is removed based on the characteristic that the speed of the false alarm target is not matched with the displacement between adjacent frames, thereby ensuring the effectiveness and accuracy of the radar target. Further, the preset threshold may be 0.05m (meter), i represents a current frame, i is an integer, and the position x_i-1Distance d_iAre known radar target information in the radar data.

In this embodiment, the respectively acquiring data detected by the cameras includes: based on the image shot by the camera, the id, the type, the upper left coordinate information and the lower right coordinate information of each target in the image are obtained by a deep learning method. Further, the acquiring the target information of the camera target includes acquiring a speed of the camera target, and specifically includes the following steps: firstly, calculating coordinate information of a camera target in a middle point of a bottom edge of a candidate frame based on upper left coordinate information and lower right coordinate information of the target candidate frame, and taking the middle point of the bottom edge as a particle; then, based on a formula (1), projecting the mass point to a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system; and finally, calculating the speed of the target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system. More specifically, the coordinates P of the object in the camera coordinate system_cSubstituting into formula (1), the actual coordinate P of the camera target in the ground coordinate system can be calculated_gAnd obtaining the real position of the camera target. For adjacent frames in the real-time image, the ground coordinates of the same target are respectively calculated by using a formula (1), the Euclidean distance between two coordinates is calculated, the displacement of the target in the two frames is obtained, and the displacement is divided by the time difference of the two frames, so that the speed can be calculated. Through the steps, all detected effective camera targets are returned after each frame of camera image is detected, and the target information of each camera target in the effective camera targets comprises id, type, left upper coordinate and right lower coordinate of the candidate frame, speed information and the like.

In this embodiment, fusing a radar target and a corresponding camera target in each frame of radar data, and traversing each camera target for each radar target for each frame, so as to fuse an effective radar target and a camera target by using an intersection-fraction method, and screening and outputting a result of the fused target specifically include the following steps, as shown in fig. 4:

s21, expanding the effective radar target into a candidate area according to the distance of the effective radar target;

s22, taking a valid radar target;

s23, acquiring the intersection ratio IOM of the effective radar target and the camera target by utilizing an intersection ratio method;

s24, judging whether the cross-to-small ratio IOM is larger than a threshold value T or not_IOMWherein, if IOM > T_IOMIf the effective radar target and the camera target are the same target, outputting a fused target result, wherein the target result comprises the target type of the camera, the candidate frame position and the speed of the corresponding effective radar target, and executing step S25; if IOM is less than or equal to T_IOMTemporarily storing the camera target and the effective radar target, respectively recording the times of the independent occurrence of the camera target and the effective radar target, and executing the step S241;

s241, judging whether the number of times of the independent occurrence of the camera target or the effective radar target is larger than a threshold value T or not_cntWherein, in the step (A),

if the number of times of the independent appearance of the camera target is more than a threshold value T_cntIf so, indicating that the camera target is the camera false detection, and rejecting the camera target; similarly, if the number of times of the single occurrence of the effective radar target is greater than the threshold value T_cntIf so, indicating that the effective radar target is false radar detection, and rejecting the effective radar target;

if the number of times of the independent appearance of the camera target is less than or equal to the threshold value T_cntIf yes, the camera target is reserved, and step S25 is executed; similarly, if the number of times of the single occurrence of the effective radar target is less than or equal to the threshold value T_cntIf yes, the valid radar target is reserved, and step S25 is executed;

s25, judging whether an undetermined radar target exists, namely judging whether an undetermined radar target exists in a frame where the effective radar target is located, and if the undetermined radar target exists, executing a step S22; and if no radar target which is not judged exists, ending the process.

In this example, T_IOMThe value of (a) may be 0.5, and further, the step S241 is also cross-validation filtering, that is, if a certain uncombined target is continuous multiple frames (threshold T)_cnt)If the false alarm rate is detected (the continuous multiframes are detected by only one device), the false detection of a single device is considered, and the false alarm rate is eliminated, so that the false alarm rate can be effectively reduced through cross validation filtering. And the fusion result is filtered by using a fusion mode based on the cross-to-small ratio and adopting a cross validation method and combining the mutual information between the camera and the radar, so that the accuracy of the fusion result is ensured.

The intersection minimality ratio of the effective radar target to the camera target is: the ratio of the area of the effective radar target candidate area overlapping the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame area. Namely, the formula of the cross-over-fraction ratio IOM is:

wherein A is_rIs the area of a candidate region of a radar target, A_cAs camera target candidate box area, A_nThe area of the candidate region of the radar target is the overlapping area of the candidate frame of the camera target.

Expanding the effective radar target into a candidate area according to the distance of the effective radar target specifically comprises: setting the distance between the radar target and the origin of the radar coordinate system as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the order of radarThe target candidate region size is λ times the standard size, i.e., (λ w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

Further, the upper left point C of the radar target candidate region can be obtained according to the formula (2)_nwAnd coordinates of lower right point C_seAnd then used to calculate the area of the radar target candidate region.

As shown in fig. 5, an embodiment of the present invention further introduces a target detection system based on camera and radar fusion, including a radar and a camera (not shown in the figure), and further including an obtaining unit, a filtering unit, an image detection unit, a fusion unit, a setting unit, and a processing unit, where the obtaining unit is configured to obtain radar data detected by the radar and an image synchronously acquired by the camera; the filtering unit is used for performing information correlation filtering and Kalman filtering on the radar target in the radar data to screen out an effective radar target; the image detection unit is used for detecting a camera target in an image by a deep learning method and acquiring target information of the camera target; the fusion unit is used for fusing the effective radar target and the camera target by using a cross-over-small ratio method, and screening and outputting a fusion target result.

The setting unit is used for adjusting the positions of the radar and the camera to enable the radar and the camera to have a common view field, establishing a radar coordinate system, a camera coordinate system and a ground coordinate system, and calibrating external parameters between the radar and the camera through a translation vector t and a rotation matrix R between the radar and the camera; wherein the content of the first and second substances,

arranging a calibration field within the field of view;

shooting a calibration field through a camera to extract a plurality of feature points;

calculating a homography matrix H between an image plane in a camera coordinate system and a ground plane in a ground coordinate system, wherein the homography matrix H satisfies:

P_g＝HP_c(1)

wherein, P_cIs the feature point coordinate, P, in the camera coordinate system_gIs the feature point coordinate in the ground coordinate system.

In this embodiment, the radar data includes id, type, position, speed, distance, and angle of the target; the target information of the camera target comprises id, type, position of a candidate frame, left upper and right lower coordinates of the candidate frame and speed information of the target; the fused target result includes the type of camera target, the candidate box position, and the velocity of the corresponding valid radar target.

The filtering unit is further configured to perform the above-mentioned steps S11-S18.

The image detection unit is further configured to acquire target information of the camera target, including acquiring a speed of the camera target, and specifically includes the following steps: calculating coordinate information of a camera target in the middle point of the bottom edge of the candidate frame based on the upper left coordinate information and the lower right coordinate information of the camera target candidate frame, and taking the middle point of the bottom edge as a particle; based on formula (1), projecting the particles into a ground coordinate system, and acquiring the position of the camera target in the ground coordinate system; and calculating the speed of the camera target by using a difference algorithm between frames based on the position of the camera target in the ground coordinate system.

The fusion unit is further configured to perform the above-mentioned steps S21-S25.

The intersection ratio of the effective radar target and the camera target is the ratio of the overlapping area of the effective radar target candidate area and the camera target candidate frame to the minimum area of the effective radar target candidate frame area and the area of the camera target candidate frame. Namely, the formula of the cross-over-fraction ratio IOM is:

wherein A is_rIs the area of a candidate region of a radar target, A_cAs camera target candidate box area, A_nThe area of the candidate region of the radar target is the overlapping area of the candidate frame of the camera target.

The processing unit is used for acquiring a candidate area of the radar target, wherein the distance between the radar target and the origin of the radar coordinate system is set as a standard distance d₀The size of the candidate region of the radar target is a standard size (w)₀,h₀) Wherein w is₀Is a standard distance d₀Width of lower candidate region, h₀Is a standard distance d₀Height of the next candidate region;

at any distance d, the size of the candidate area of the radar target is lambda times of the standard size, namely (lambda w)₀,λh₀) Wherein λ satisfies:

wherein d is_minIs the minimum detection distance, λ, of the radar_minFor radar targets at distance d_minMultiple of the size of the lower candidate region and the standard size, d_maxIs the maximum detection range, λ, of the radar_maxFor radar targets at distance d_maxThe lower candidate region size is a multiple of the standard size.

Compared with an independent camera-based detection system or a radar-based detection system, the target detection method has stronger robustness, can realize all-weather detection, and has extremely low omission factor and low false alarm rate; meanwhile, the advantages of the camera and the radar are combined, and the target result is fused, and the object type and position recognized by the camera (image) and the speed recognized by the radar have high precision.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.