阅读说明：本技术 一种基于互相关系数的摄像机与雷达目标匹配方法 (Camera and radar target matching method based on cross-correlation coefficient ) 是由 孙煜华 张弓 吴彬倩 于 2020-11-30 设计创作，主要内容包括：本发明公开了一种基于互相关系数的摄像机与雷达目标匹配方法。属于智能交通中多元传感器的信息融合技术领域,具体步骤：将毫米波雷达和摄像机检测到的目标根据路面坐标值投影到两个相同规格的零矩阵中；基于互相关系数的算法,对得到的雷达目标位置信息矩阵及摄像机目标位置信息矩阵进行整体校正；对于监控的每个车道,将校正后的雷达目标位置信息矩阵及摄像机目标位置信息矩阵设为两个集合,利用带权值的二分图匹配算法对两个集合中的目标位置进行局部校正,从而使两者相互匹配；两者相互匹配后即完成摄像机与雷达的目标匹配。本发明解决了智能交通监控系统中多元传感器的目标匹配问题,在普通计算机上具有计算量小、耗时少的优点。(The invention discloses a camera and radar target matching method based on cross-correlation coefficients. Belongs to the technical field of information fusion of a plurality of sensors in intelligent traffic, and specifically comprises the following steps: projecting targets detected by a millimeter wave radar and a camera into two zero matrixes with the same specification according to the coordinate value of the road surface; based on the algorithm of the cross correlation coefficient, integrally correcting the obtained radar target position information matrix and the camera target position information matrix; for each monitored lane, setting the corrected radar target position information matrix and camera target position information matrix into two sets, and locally correcting target positions in the two sets by using a bipartite graph matching algorithm with a weight value so as to enable the two sets to be matched with each other; and matching the camera and the radar after the camera and the radar are matched with each other. The invention solves the target matching problem of the multi-sensor in the intelligent traffic monitoring system, and has the advantages of small calculation amount and less time consumption on a common computer.)

1. A camera and radar target matching method based on cross correlation coefficients is characterized by comprising the following specific steps:

step (1.1), at the same time, projecting coordinate values of targets detected by a millimeter wave radar and a camera in a road coordinate system into two zero matrixes with the same specification, so as to obtain a radar target position information matrix and a camera target position information matrix;

step (1.2), performing overall correction on the obtained radar target position information matrix and camera target position information matrix based on an algorithm of a cross-correlation coefficient;

step (1.3), for each monitored lane, setting a corrected radar target position information matrix and a corrected camera target position information matrix into two sets, and locally correcting target positions in the two sets by using a bipartite graph matching algorithm with a weight value so as to enable the two sets to be matched with each other;

and (1.4) matching the camera and the radar target after the camera and the radar target are matched with each other.

2. The method for matching a camera with a radar target based on cross-correlation coefficient as claimed in claim 1, wherein in step (1.1), the two zero matrices with the same specification are specifically: and assigning 1 to the position of the target in the matrix and the periphery of the target, representing the target by 1 matrix of 3x3, performing line number expansion on the matrix where the camera target is located, and expanding zero matrixes with the same specification from top to bottom.

3. The method for matching a camera with a radar target based on a cross-correlation coefficient as claimed in claim 1, wherein in the step (1.2), the specific step of performing the overall correction on the radar target position information matrix and the camera target position information matrix is: and sliding the radar target position matrix up and down in an extended matrix of the camera target position information matrix, calculating the cross correlation coefficient of the two matrixes in the sliding window to obtain the sliding window position with the maximum cross correlation coefficient, and taking out the two matrixes at the overlapped part to obtain the radar target position information matrix and the camera target position information matrix after the integral correction.

4. The method for matching a camera with a radar target based on cross-correlation coefficient as claimed in claim 3, wherein the cross-correlation coefficient of the two matrices within the sliding window is calculated as follows:

wherein the content of the first and second substances,

where i and j represent the row number and column number of the matrix, x (i, j), y (i, j) represents the value of a certain point in the two matrices, and m_x，m_yAnd representing the corresponding mean value of the two matrixes, r representing the cross-correlation coefficient, M representing the row number of the matrixes, and N representing the column number of the matrixes.

5. The method for matching a camera with a radar target based on cross-correlation coefficient as claimed in claim 1, wherein in step (1.3), the weighted bipartite graph matching algorithm operates as follows:

(1.3.1) for each lane, dividing radar targets and camera targets into a set X and a set Y, calculating Euclidean distances between each radar target and all camera targets, and distributing weights according to the Euclidean distances, wherein the smaller the distance, the larger the weight, and otherwise, the smaller the distance, the smaller the weight, the total weight is 1;

the weight assignment is performed according to the following formula,

wherein λ is_i,jRepresents the second in the set XThe weight, X, between the i target and the jth target in the set Y_iAnd Y_kRepresents the ith target in the set X and a certain target in Y, | X_i-Y_kI represents X_iAnd Y_kThe Euclidean distance between two targets, and n represents the number of the targets;

(1.3.2) initializing a top mark, wherein for the set X, the top mark of each target is the maximum value of the weight values of the target and all targets in the set Y, and for all targets in the set Y, the top mark is 0;

(1.3.3) searching for a perfect match by using a Hungarian algorithm;

(1.3.4) if a complete match is not found, modifying the feasible benchmark value of the current target;

(1.3.5), repeating the steps (1.3.3) and (1.3.4) until a perfect match is found, i.e. one radar target matches one camera target and the sum of the weights is maximal.

6. The method of claim 1, wherein the camera and radar target matching based on cross-correlation coefficient,

the millimeter wave radar acquires target information in real time through a sensor thereof, wherein the target information comprises a radial distance between a target and the radar and an azimuth angle of the target relative to a radar detection normal direction, and two-dimensional coordinates of the target under a road surface coordinate system are obtained according to a geometric relationship by combining the two information and the known radar height;

the camera acquires image data in real time through a sensor of the camera, a target pixel coordinate is obtained through a target detection algorithm, and a two-dimensional coordinate of a target under a road surface coordinate system is obtained through a corresponding conversion algorithm.

Technical Field

The invention belongs to the technical field of data fusion of a multi-element sensor in intelligent traffic, and particularly relates to a camera and radar target matching method based on cross-correlation coefficients.

Background

Most of the existing traffic detection systems only adopt a single camera to acquire road information, but the acquired target position information has low precision, the detection effect of the system is easily influenced by weather such as rain, fog and the like, and the requirement is hardly met by using a sensor in an actual scene. With the development of intelligent traffic, more and more sensors are added into a traffic detection system, such as millimeter wave radar, geomagnetic, laser radar and other sensors, and the respective advantages of different sensors are utilized to perform multi-sensor data fusion, so that the detection performance of the system is greatly improved. The millimeter wave radar can detect the position and speed information of a target in real time, has strong environmental adaptability, can work all day long, but has the defect of being incapable of visualization.

In order to utilize the advantages of the camera and the millimeter wave radar, the targets acquired by the camera and the millimeter wave radar can be matched and information fusion can be carried out. And transmitting the position and speed information with higher precision acquired by the radar to a camera target, and displaying on an image to enable radar data to be visually displayed. The accuracy of the target position and the speed acquired by the camera is not high, the inclination angle of the camera is changed when the camera is erected on a road support for a long time, and some road surfaces are inclined, so that the target space position information acquired by the camera has integral deviation, the acquired position information is possibly far away from the actual position and is possibly close to the actual position, and target matching and information fusion can be performed only by performing necessary data processing.

Disclosure of Invention

Aiming at the problems, the invention provides a camera and radar target matching method based on cross correlation coefficients; the problem that when the camera is matched with a target acquired by the radar, the position information possibly has a large gap is solved.

The technical scheme of the invention is as follows: a camera and radar target matching method based on cross correlation coefficients is characterized by comprising the following specific steps:

step (1.1), at the same time, projecting coordinate values of targets detected by a millimeter wave radar and a camera in a road coordinate system into two zero matrixes with the same specification, so as to obtain a radar target position information matrix and a camera target position information matrix;

step (1.2), performing overall correction on the obtained radar target position information matrix and camera target position information matrix based on an algorithm of a cross-correlation coefficient;

step (1.3), for each monitored lane, setting a corrected radar target position information matrix and a corrected camera target position information matrix into two sets, and locally correcting target positions in the two sets by using a bipartite graph matching algorithm with a weight value so as to enable the two sets to be matched with each other;

and (1.4) matching the camera and the radar target after the camera and the radar target are matched with each other.

Further, in step (1.1), the two zero matrices with the same specification are specifically: and assigning 1 to the position of the target in the matrix and the periphery of the target, representing the target by 1 matrix of 3x3, performing line number expansion on the matrix where the camera target is located, and expanding zero matrixes with the same specification from top to bottom.

Further, in the step (1.2), the specific step of performing overall correction on the radar target position information matrix and the camera target position information matrix is: and sliding the radar target position matrix up and down in an extended matrix of the camera target position information matrix, calculating the cross correlation coefficient of the two matrixes in the sliding window to obtain the sliding window position with the maximum cross correlation coefficient, and taking out the two matrixes at the overlapped part to obtain the radar target position information matrix and the camera target position information matrix after the integral correction.

Further, the specific calculation process of the cross-correlation coefficient of the two matrices in the sliding window is as follows:

wherein the content of the first and second substances,

where i and j represent the row number and column number of the matrix, x (i, j), y (i, j) represents the value of a certain point in the two matrices, and m_x，m_yAnd representing the corresponding mean value of the two matrixes, r representing the cross-correlation coefficient, M representing the row number of the matrixes, and N representing the column number of the matrixes.

Further, in step (1.3), a specific operation method of the weighted bipartite graph matching algorithm is as follows:

(1.3.1) for each lane, dividing radar targets and camera targets into a set X and a set Y, calculating Euclidean distances between each radar target and all camera targets, and distributing weights according to the Euclidean distances, wherein the smaller the distance, the larger the weight, and otherwise, the smaller the distance, the smaller the weight, the total weight is 1;

the weight assignment is performed according to the following formula,

wherein λ is_i,jRepresents the weight between the ith target in the set X and the jth target in the set Y, X_iAnd Y_kRepresents the ith target in the set X and a certain target in Y, | X_i-Y_kI represents X_iAnd Y_kThe Euclidean distance between two targets, and n represents the number of the targets;

(1.3.2) initializing a top mark, wherein for the set X, the top mark of each target is the maximum value of the weight values of the target and all targets in the set Y, and for all targets in the set Y, the top mark is 0;

(1.3.3) searching for a perfect match by using a Hungarian algorithm;

(1.3.4) if a complete match is not found, modifying the feasible benchmark value of the current target;

(1.3.5), repeating the steps (1.3.3) and (1.3.4) until a perfect match is found, i.e. one radar target matches one camera target and the sum of the weights is maximal.

Further, the millimeter wave radar acquires target information in real time through a sensor of the millimeter wave radar, wherein the target information comprises a radial distance between a target and the radar and an azimuth angle of the target relative to a detection normal direction of the radar, and two-dimensional coordinates of the target under a road coordinate system are obtained according to a geometric relation by combining the two information and the known radar height;

the camera acquires image data in real time through a sensor of the camera, a target pixel coordinate is obtained through a target detection algorithm, and a two-dimensional coordinate of a target under a road surface coordinate system is obtained through a corresponding conversion algorithm.

The invention has the beneficial effects that: the invention provides a target matching method based on cross-correlation coefficients; at the same time in an actual scene, coordinate values of respective targets under a road coordinate system are obtained through targets detected by a radar and a camera through corresponding algorithms, and due to the fact that imaging mechanisms of the radar and the camera are different and the camera has different inclination angles in the long-term use process, the coordinate values obtained by the camera have local precision errors and integral position deviation and cannot be directly matched with radar targets; if the target coordinate value detected by the radar is taken as the standard, the target coordinate value detected by the camera has local precision error and integral remote or close, the camera target is integrally moved by utilizing the cross-correlation coefficient, so that the cross-correlation coefficient between the camera target and the coordinate value of the radar target is maximum, and the integral deviation is eliminated; a bipartite graph matching algorithm with a weight is used on each lane, so that the matching problem caused by local precision errors can be solved, and the precise matching of the target is completed; the invention solves the target matching problem of a plurality of sensors in the intelligent traffic monitoring system, has the advantages of small calculation amount and less time consumption on a common computer, and can meet the requirement of real-time property.

Drawings

FIG. 1 is a flow chart of the architecture of the present invention;

FIG. 2 is a schematic diagram showing the installation positions of the millimeter wave radar and the camera according to the present invention.

Detailed Description

In order to more clearly illustrate the technical solution of the present invention, the following detailed description is made with reference to the accompanying drawings:

as depicted in fig. 1; a camera and radar target matching method based on cross correlation coefficients is characterized by comprising the following specific steps:

step (1.1), at the same time, projecting coordinate values of targets detected by a millimeter wave radar and a camera in a road coordinate system into two zero matrixes with the same specification, so as to obtain a radar target position information matrix and a camera target position information matrix;

step (1.2), performing overall correction on the obtained radar target position information matrix and camera target position information matrix based on an algorithm of a cross-correlation coefficient;

step (1.3), for each monitored lane, setting a corrected radar target position information matrix and a corrected camera target position information matrix into two sets, and locally correcting target positions in the two sets by using a bipartite graph matching algorithm with a weight value so as to enable the two sets to be matched with each other;

and (1.4) matching the camera and the radar target after the camera and the radar target are matched with each other.

Further, in step (1.1), the two zero matrices with the same specification are specifically: and assigning 1 to the position of the target in the matrix and the periphery of the target, representing the target by 1 matrix of 3x3, performing line number expansion on the matrix where the camera target is located, and expanding zero matrixes with the same specification from top to bottom.

Further, in the step (1.2), the specific step of performing overall correction on the radar target position information matrix and the camera target position information matrix is: and sliding the radar target position matrix up and down in an extended matrix of the camera target position information matrix, calculating the cross correlation coefficient of the two matrixes in the sliding window to obtain the sliding window position with the maximum cross correlation coefficient, and taking out the two matrixes at the overlapped part to obtain the radar target position information matrix and the camera target position information matrix after the integral correction.

Further, the specific calculation process of the cross-correlation coefficient of the two matrices in the sliding window is as follows:

wherein the content of the first and second substances,

where i and j represent the row number and column number of the matrix, x (i, j), y (i, j) represents the value of a certain point in the two matrices, and m_x，m_yAnd representing the corresponding mean value of the two matrixes, r representing the cross-correlation coefficient, M representing the row number of the matrixes, and N representing the column number of the matrixes.

Further, in step (1.3), a specific operation method of the weighted bipartite graph matching algorithm is as follows:

(1.3.1) for each lane, dividing radar targets and camera targets into a set X and a set Y, calculating Euclidean distances between each radar target and all camera targets, and distributing weights according to the Euclidean distances, wherein the smaller the distance, the larger the weight, and otherwise, the smaller the distance, the smaller the weight, the total weight is 1;

the weight assignment is performed according to the following formula,

wherein λ is_i,jRepresents the weight between the ith target in the set X and the jth target in the set Y, X_iAnd Y_kRepresents the ith target in the set X and a certain target in Y, | X_i-Y_kI represents X_iAnd Y_kThe Euclidean distance between two targets, and n represents the number of the targets;

(1.3.2) initializing a top mark, wherein for the set X, the top mark of each target is the maximum value of the weight values of the target and all targets in the set Y, and for all targets in the set Y, the top mark is 0;

(1.3.3) searching for a perfect match by using a Hungarian algorithm;

(1.3.4) if a complete match is not found, modifying the feasible benchmark value of the current target;

(1.3.5), repeating the steps (1.3.3) and (1.3.4) until a perfect match is found, i.e. one radar target matches one camera target and the sum of the weights is maximal.

Further, the millimeter wave radar acquires target information in real time through a sensor of the millimeter wave radar, wherein the target information comprises a radial distance between a target and the radar and an azimuth angle of the target relative to a detection normal direction of the radar, and two-dimensional coordinates of the target under a road coordinate system are obtained according to a geometric relation by combining the two information and the known radar height;

the camera acquires image data in real time through a sensor of the camera, a target pixel coordinate is obtained through a target detection algorithm, and a two-dimensional coordinate of a target under a road surface coordinate system is obtained through a corresponding conversion algorithm.

The method comprises the following specific implementation steps:

1. projecting targets detected by a millimeter wave radar and a camera into two zero matrixes with the same specification according to road coordinate values, wherein the size of the matrixes is set according to an actual range, the maximum distance is 64 meters in the embodiment, the width of a double lane is 8 meters, the matrixes are set into zero matrixes of 21 rows and 64 lines, the distance of the matrixes from bottom to top is represented from near to far, the actual distance between the two rows is 1 meter, as shown in figure 2, the radar and the camera are arranged above the middle of a road, so that the 11 th column is taken as a central axis and represents a central lane line, and the actual distance between the two rows is 0.5 meter; quantizing the two-dimensional coordinates acquired by the millimeter wave radar and the camera, projecting the two-dimensional coordinates onto a matrix to make the two-dimensional coordinates 1, simultaneously making 8 surrounding numerical values 1, and representing a target by using a matrix of 3X 3; performing line number expansion on a matrix where a camera target is located, vertically expanding all-zero matrixes with the same specification, vertically sliding a millimeter wave radar target position information matrix in the expanded matrix of the camera target, calculating a cross correlation coefficient of two matrixes in a sliding window to obtain a sliding window position with the maximum cross correlation coefficient, and taking out the two matrixes at the overlapped part, thereby solving the problem of integral coordinate offset;

2. after the integral correction is completed, enabling a radar target and a camera target to be two sets on each lane, and matching the two sets of targets by using a bipartite graph matching algorithm with a weight value, so as to solve the problem of local deviation;

compared with a method for matching targets only by Euclidean distance, the method can better match the targets under the conditions that the camera generates different degrees of inclination angles and the number of the two detected targets is inconsistent, can solve the interference of the targets between adjacent lanes, and can adapt to more road conditions.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of embodiments of the present invention; other variations are possible within the scope of the invention; thus, by way of example, and not limitation, alternative configurations of embodiments of the invention may be considered consistent with the teachings of the present invention; accordingly, the embodiments of the invention are not limited to the embodiments explicitly described and depicted.