阅读说明：本技术 基于双目视觉和雷达的海上障碍物识别方法及系统 (Marine obstacle identification method and system based on binocular vision and radar ) 是由 曹琳 刘海林 袁健 李辉 张照文 胡一帆 吕斌 陈杰 于 2020-07-31 设计创作，主要内容包括：本公开提供了一种基于双目视觉和雷达的海上障碍物识别方法及系统,根据训练好的深度学习网络模型和时间融合后的双目视觉图像,得到目标物体分类标签,对目标物体进行三维重建后得到目标物体的双目视觉坐标系下的目标物体信息；根据时间融合后的雷达数据,得到雷达坐标系下的目标物体信息；对双目视觉坐标系下和雷达坐标系下的目标物体信息分别进行空间融合,将空间融合后的目标物体信息进行数据匹配,得到目标物体最优匹配对；根据最优匹配对将雷达检测的目标物体打上分类标签,得到最终的目标物体识别结果；克服了单一传感器存在的不足,在确保定位精度和识别准确率的前提下,更加适应在海上恶劣环境下实施。(The invention provides a method and a system for identifying marine obstacles based on binocular vision and radar, wherein a target object classification label is obtained according to a trained deep learning network model and a binocular vision image after time fusion, and target object information under a binocular vision coordinate system of a target object is obtained after three-dimensional reconstruction is carried out on the target object; obtaining target object information under a radar coordinate system according to the radar data after time fusion; respectively carrying out spatial fusion on target object information under a binocular vision coordinate system and a radar coordinate system, and carrying out data matching on the target object information after the spatial fusion to obtain an optimal matching pair of target objects; according to the optimal matching, a classification label is marked on a target object detected by the radar to obtain a final target object identification result; the defects of a single sensor are overcome, and the method is more suitable for being implemented in severe marine environments on the premise of ensuring the positioning precision and the identification accuracy.)

1. A marine obstacle identification method based on binocular vision and radar is characterized by comprising the following steps:

acquiring binocular vision images and radar data, and performing time fusion;

obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

obtaining position and speed information of the target object according to the radar data after time fusion;

respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

2. The binocular vision and radar-based marine obstacle recognition method of claim 1, wherein the spatial fusion is specifically: and converting the target object information in the binocular vision coordinate system and the target object information in the radar coordinate system into target object information in a ship body coordinate system.

3. The binocular vision and radar-based marine obstacle recognition method of claim 1, wherein the time fusion specifically is: and selecting the data cached in the previous frame of the radar when one frame of the binocular vision image is processed.

4. The binocular vision and radar-based marine obstacle recognition method of claim 1, wherein a target object classification label is obtained according to a trained deep learning network model and a binocular vision image after time fusion, and target object information under a binocular vision coordinate system of a target object is obtained after three-dimensional reconstruction of the target object, specifically:

preprocessing, distortion correction and parallel correction are carried out on the binocular vision image;

detecting and identifying the types of target objects in the left image and the right image by adopting a deep learning network model, and framing out an interest area;

performing edge detection on the image in the interest area to obtain an edge characteristic graph, evaluating pixel blocks by adopting a matching function, finally obtaining an optimal matching pixel block and calculating a parallax value;

obtaining a three-dimensional coordinate point cloud of a target object according to the parameters and the parallax value calibrated by the binocular camera;

and performing Kalman filtering on the identification code, the type, the position and the speed information of the target object to obtain the coordinate information and the category information of the target object.

5. The binocular vision and radar-based marine obstacle recognition method of claim 4, wherein the deep learning network model is a YOLOV4 convolutional neural network, and the matching function is edge detection of the image in the region of interest by using a laplacian operator to obtain an edge feature map by using a SAD feature point evaluation operator.

6. The binocular vision and radar-based marine obstacle recognition method of claim 1, wherein target object information in a radar coordinate system is obtained according to radar data after time fusion, and specifically:

analyzing the obtained radar data by a TCP/UDP packet;

carrying out pretreatment on the data after packet analysis to remove invalid targets, and obtaining the distance, the direction angle and the change rate of the distance of the targets under a polar coordinate system;

converting the obtained characteristic information of the target object into a ship body coordinate system;

performing Kalman filtering on the identity identification code, the position and the speed information of the target object after coordinate conversion to obtain target object information in a perception environment;

or, the optimal matching pair of the target object specifically comprises: adopting bidirectional matching, selecting a radar detection target with the highest matching degree as a candidate target for each target detected by binocular vision, selecting a binocular vision detection target with the highest matching degree as a candidate target for each target detected by radar, and determining as an optimal matching pair when a pair of target points are mutually optimally matched and are less than or equal to a matching threshold value;

or, the target optimal matching pair specifically includes: and establishing a matching feature vector for the measurement result of the target position and speed obtained by adopting binocular vision and a radar sensor, and adopting the constructed fusion matching evaluation function to take the target pair with the optimal fusion matching evaluation function value as the optimal matching pair.

7. A marine barrier identification system based on binocular vision and radar, comprising:

a data acquisition module configured to: acquiring binocular vision images and radar data, and performing time fusion;

a binocular vision recognition module configured to: obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

a radar data processing module configured to: obtaining position and speed information of the target object according to the radar data after time fusion;

a data fusion module configured to: respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

an identification module configured to: and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

8. A marine obstacle identification system based on binocular vision and radar is characterized by comprising an optical sensor, a radar sensor and a processor;

the optical sensor is configured to acquire binocular vision images, the radar sensor is configured to acquire radar data, the processor comprises:

a data acquisition module configured to: acquiring binocular vision images and radar data, and performing time fusion;

a binocular vision recognition module configured to: obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

a radar data processing module configured to: obtaining position and speed information of the target object according to the radar data after time fusion;

a data fusion module configured to: respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

an identification module configured to: and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

9. A medium having a program stored thereon, wherein the program, when executed by a processor, implements the steps in the binocular vision and radar-based marine obstacle recognition method of any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps in the binocular vision and radar-based marine obstacle recognition method of any one of claims 1 to 6.

Technical Field

The disclosure relates to the technical field of marine obstacle identification, in particular to a method and a system for identifying marine obstacles based on binocular vision and radar.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

The unmanned ship can be used for surveying and mapping, disaster relief, investigation, monitoring and the like on the sea, reduces the danger of manually executing tasks, and has important application in the field of civil and military industry. Autonomous navigation under the condition that beyond visual range and remote control cannot play a role needs to have the capability of environmental perception, so the environmental perception technology is the first link and key technology for realizing autonomous navigation of the unmanned ship. Compared with the land environment, the sea surface has the difficulties of fog, high salinity, high humidity, large wave fluctuation, approximate wave textures and the like, and the difficulties all put forward higher requirements for the sea environment perception. The environment sensing technology is specifically used for detecting, positioning and classifying and identifying the marine obstacles, so that the coordinates of ships, reefs, buoys, other floaters and the like under the body coordinate system of the unmanned ship can be obtained, and obstacle position information is provided for subsequent obstacle avoidance of the unmanned ship. At present, sensors for sensing the sea surface environment of an unmanned surface vehicle mainly include a vision sensor, a navigation radar, an Automatic Identification System (AIS) for a ship, and the like.

The inventor of the disclosure finds that the existing visual sensor comprises a visible light camera, an infrared camera and the like, the visible light camera seriously reduces the imaging quality and is difficult to work normally in severe weather such as rain, snow, fog and the like, the infrared camera has the all-weather characteristic relative to the visible light camera, but the infrared image texture information is weak, and the target identification accuracy is low; AIS has detection capability only for vessels with AIS transponders installed, but cannot sense obstacles such as transponder-free vessels and buoys; the navigation radar has a long detection distance but low sensing precision, and a dead zone for detecting the whole ship exists, so that the unmanned ship loses the sensing capability on a short-distance obstacle; compared with a camera, the laser radar has the advantages of high ranging precision and capability of acquiring point cloud data of surrounding obstacles in real time, but has the defects of low remote detection precision and low precision due to the fact that the laser radar is easily influenced by the severe sea environment.

Disclosure of Invention

In order to solve the defects of the prior art, the method and the system for identifying the marine barrier based on binocular vision and radar provide category information through the binocular camera, the defect that a millimeter wave radar cannot detect the category of a target is overcome, the millimeter wave radar is high in ranging precision, the defect that a visual sensor is low in ranging precision is overcome, the defects of a single sensor are overcome, and the method and the system are more suitable for being implemented under a severe marine environment on the premise that the positioning precision and the identification accuracy are ensured.

In order to achieve the purpose, the following technical scheme is adopted in the disclosure:

the first aspect of the disclosure provides a marine obstacle identification method based on binocular vision and radar.

A marine obstacle identification method based on binocular vision and radar comprises the following steps:

acquiring binocular vision images and radar data, and performing time fusion;

obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

obtaining position and speed information of the target object according to the radar data after time fusion;

respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

A second aspect of the present disclosure provides a binocular vision and radar-based marine obstacle identification system.

A binocular vision and radar based marine obstacle identification system comprising:

a data acquisition module configured to: acquiring binocular vision images and radar data, and performing time fusion;

a binocular vision recognition module configured to: obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

a radar data processing module configured to: obtaining position and speed information of the target object according to the radar data after time fusion;

a data fusion module configured to: respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

an identification module configured to: and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

A third aspect of the present disclosure provides a binocular vision and radar-based marine obstacle identification system.

A marine obstacle identification system based on binocular vision and radar comprises an optical sensor, a radar sensor and a processor;

the optical sensor is configured to acquire binocular vision images, the radar sensor is configured to acquire radar data, the processor comprises:

a data acquisition module configured to: acquiring binocular vision images and radar data, and performing time fusion;

a binocular vision recognition module configured to: obtaining a target object classification label according to the trained deep learning network model and the time-fused binocular vision image, and obtaining position and speed information of the target object after three-dimensional reconstruction of the target object;

a radar data processing module configured to: obtaining position and speed information of the target object according to the radar data after time fusion;

a data fusion module configured to: respectively carrying out spatial fusion on the position and speed information of the target object obtained by binocular vision and radar, and matching the data after the spatial fusion to obtain an optimal matching pair of the target object;

an identification module configured to: and according to the optimal matching, a classification label is marked on a target object detected by the radar, and target coordinate information and the target object classification label obtained by the radar detection are used as a target object identification result.

A fourth aspect of the present disclosure provides a medium having stored thereon a program that, when being executed by a processor, performs the steps in the binocular vision and radar-based marine obstacle recognition method according to the first aspect of the present disclosure.

A fifth aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a program stored on the memory and executable on the processor, the processor implementing the steps in the binocular vision and radar-based marine obstacle identification method according to the first aspect of the present disclosure when executing the program.

Compared with the prior art, the beneficial effect of this disclosure is:

1. according to the method, the system, the medium and the electronic equipment, the category information is provided through the binocular camera, the defect that a millimeter wave radar cannot detect the category of the target is overcome, the distance measurement precision of the millimeter wave radar is high, the defect that the distance measurement precision of a visual sensor is low is overcome, the defects of a single sensor are overcome, and the method, the system, the medium and the electronic equipment are more suitable for being implemented in a severe marine environment on the premise that the positioning precision and the identification accuracy are ensured.

2. According to the method, the system, the medium and the electronic equipment, the target information measured by the two sensors can be aligned and fused without jointly calibrating the vision module and the radar module, and finally the classification information and the accurate three-dimensional coordinates of the target object are obtained.

3. According to the method, the system, the medium and the electronic equipment, the binocular stereo vision module improves a feature matching algorithm and adds a primary positioning algorithm based on a real-time convolutional neural network YOLOV4, and the real-time performance of the binocular stereo vision module algorithm is greatly improved.

4. According to the method, the system, the medium and the electronic equipment, the Kalman filtering algorithm is designed for the detection data of the two sensor modules, and the accuracy and the robustness of the sensor are improved.

5. According to the method, the system, the medium and the electronic equipment, the millimeter wave radar is adopted for radar detection, the method has good remote measurement precision, can penetrate sea fog, is not influenced by rain, snow and dust environments, has the characteristics of small size, light weight, low cost, interference resistance and the like, and is more suitable for all-weather work in severe marine environments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.

Fig. 1 is a schematic flow chart of a binocular vision and radar-based marine obstacle identification method provided in embodiment 1 of the present disclosure.

Fig. 2 is a schematic diagram of a transformation matrix between coordinate systems in sensor composition and spatial fusion provided in embodiment 1 of the present disclosure.

Detailed Description

The present disclosure is further described with reference to the following drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.