阅读说明：本技术 具有多信息融合和光流测速功能的超小型双光成像系统 (Subminiature dual-light imaging system with multi-information fusion and light stream speed measurement functions ) 是由 徐保树 张程硕 魏东溟 陆朝阳 李凯 周毅伟 于 2020-06-24 设计创作，主要内容包括：本发明属于无人机携带的双光成像系统技术领域,具体涉及一种具有多信息融合和光流测速作用的超小型双光成像系统,本系统应用在飞行器上,包括红外成像模块、可见光成像模块、超声波测距模块、惯性测量模块和核心图像处理器,核心图像处理器包括多信息融合模块、光流测速模块、OSD模块和视频压缩模块。本发明将红外成像技术、可见光成像技术与信息融合技术一体化设计,创新性地在双光系统中增加光流测速和人脸识别功能,实现超小型双光成像系统,进一步拓展小型侦察式无人机在复杂环境中的作业能力,为地面作战单位在隐匿的环境下充分的扩大视野,使行动人员能够随时随地保持态势感知、威胁探测和监视能力。(The invention belongs to the technical field of double-light imaging systems carried by unmanned aerial vehicles, and particularly relates to a subminiature double-light imaging system with multi-information fusion and optical flow speed measurement functions. The invention integrates the infrared imaging technology, the visible light imaging technology and the information fusion technology, innovatively adds the functions of optical flow speed measurement and face recognition in a double-optical system, realizes a subminiature double-optical imaging system, further expands the operation capability of the small reconnaissance type unmanned aerial vehicle in a complex environment, fully expands the visual field of a ground operation unit in a hidden environment, and enables action personnel to keep situation perception, threat detection and monitoring capability anytime and anywhere.)

1. The subminiature dual-light imaging system with the functions of multi-information fusion and light stream speed measurement is characterized by being applied to an aircraft and comprising an infrared imaging module, a visible light imaging module, an ultrasonic ranging module, an inertia measurement module and a core image processor, wherein the core image processor comprises the multi-information fusion module, the light stream speed measurement module, an OSD module and a video compression module, the infrared imaging module is connected with the input end of the multi-information fusion module, the visible light imaging module is respectively connected with the multi-information fusion module and the light stream speed measurement module, the ultrasonic ranging module and the inertia measurement module are respectively connected with the input end of the light stream speed measurement module, the light speed flow measurement module is respectively connected with the input end of the multi-information fusion module and the input end of the OSD module in the core image processor, the output end of the OSD module is connected with the multi-information fusion module, and the output end of, and the video compression module sends the processed image information out through a network.

2. The system of claim 1, further comprising a face recognition module connected between the multi-information fusion module and the OSD module, wherein the face recognition module comprises a face image feature extraction module, a face image matching and recognition module, and a face image archive storage module, the face image feature extraction module is connected to the multi-information fusion module and is configured to extract the captured face features, and the face image matching and recognition module matches the captured face features with the face images stored in the face image archive storage module for face recognition, and then transmits the recognition results to the multi-information fusion module through the OSD module for fusion.

3. The subminiature dual-light imaging system with multi-information fusion and optical flow velocimetry of claim 1, further comprising a sound pick-up connected to the input of the multi-information fusion module for fusion of video image capture audio.

4. The subminiature dual-light imaging system with multi-information fusion and optical flow velocimetry of claim 1, further comprising a searchlight coupled to the visible light imaging module for providing a light source to the visible light imaging module at low light levels.

5. The ultra-small dual-light imaging system with multi-information fusion and optical flow velocity measurement functions as claimed in claim 1, wherein the multi-information fusion module fuses the infrared image inputted by the infrared imaging module and the optical image inputted by the visible light imaging module in a manner that the infrared image inputted by the infrared imaging module is superimposed and embedded into the visible light image inputted by the visible light imaging module or the infrared image and the visible light image are superimposed on a pixel level.

6. The subminiature dual-light imaging system with multi-information fusion and optical flow velocimetry of claim 5, wherein the step of superimposing the infrared image and the visible image at pixel level comprises:

501) Aligning optical axes of the visible light imaging module and the infrared imaging module by using an off-line calibration method;

502) then, pixel matching of the visible light image and the infrared image is realized through a bilinear difference method;

503) calculating the significance of the matched visible light image block and infrared image block;

504) and in the process of superposing the visible light pixels and the infrared pixels, high weights are distributed to the pixels with high significance, and the superposition on the pixel level is completed.

7. The subminiature dual-optical imaging system with multi-information fusion and optical flow velocimetry of claim 1, wherein the infrared imaging module has a size of 10.50 x 12.701 x 7.14mm, a weight of 0.9g, and is configured with an athermal infrared optical lens with a horizontal field angle of 56 ° HFOV, a focal length of 1.8mm and an F-number of 1.1; the visible light imaging module is an automatic focusing camera module, the size of the automatic focusing camera module is 8.50 multiplied by 8.5 multiplied by 5.03mm, the mass of the automatic focusing camera module is less than 0.5g, and a visible light automatic focusing lens with the visual angle of 60-69 degrees, the focal length of 3.34mm and the F number of 2.8 is configured; the sound pickup is an MEMS microphone; the ultrasonic ranging module comprises an ultrasonic transmitter, an ultrasonic receiver and a control circuit; the inertial measurement module is a MEMS gyroscope.

Technical Field

The invention belongs to the technical field of double-light imaging systems carried by unmanned aerial vehicles, and particularly relates to a subminiature double-light imaging system with multi-information fusion and optical flow speed measurement functions and an optical flow speed measurement method.

Background

With the rapid development of electronic technology, the small unmanned aerial vehicle is widely applied to various fields such as security monitoring, search and rescue, police law enforcement, forest fire prevention, electric power line patrol, environmental protection and scientific research by means of high maneuvering performance, good environmental adaptability and economy. Technological development is so far, except the development of unmanned aerial vehicle platform itself, the continuous promotion of the continuous abundance and the performance of payload has also greatly expanded unmanned aerial vehicle's application scope. The unmanned aerial vehicle mainly acquires ground image information through imaging detection equipment to complete tasks such as monitoring and reconnaissance. In the present situation, unmanned airborne equipment is restricted by the technical characteristics of unmanned aerial vehicles, and has different requirements from other carrier equipment:

1. the unmanned aerial vehicle has small volume, and the size and the shape of a circuit board of airborne equipment of the unmanned aerial vehicle are limited;

2. because the load capacity of the unmanned aerial vehicle, especially a small unmanned aerial vehicle, is limited, the airborne mission load needs to meet the light weight requirement so as to prolong the endurance time of the unmanned aerial vehicle;

3. the power supply provided by the unmanned aerial vehicle is limited, and cannot provide a high-power supply as the ground equipment, so that the airborne task load is required to be low in power consumption.

Based on the characteristics, the unmanned aerial vehicle has strict requirements on the weight, the volume and the power consumption of the load, and the subminiature load is particularly important.

Due to the fact that different imaging detection devices have different factors such as imaging mechanisms, working wave bands and working environments, when different types of sensors image the same target or scene, the acquired target or scene information is different. The visible light sensor can obtain rich and fine visible light images, but the imaging quality of the visible light sensor is seriously influenced when the illumination condition is poor; the infrared detector has better target detection capability under poor illumination conditions such as night and the like, but the detail expression capability of the infrared image formed by the infrared detector is poor. Particularly for low-resolution infrared imaging equipment, the number of detector pixels is limited, the edge of an output infrared image is fuzzy, and the development of the infrared imaging equipment towards high-performance light weight is limited. In addition, if unmanned aerial vehicle appeared losing the star condition in the flight process, the gesture was dispersed easily, can directly lead to the flight mission to fail even the aircraft crash.

Disclosure of Invention

In order to solve the problems, the invention provides a subminiature dual-light imaging system with multi-information fusion and optical flow velocity measurement functions and an optical flow velocity measurement method.

The invention is realized in this way, provides a subminiature double-light imaging system with multi-information fusion and light stream speed measurement functions, which is applied to an aircraft and comprises an infrared imaging module, a visible light imaging module, an ultrasonic ranging module, an inertia measurement module and a core image processor, wherein the core image processor comprises a multi-information fusion module, a light stream speed measurement module, an OSD module and a video compression module, the infrared imaging module is connected with the input end of the multi-information fusion module, the visible light imaging module is respectively connected with the multi-information fusion module and the light stream speed measurement module, the ultrasonic ranging module and the inertia measurement module are respectively connected with the input end of the light stream speed measurement module, in the core image processor, the light speed flow measurement module is respectively connected with the input end of the multi-information fusion module and the OSD module, the output end of the OSD module is connected with the multi-information fusion module, and the output end of the, and the video compression module sends the processed image information out through a network.

Preferably, the system further comprises a face recognition module, the face recognition module is connected between the multi-information fusion module and the OSD module, the face recognition module comprises a face image feature extraction module, a face image matching and recognition module and a face image archive storage module, the face image feature extraction module is connected with the multi-information fusion module and is used for extracting captured face features, then the face image matching and recognition module matches the captured face features with face images stored in the face image archive storage module for face recognition, and then recognition results are transmitted to the multi-information fusion module through the OSD module for fusion.

Further preferably, the system also comprises a sound pick-up, wherein the sound pick-up is connected with the input end of the multi-information fusion module and is used for collecting and fusing audio for the video image.

Further preferably, the system further comprises a searchlight, wherein the searchlight is connected with the visible light imaging module and is used for providing a light source for the visible light imaging module under low illumination.

Further preferably, the multi-information fusion module fuses the infrared image input by the infrared imaging module and the light image input by the visible light imaging module in a manner that the infrared image input by the infrared imaging module is overlapped and embedded into the visible light image input by the visible light imaging module or the infrared image and the visible light image are overlapped on a pixel level.

Further preferably, the specific steps of superimposing the infrared image and the visible light image at the pixel level are as follows:

501) aligning optical axes of the visible light imaging module and the infrared imaging module by using an off-line calibration method;

502) then, pixel matching of the visible light image and the infrared image is realized through a bilinear difference method;

503) calculating the significance of the matched visible light image block and infrared image block;

504) and in the process of superposing the visible light pixels and the infrared pixels, high weights are distributed to the pixels with high significance, and the superposition on the pixel level is completed.

Further preferably, the infrared imaging module has a size of 10.50 × 12.701 × 7.14mm, a weight of 0.9g, and an athermal infrared optical lens with a horizontal field angle of 56 ° HFOV, a focal length of 1.8mm, and an F-number of 1.1; the visible light imaging module is an automatic focusing camera module, the size of the automatic focusing camera module is 8.50 multiplied by 8.5 multiplied by 5.03mm, the mass of the automatic focusing camera module is less than 0.5g, and a visible light automatic focusing lens with the visual angle of 60-69 degrees, the focal length of 3.34mm and the F number of 2.8 is configured; the sound pickup is an MEMS microphone; the ultrasonic ranging module comprises an ultrasonic transmitter, an ultrasonic receiver and a control circuit; the inertial measurement module is a MEMS gyroscope.

Compared with the prior art, the invention has the advantages that:

the invention integrates the infrared imaging technology, the visible light imaging technology and the information fusion technology, innovatively adds the functions of optical flow speed measurement and face recognition in a double-optical system, realizes a subminiature double-optical imaging system, further expands the operation capability of the small reconnaissance type unmanned aerial vehicle in a complex environment, fully expands the visual field of a ground operation unit in a hidden environment, and enables action personnel to keep situation perception, threat detection and monitoring capability anytime and anywhere.

Drawings

FIG. 1 is a block diagram of the system;

FIG. 2 is a diagram of a face recognition module;

fig. 3 is a composition diagram of an ultrasonic ranging module.

FIG. 4 is a schematic diagram illustrating an infrared image input by an infrared imaging module in a multi-information fusion module being superimposed and embedded in a visible light image input by a visible light imaging module;

fig. 5 is a schematic diagram of superposition of an infrared image and a visible light image at a pixel level in the multi-information fusion module.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, the present invention provides a subminiature dual-optical imaging system with multi-information fusion and optical flow velocity measurement functions, which is applied to an aircraft, and comprises an infrared imaging module, a visible light imaging module, an ultrasonic ranging module, an inertia measurement module and a core image processor, wherein the core image processor comprises a multi-information fusion module, an optical flow velocity measurement module, an OSD module and a video compression module, the infrared imaging module is connected with the input end of the multi-information fusion module, the visible light imaging module is respectively connected with the multi-information fusion module and the optical flow velocity measurement module, the ultrasonic ranging module and the inertia measurement module are respectively connected with the input end of the optical flow velocity measurement module, the optical velocity flow measurement module is respectively connected with the input end of the multi-information fusion module and the OSD module, the output end of the OSD module is connected with the multi-information fusion module, and the output end of the multi-information fusion module, and the video compression module sends the processed image information out through a network.

Referring to fig. 2, the core image processor includes a face recognition module, the face recognition module is connected between the multi-information fusion module and the OSD module, the face recognition module includes a face image feature extraction module, a face image matching and recognition module, and a face image archive storage module, the face image feature extraction module is connected with the multi-information fusion module for extracting captured face features, and then the face image matching and recognition module matches the captured face features with the face images stored in the face image archive storage module for face recognition, and then transmits recognition results to the multi-information fusion module through the OSD module for fusion.

In order to increase audio and improve the visible light image acquisition effect, the system further comprises a sound pick-up and a searchlight, wherein the sound pick-up is connected with the input end of the multi-information fusion module and is used for fusing the video image acquisition audio. The searchlight is connected with the visible light imaging module and is used for providing a light source for the visible light imaging module under low illumination.

An infrared imaging module: an infrared video image of 9Hz resolution of 160 x 120 is provided, an HFOV with a horizontal field angle of 56 DEG is configured, and an athermal infrared optical lens with a focal length of 1.8mm and an F number of 1.1 is provided. Under the conditions that the visibility is more than 5km, the relative humidity is less than 70 percent and the target background temperature difference is more than 5k, the personnel detection distance of 60m and the personnel identification distance of 30m can be realized. The infrared imaging module is connected with the core image processor through the SPI interface, and can realize multiple intelligent infrared display by utilizing the powerful image processing function of the infrared imaging module, and the infrared imaging module comprises up to 9 pseudo-color templates. The diversified infrared intelligent display mode can meet the detection requirements of different tasks in different environments. The infrared imaging module can also provide temperature data which cannot be sensed by naked eyes, and the rapid analysis and decision-making on site are facilitated. The temperature measurement mode comprises point temperature measurement, regional temperature measurement, isotherm display and the like, and is mainly used for temperature-related searching and detecting tasks, such as application fields of city inspection, forest fire prevention, dangerous object detection, fire fighting, search and rescue and the like. The infrared imaging module has the size of only 10.50 multiplied by 12.701 multiplied by 7.14mm, the weight is 0.9g, the power consumption is 140mW, and powerful support is provided for the miniaturization and the light weight of the whole subminiature dual-optical system.

Visible light imaging module: the visible light imaging module is composed of an automatic focusing camera module. The support outputs a maximum of 500-ten-thousand-pixel image (2592x1944 resolution), 720p HD video at 60fps, and 1080p HD video at 30 fps. The visual light automatic focusing lens with the field angle of 60-69 degrees, the focal length of 3.34mm and the F number of 2.8 is configured, so that the personnel detection distance of more than 200m, the personnel identification distance of more than 100m and the face identification distance of more than 25m can be realized. The visible light imaging module can perform self-adaptive adjustment on brightness, contrast chroma and saturation by utilizing the powerful image processing function of the core processor, so that a night scene mode and a defogging algorithm are realized. The automatic focusing camera module is only 8.50 multiplied by 8.5 multiplied by 5.03mm in size, the mass is less than 0.5g, and the power is between 150mW and 200mW during working.

The ultrasonic ranging module: referring to fig. 3, the ultrasonic ranging module includes an ultrasonic transmitter, an ultrasonic receiver, and a control circuit, providing 2-400 cm of non-contact distance sensing.

An inertia measurement module: is a MEMS gyroscope.

A sound pick-up: is a MEMS microphone.

Searchlight: the searchlight uses the LED lamp pearl of low-power consumption to throw light on, uses under the illumination condition is unsatisfactory, needs visible light video output's condition. LED lamp pearl consumption 0.5W of low-power consumption.

In the core image processor:

the multi-information fusion module: the multi-information fusion module realizes effective fusion of visible light and infrared imaging, video and sound. The fusion mode of the visible light image and the infrared image is divided into two modes of picture-in-picture and pixel level fusion, wherein the picture-in-picture fusion carries out on-screen display by embedding the infrared into the visible light image, as shown in fig. 4. The pixel-level fusion aligns the optical axes of the infrared image and the visible light image through image calibration, and then superposes the pixel level according to the information quantity represented by the visible light image and the infrared image to obtain a video image with richer information, as shown in fig. 5, the specific fusion method is as follows:

501) aligning optical axes of the visible light imaging module and the infrared imaging module by using an off-line calibration method;

502) then, pixel matching of the visible light image and the infrared image is realized through a bilinear difference method;

503) calculating the significance of the matched visible light image block and infrared image block;

504) and in the process of superposing the visible light pixels and the infrared pixels, high weights are distributed to the pixels with high significance, and the superposition on the pixel level is completed.

The multi-information fusion module utilizes complementary information provided by the modules, and the fused image contains more comprehensive and abundant information, so that the multi-information fusion module is more in line with the visual characteristics of people or machines and more beneficial to further analysis and processing of the image and automatic target identification; under adverse environmental conditions (e.g., smoke, dust, cloud, fog, rain, etc.), detection performance may be improved by multi-sensor image fusion. For example, in smoke and fog environments, the quality of visible light images is poor (even objects cannot be seen clearly), and the obtained infrared images have strong penetration capacity to smoke and fog, and clear images can be obtained despite some attenuation of signals.

The optical flow speed measurement module:

and processing the surface images of the object continuously acquired by the visible light imaging module through an optical flow tracking algorithm, and analyzing and processing the generated image digital matrix. Because two adjacent images always have the same characteristics, the average motion of the surface characteristics of the object can be judged by comparing the position change information of the characteristic points, the analysis result is finally converted into two-dimensional coordinate offset and is stored in a specific register in the form of pixel number, and the detection of the moving object is realized. Because the unit of the visible light image is pixel, the speed calculated by the optical flow method is not provided with scale, the distance of the plane movement from the camera is measured by the ultrasonic module, and the pixel movement is converted into real movement. In order to enable the optical flow velocity measurement module to be used on an unmanned aerial vehicle, a plane represented by an image is found, the ultrasonic module and the two sensors of the inertial measurement element are fused on an algorithm to obtain stable velocity output, and the optical flow velocity measurement algorithm flow is as follows:

801) the visible light imaging module collects the ith frame image, selects a characteristic salient target in the image, and extracts a characteristic point set F of the target _i；

802) The visible light imaging module collects the (i + 1) th frame image, and the feature point set F matched with the feature point set of the (i + 1) th frame image is searched in the (i + 1) th frame image_i+1；

803) According to the obtained feature point set F_iAnd F_i+1Calculating an average optical flow vector by the matched feature point pairs:

whereinSet of feature points F_iAnd F_i+1The coordinate value of the matched characteristic point;

804) and obtaining the relative speed of the aircraft to the ground according to the average optical flow vector:

wherein T is the time interval between two adjacent frames i, i + 1;

805) according to the distance Z between the visible light imaging module and the ground target measured by the ultrasonic ranging module, converting the relative speed of the aircraft to the ground into an absolute speed:

wherein f is the focal length of the visible light lens;

806) since the aircraft shakes itself during the flight, the deviation caused by the shake is measured by the inertial measurement unit, and the deviation is eliminated from the absolute speed calculated in step 805), so as to obtain the target ground absolute speed.

A face recognition module:

the face recognition module comprises a face image archive, a face image feature extraction module and a face image matching and recognition module. The face image archive uses a camera to collect face image files of unit personnel, and generates feature templates of the face image files to be stored in a database. The face image feature extraction module adopts a regional feature analysis method, integrates a computer image processing technology and a biological statistics principle, extracts face feature points from an image returned by the pan-tilt camera by using the computer image processing technology, and establishes a face feature template by using the biological statistics principle. The face image matching and identifying module compares and identifies the extracted face characteristic template with the characteristic templates stored in the database, and finally judges and confirms the identity information of the personnel according to the compared similar values. The whole work flow of face recognition is as follows:

2) The subminiature dual-optical system captures and tracks the human face through the visible light imaging module;

3) the visible light imaging module inputs the collected image into the face recognition module;

4) the face recognition module extracts feature points of the input face picture/image and establishes a feature template;

5) and identifying and comparing the extracted face feature template with the feature templates in the database so as to confirm the identity of the personnel.

An OSD module: the speed measured by the optical flow speed measuring module is input into the multi-information fusion module, and the speed information is embedded into the visual flow for offline analysis.

A video compression module: and carrying out compression coding on the fused video and transmitting the video to the outside.

The subminiature dual-light imaging system with the multi-information fusion and the optical flow velocity measurement functions has the following main performance indexes:

1. infrared index:

the field angle: 56 ° × 48 °

Resolution ratio: 160 (h). times.120 (v)

Frame rate: 9HZ

NETD<50mk

Temperature measurement precision: 5 degree @25 deg.C

2. Visible light index:

the field angle: 60-69 deg

Highest resolution: 2592x1944

Frame rate: 1080p30Hz

720p 60Hz

3. Weight: <30g

4. Volume: 72 (length) × 36 (width) × 20 (height) mm

5. Power consumption: <2W

6. Scout performance:

the target detection distance is more than or equal to 200m (visible) and more than or equal to 60m (infrared)

The target detection distance is more than or equal to 200m (visible) and more than or equal to 30m (infrared)

The face recognition distance is more than or equal to 25m (visible).