阅读说明：本技术 被动式红外热成像3d摄像机及摄像方法 (Passive infrared thermal imaging 3D camera and shooting method ) 是由 赵照 于 2021-08-16 设计创作，主要内容包括：本申请公开了一种被动式红外热成像3D摄像机及摄像方法,通过偏振光热成像传感器在不同焦距下获得多个红外偏振图像对应的原始数据,图像处理器对原始数据进行图像重建,获得多个含有空间信息的红外偏振图像,对红外偏振图像进行特征点匹配,根据焦距和像点向量计算深度信息,获得3D图像。本申请提供的被动式红外热成像3D摄像机及摄像方法无需采用大功率光源,对于人体无任何影响,而且采用偏振光热成像传感器可以全天候工作,检测距离长达200米以上,满足自动驾驶、智能机器人或无人机等高端应用领域要求。(The application discloses passive infrared thermal imaging 3D camera and camera shooting method obtains the corresponding raw data of a plurality of infrared polarization images under different focuses through polarized light thermal imaging sensor, and image processor carries out image reconstruction to raw data, obtains a plurality of infrared polarization images that contain spatial information, carries out the characteristic point to infrared polarization image and matches, calculates depth information according to focus and image point vector, obtains the 3D image. The application provides a passive form infrared thermal imaging 3D camera and camera shooting method need not to adopt high-power light source, does not have any influence to the human body, adopts polarized light thermal imaging sensor can work in all weather moreover, and the detection distance is up to more than 200 meters, satisfies high-end application field requirements such as autopilot, intelligent robot or unmanned aerial vehicle.)

1. A passive infrared thermal imaging 3D camera, the camera comprising:

an optical system configured to focus the thermal radiation and adjust a focal length;

a polarized photothermographic sensor comprising a plurality of pixel elements comprising polarizing means, each pixel element configured to generate an electrical signal in response to detection of thermal radiation; a readout circuit configured to read out the electrical signal; an ADC configured to convert the electrical signal into infrared polarization image raw data; an interface circuit configured to transmit the infrared polarized image raw data to an image processor;

and the image processor is configured to reconstruct the original data of the infrared polarization image to obtain an infrared polarization image, perform depth calculation on the infrared polarization image and acquire a 3D image.

2. The camera of claim 1, wherein the plurality of pixels comprising polarizing means are arranged in a diamond-shaped staggered array.

3. The camera of claim 1, wherein the polarizing means of each four mutually adjacent picture elements of the plurality of picture elements comprising polarizing means have a different respective polarization angle.

4. A camera according to claim 3, wherein the polarizing means has a polarization angle of 0 °, 45 °, 90 ° or 135 °.

5. The camera of claim 1, wherein the polarizing means is integrated on the pixel by a MEMS process.

6. The camera of claim 1, wherein the pixel thermosensitive material comprises a ferroelectric material, vanadium oxide, graphene, mercury cadmium telluride, quantum wells, or class two superlattices or quantum dots.

7. An image pickup method, comprising:

acquiring first infrared polarization image original data under a first focal length;

acquiring original data of a second infrared polarization image at a second focal length; wherein the first focal length is smaller than the second focal length;

respectively reconstructing the first infrared polarization image original data and the second infrared polarization image original data to obtain a first infrared polarization image and a second infrared polarization image;

and performing feature matching on the first infrared polarization image and the second infrared polarization image, calculating depth information, and acquiring a 3D image.

8. The method according to claim 7, wherein the computing depth information is specifically:

;

wherein Z is the distance value from the camera to the plane of the object point, f₁And f₂Respectively representing a first focal length and a second focal length, l₁And l₂And the two object points respectively represent the vector size of two image points of the two object points in the plane space vertical to the optical axis under the second focal length and the first focal length.

Technical Field

The application relates to the field of infrared thermal imaging, in particular to a passive infrared thermal imaging 3D camera.

Background

3D imaging technology is one of the more popular research directions in recent years, and 3D cameras are also used in many fields, such as gesture recognition, robots, and the like.

The double-lens camera can realize a three-dimensional effect, but the structure system is complex, the algorithm is very complex, and the double-lens camera has no large-scale popularization and application for many years, and is only applied to specific occasions such as 3D movies and the like;

the active structure optical camera can also realize the stereoscopic effect, is mainly applied to occasions with very short distance, and has specific requirements on algorithm and hardware structure, so that the trend of being replaced gradually along with the appearance of the active TOF camera is gradually realized.

An active TOF (Time of Flight) camera is a main 3D imaging camera at present, and mainly comprises two products of wired scanning and surface array, wherein the active TOF camera adopts a Time of Flight method for non-contact distance measurement, transmits emitted light waves to a target continuously, receives light returned from an object by using a sensor, and obtains the distance of the target object by detecting the Time of Flight of the light waves, so that the target can be identified and tracked quickly, the depth information can be used for segmenting, marking, identifying, tracking and the like of a target image, and the application of three-dimensional modeling and the like can be finished by further deepening treatment without auxiliary work of scanning equipment.

However, due to the problems of power of the actively-emitted laser light source, eye protection and the like, particularly the high-power light source has high cost, and the application of the high-power light source is mainly focused on the application of mobile phones, tablet computers, internet of things or other intelligent hardware, and the high-end application fields of automatic driving, intelligent robots, unmanned aerial vehicles and the like have certain difficulty.

Disclosure of Invention

In view of the above, the present application provides a passive infrared thermal imaging 3D camera and an imaging method.

In order to solve the technical problem, the following technical scheme is adopted in the application:

in a first aspect, a passive infrared thermal imaging 3D camera is provided, wherein the camera includes:

an optical system configured for transmitting thermal radiation and adjusting a focal length;

a polarized photothermographic sensor comprising a plurality of pixel elements comprising polarizing means, each pixel element configured to generate an electrical signal in response to detection of thermal radiation; a readout circuit configured to read out the electrical signal; an ADC configured to convert the electrical signal into infrared polarization image raw data; an interface circuit configured to transmit the infrared polarized image raw data to an image processor;

and the image processor is configured to reconstruct the original data of the infrared polarization image to obtain an infrared polarization image, perform depth calculation on the infrared polarization image and acquire a 3D image.

Preferably, the plurality of pixels containing the polarization devices are arranged in a diamond-shaped staggered array mode.

Preferably, the polarizing means of every four mutually adjacent picture elements of said plurality of picture elements comprising polarizing means have different respective polarization angles.

Preferably, the polarizing means has a polarization angle of 0 °, 45 °, 90 ° or 135 °.

Preferably, the polarizing means are integrated on the picture element by a MEMS process.

Preferably, the pixel thermosensitive material comprises a ferroelectric material, vanadium oxide, graphene, mercury cadmium telluride, quantum wells or two types of superlattices or quantum dots

In a second aspect, there is provided an imaging method, comprising:

acquiring first infrared polarization image original data under a first focal length;

acquiring original data of a second infrared polarization image at a second focal length; wherein the first focal length is smaller than the second focal length;

respectively reconstructing the first infrared polarization image original data and the second infrared polarization image original data to obtain a first infrared polarization image and a second infrared polarization image;

and performing feature matching on the first infrared polarization image and the second infrared polarization image, calculating depth information, and acquiring a 3D image.

Preferably, the calculating depth information specifically includes:

;

wherein Z is the distance value from the camera to the plane of the object point, f₁And f₂Respectively representing a first focal length and a second focal length, l₁And l₂And the two object points respectively represent the vector size of two image points of the two object points in the plane space vertical to the optical axis under the second focal length and the first focal length.

Compared with the prior art, the method has the following beneficial effects:

based on the technical scheme, the passive infrared thermal imaging 3D camera and the shooting method thereof obtain original data corresponding to a plurality of infrared polarization images under different focal lengths through the polarized photothermal imaging sensor, reconstruct the original data through the image processor to obtain a plurality of infrared polarization images containing space information, perform feature point matching on the infrared polarization images, and calculate depth information according to the focal lengths and image point vectors to obtain the 3D images. The application provides a passive form infrared thermal imaging 3D camera and camera shooting method need not to adopt high-power light source, does not have any influence to the human body, adopts polarized light thermal imaging sensor can work in all weather moreover, and the detection distance is up to more than 200 meters, satisfies high-end application field requirements such as autopilot, intelligent robot or unmanned aerial vehicle.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a block diagram of a passive infrared thermal imaging 3D camera according to an embodiment of the present disclosure.

Fig. 2 is an arrangement of a pixel including a polarization device according to an embodiment of the present application.

Fig. 3 is a schematic diagram of bifocal imaging provided in an embodiment of the present application.

Fig. 4 is an image capturing method according to an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or device that comprises a list of elements does not include only those elements but may include other elements not expressly listed. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of additional like elements in the article or device comprising the element.

Fig. 1 is a block diagram of a passive infrared thermal imaging 3D camera according to an embodiment of the present disclosure.

Referring to fig. 1, a passive infrared thermal imaging 3D camera provided by an embodiment of the present application includes an optical system, a polarized photothermal imaging sensor, and an image processor.

The optical system comprises one or more lenses for converging light rays, and the focal length of the camera can be changed by adjusting the physical position of the lenses.

The polarized photothermal imaging sensor comprises a plurality of pixels including a polarizing device, a readout circuit, an ADC, and an interface circuit.

The picture element may receive electromagnetic radiation in a spectral range corresponding to thermal radiation, e.g. having a wavelength of 3.5 to 14 micrometer, reflected from an ambient object.

The pixel is a multilayer structure prepared by an MEMS process, wherein a component mainly used for detecting infrared heat radiation is a thermosensitive film layer. When the thermosensitive film layer is irradiated by infrared rays, the resistance of the thermosensitive film layer changes, so that an electric signal, namely raw data of infrared thermal imaging can be generated. The components of the thermosensitive thin film layer may include, but are not limited to, ferroelectric materials, vanadium oxide VOx, graphene, mercury cadmium telluride HgCdTe, quantum well QWIP, new type two superlattices, or quantum dots QDIP.

In one embodiment, a plurality of picture elements comprising polarizing means are arranged in a diamond-shaped staggered array.

After the natural light is irradiated, the amplitudes of the electric vector vertical component and the parallel component in the reflected light are changed, so that the reflected light is not isotropic natural light any more and is partially polarized light or linearly polarized light, polarization characteristics determined by the properties of the reflected light and the electromagnetic radiation process of an object can be generated, different states (such as roughness, water content, material physical and chemical characteristics and the like) of different objects or the same object often have different polarization states in a thermal infrared band, and by utilizing an infrared polarization imaging technology that a target shows radiation or reflected polarization information, multi-dimensional characteristic information such as the intensity, polarization, images and the like of the target can be comprehensively obtained, the contrast ratio of the target and a background is effectively improved, the detailed characteristics of the target are highlighted, the target identification effect is enhanced, and the attributes and behaviors of the target are more comprehensively and deeply known.

The image element containing the polarization device can receive polarization characteristic information reflected by an external object, the polarization device can be an adjustable polarization element additionally arranged at the front section of the camera, and the image element can also be manufactured with a grating array by adopting an MEMS (micro-electromechanical systems) process, and the polarization device is integrated on the image element.

In one embodiment, referring to fig. 2, each four mutually adjacent pixels in the plurality of pixels containing polarizing means form a pixel group, and the polarizing means corresponding to four independent pixels in the pixel group have different polarization angles. The polarization angles of the four individual pixels within an image element group may be 0 °, 45 °, 90 ° and 135 °, respectively.

The reading circuit is composed of a plurality of columns, each column of the reading circuit is electrically connected with all the pixels of the column, the pixels receive infrared waves radiated by an object, the resistance changes, the row selection switches of two adjacent rows of the pixels are switched on each time, the resistance changed by the pixels of two adjacent rows is read, and the read signals are in an analog signal format.

Each column readout circuit may include: the circuit comprises a bias voltage, an auxiliary resistor, a bias voltage, a transistor, an integrator, a sampling capacitor, a row selection switch and a pixel equivalent resistor.

After the readout circuit outputs the pixel analog signal, it is sent to an analog-to-digital conversion circuit (ADC), which converts the analog signal to a digital signal and outputs it to the image processor via an interface circuit.

The interface circuit is used for transmitting the infrared polarization image digital signals output by the ADC to the image processor, so that subsequent image analysis processing is facilitated.

The interface circuit may include one of an AER interface, a Mobile Industrial Processor Interface (MIPI), and a parallel interface.

The image processor includes an image reconstruction module and a depth calculation module.

The image processor may more specifically include, but is not limited to, a Graphics Processor (GPU), a Central Processing Unit (CPU), an Arithmetic Logic Unit (ALU), a Digital Signal Processor (DSP), a microcomputer, a Field Programmable Gate Array (FPGA), a system on chip (SoC), a programmable logic unit, a microprocessor, an Application Specific Integrated Circuit (ASIC), and the like.

The image reconstruction module is used for converting the original data containing the polarization information into an infrared polarization image, and the infrared polarization image has spatial information.

In one embodiment, the reconstruction process reconstructs four adjacent pixels having different polarization angles into one pixel. Specifically, in the image reconstruction module, the original data acquired by the pixel element is translated for multiple times, so that the reconstructed pixel position contains the original data of multiple polarization angles. The method comprises the steps of reconstructing a Stokes vector of a target according to original data information acquired by the four pixels by adopting a Stokes formula, calculating the polarization degree of a pixel to be reconstructed according to the Stokes vector, calculating an incident azimuth angle and an incident zenith angle of the corresponding target surface of the reconstructed pixel by using the polarization degree, calculating a surface normal of the reconstructed target according to the incident azimuth angle and the incident zenith angle, calculating spatial information of the target to be reconstructed according to the surface normal, calculating the real spatial information of the structure of the target to be reconstructed by using a surface area fraction Frankot-Chellappa algorithm, and obtaining an infrared polarization image with the spatial information. Through the process of reconstructing the images, the first infrared polarization image original data and the second infrared polarization image original data are reconstructed into a first infrared polarization image and a second infrared polarization image respectively, the first infrared polarization image and the second infrared polarization image are transmitted to a depth calculation module after the infrared polarization images are obtained, and subsequent depth information calculation is carried out.

The depth calculation module is used for calculating the depth information of one or more object points in the infrared polarization image.

In one embodiment, the images are first feature matched. Image matching can be performed by using an algorithm based on SIFT and improved algorithm PCA-SIFT and SURF feature matching. Further, a simplified SURF algorithm can be adopted, which is to iterate based on the fixed direction of the information in the circular neighborhood of the interest point, then construct a square region for the corresponding selected direction, and extract SURF descriptors according to the square region. The image under the monocular bifocal vision condition is approximate to an ideal condition without rotation transformation, and the simplified SURF algorithm can improve the arithmetic efficiency and enhance the practicability of operators.

The method comprises the steps of adopting a two-step matching method, firstly selecting a first infrared polarization image with a small focal length as a reference image, taking a second infrared polarization image with a large focal length as an image to be matched to obtain a matching feature point set, then sequentially obtaining the matching feature point set by exchanging the reference image and the image to be matched, and finally taking the intersection of the first two matching feature points as a final matching feature point set.

And after the characteristic points are matched, calculating the distance of the space object points by using the geometric relation.

Fig. 3 is a schematic diagram of bifocal imaging provided in an embodiment of the present application.

Referring to fig. 3, assuming that at least 2 object points M and N exist in the vertical optical axis plane space, imaging at a small focal distance corresponds to the image points M1 and N1, respectively, and imaging at a large focal distance corresponds to the image points M0 and N0. Z is the distance value from the origin to the plane where the object point is located, namely the depth. f1 and f2 represent a smaller focal length and a larger focal length, respectively, and let l1= M0N0 and l2= M1N1 represent the pixel vector size at the corresponding focal length, respectively, and the calculation formula of the depth Z can be derived according to the relationship of the geometric model:

。

and then selecting the intersection of the characteristic points acquired by two image matching, performing monocular stereoscopic vision depth estimation, substituting the intersection of the characteristic points into an equation, calibrating by using a camera to acquire internal parameters to obtain the depth value of the characteristic point pair, wherein the depth value basically accords with an ideal depth value, the depth condition of a space object point can be reflected, and the 3D image is finally acquired by combining the preorder polarized image reconstruction process.

Fig. 4 is an image capturing method according to an embodiment of the present application.

Referring to fig. 4, the method includes: s401, acquiring first infrared polarization image original data at a first focal length; s402, acquiring original data of a second infrared polarization image at a second focal length, wherein the first focal length is smaller than the second focal length; s403, respectively reconstructing the first infrared polarization image original data and the second infrared polarization image original data to obtain a first infrared polarization image and a second infrared polarization image; s404, performing feature matching on the first infrared polarization image and the second infrared polarization image, calculating depth information, and obtaining a 3D image.

The passive infrared thermal imaging 3D camera and the image pickup method can be installed in electronic equipment with an image sensing function and/or a light sensing function. For example, the passive infrared thermography 3D camera may be installed in electronic devices such as smartphones, cameras, internet of things (IoT) devices, tablet Personal Computers (PCs), Personal Digital Assistants (PDAs), Portable Multimedia Players (PMPs), navigation devices, drones, and Advanced Driver Assistance Systems (ADAS). In addition, passive infrared thermography 3D cameras may be provided as elements in vehicles, furniture, manufacturing equipment, various measurement equipment, and the like.

The foregoing is only a preferred embodiment of the present invention, and although the present invention has been disclosed in the preferred embodiments, it is not intended to limit the present invention. Those skilled in the art can make numerous possible variations and modifications to the present teachings, or modify equivalent embodiments to equivalent variations, without departing from the scope of the present teachings, using the methods and techniques disclosed above. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.