阅读说明：本技术 被动式单光子成像3d摄像机及摄像方法 (Passive single photon imaging 3D camera and shooting method ) 是由 赵照 于 2021-08-16 设计创作，主要内容包括：本申请公开了一种被动式单光子成像3D摄像机及摄像方法,通过偏振光单光子传感器在不同焦距下获得多个单光子偏振图像对应的原始数据,图像处理器对原始数据进行图像重建,获得多个含有空间信息的单光子偏振图像,对单光子偏振图像进行特征点匹配,根据焦距和像点向量计算深度信息,获得3D图像。本申请提供的被动式单光子成像3D摄像机及摄像方法无需采用大功率光源,对于人体无任何影响,而且采用偏振光单光子传感器可以全天候工作,检测距离长达200米以上,满足自动驾驶、智能机器人或无人机等高端应用领域要求。(The application discloses passive single photon imaging 3D camera and a shooting method, original data corresponding to a plurality of single photon polarization images are obtained through a polarized light single photon sensor under different focal lengths, an image processor carries out image reconstruction on the original data to obtain a plurality of single photon polarization images containing space information, feature point matching is carried out on the single photon polarization images, depth information is calculated according to focal lengths and image point vectors, and 3D images are obtained. The passive single-photon imaging 3D camera and the shooting method do not need to adopt a high-power light source, have no influence on a human body, adopt the polarized light single-photon sensor to work in all weather, have the detection distance of more than 200 meters, and meet the requirements of high-end application fields such as automatic driving, intelligent robots or unmanned aerial vehicles.)

1. A passive single photon imaging (3D) camera, comprising:

an optical system configured to converge incident light and adjust a focal length;

a polarized single photon sensor comprising a polarizing arrangement configured to generate light; a photo-sensing cathode configured to capture photons and generate photoelectrons; a collection anode configured to capture photoelectrons to produce an electrical signal; a readout circuit configured to read out the electrical signal; an ADC configured to convert the electrical signal into single photon polarization image raw data; an interface circuit configured to transmit the single photon polarization image raw data to an image processor;

and the image processor is configured to reconstruct the original data of the single-photon polarization image to obtain a single-photon polarization image, perform feature matching and depth calculation on the single-photon polarization image, and acquire a 3D image.

2. The camera of claim 1, wherein the plurality of pixels comprising polarizing means are arranged in a diamond-shaped staggered array.

3. The camera of claim 1, wherein the polarizing means of each four mutually adjacent picture elements of the plurality of picture elements comprising polarizing means have a different respective polarization angle.

4. A camera according to claim 3, wherein the polarizing means has a polarization angle of 0 °, 45 °, 90 ° or 135 °.

5. The camera of claim 1, wherein the polarizing means is integrated on the pixel by a MEMS process.

6. The camera of claim 1, wherein the photosensitive cathode material comprises a Si, Ge, GaAs, InGaAsP, InGaAs, or InAs/GaAsSb class II superlattice.

7. The camera of claim 1, wherein the collection anode comprises a photomultiplier tube, an avalanche photodiode, a single photon avalanche diode, a silicon photomultiplier tube, an enhanced photodiode, an electron multiplying CCD, or a scientific grade sCMOS.

8. An image pickup method, comprising:

acquiring original data of a first single photon polarization image at a first focal length;

acquiring original data of a second single photon polarization image at a second focal length; wherein the first focal length is smaller than the second focal length;

respectively reconstructing the original data of the first single-photon polarization image and the original data of the second single-photon polarization image to obtain a first single-photon polarization image and a second single-photon polarization image;

and performing feature matching on the first single-photon polarization image and the second single-photon polarization image, calculating depth information, and acquiring a 3D image.

9. The method according to claim 8, wherein the computing depth information is specifically:；

wherein Z is the distance value from the camera to the plane of the object point, f₁And f₂Respectively representing a first focal length and a second focal length, l₁And l₂And the two object points respectively represent the vector size of two image points of the two object points in the plane space vertical to the optical axis under the second focal length and the first focal length.

Technical Field

The application relates to the field of single photon imaging, in particular to a passive single photon imaging 3D camera.

Background

3D imaging technology is one of the more popular research directions in recent years, and 3D cameras are also used in many fields, such as gesture recognition, robots, and the like.

The double-lens camera can realize a three-dimensional effect, but the structure system is complex, the algorithm is very complex, and the double-lens camera has no large-scale popularization and application for many years, and is only applied to specific occasions such as 3D movies and the like;

the active structure optical camera can also realize the stereoscopic effect, is mainly applied to occasions with very short distance, and has specific requirements on algorithm and hardware structure, so that the trend of being replaced gradually along with the appearance of the active TOF camera is gradually realized.

An active TOF (Time of Flight) camera is a main 3D imaging camera at present, and mainly comprises two products of wired scanning and surface array, wherein the active TOF camera adopts a Time of Flight method for non-contact distance measurement, transmits emitted light waves to a target continuously, receives light returned from an object by using a sensor, and obtains the distance of the target object by detecting the Time of Flight of the light waves, so that the target can be identified and tracked quickly, the depth information can be used for segmenting, marking, identifying, tracking and the like of a target image, and the application of three-dimensional modeling and the like can be finished by further deepening treatment without auxiliary work of scanning equipment.

However, due to the problems of power of the actively-emitted laser light source, eye protection and the like, particularly the high-power light source has high cost, and the application of the high-power light source is mainly focused on the application of mobile phones, tablet computers, internet of things or other intelligent hardware, and the high-end application fields of automatic driving, intelligent robots, unmanned aerial vehicles and the like have certain difficulty.

Disclosure of Invention

In view of this, the present application provides a passive single photon imaging 3D camera and an imaging method.

In order to solve the technical problem, the following technical scheme is adopted in the application:

in a first aspect, a passive single photon imaging 3D camera is provided, where the camera includes:

an optical system configured to converge incident light and adjust a focal length;

a polarized single photon sensor comprising a polarizing arrangement configured to generate light; a photo-sensing cathode configured to capture photons and generate photoelectrons; a collection anode configured to capture photoelectrons to produce an electrical signal; a readout circuit configured to read out the electrical signal; an ADC configured to convert the electrical signal into single photon polarization image raw data; an interface circuit configured to transmit the single photon polarization image raw data to an image processor;

and the image processor is configured to reconstruct the original data of the single-photon polarization image to obtain a single-photon polarization image, perform feature matching and depth calculation on the single-photon polarization image, and acquire a 3D image.

Preferably, the plurality of pixels containing the polarization devices are arranged in a diamond-shaped staggered array mode.

Preferably, the polarizing means of every four mutually adjacent picture elements of said plurality of picture elements comprising polarizing means have different respective polarization angles.

Preferably, the polarizing means has a polarization angle of 0 °, 45 °, 90 ° or 135 °.

Preferably, the polarizing means are integrated on the picture element by a MEMS process.

Preferably, the photosensitive cathode comprises a Si, Ge, GaAs, InGaAsP, InGaAs or InAs/GaAsSb class II superlattice.

Preferably, the collection anode comprises a photomultiplier tube, an avalanche photodiode, a single photon avalanche diode, a silicon photomultiplier tube, an enhanced photodiode, an electron multiplying CCD, or a scientific grade sCMOS.

In a second aspect, there is provided an imaging method, comprising:

acquiring original data of a first single photon polarization image at a first focal length;

acquiring original data of a second single photon polarization image at a second focal length; wherein the first focal length is smaller than the second focal length;

respectively reconstructing the original data of the first single-photon polarization image and the original data of the second single-photon polarization image to obtain a first single-photon polarization image and a second single-photon polarization image;

and performing feature matching on the first single-photon polarization image and the second single-photon polarization image, calculating depth information, and acquiring a 3D image.

Preferably, the calculating depth information specifically includes:

；

wherein Z is the distance value from the camera to the plane of the object point, f₁And f₂Respectively representing a first focal length and a second focal length, l₁And l₂And the two object points respectively represent the vector size of two image points of the two object points in the plane space vertical to the optical axis under the second focal length and the first focal length.

Compared with the prior art, the method has the following beneficial effects:

based on the technical scheme, the passive single photon imaging 3D camera and the image pickup method have the advantages that the polarized light single photon sensor obtains the original data corresponding to the single photon polarization images under different focal lengths, the image processor reconstructs the original data to obtain the single photon polarization images containing the space information, the single photon polarization images are subjected to feature point matching, the depth information is calculated according to the focal lengths and the image point vectors, and the 3D image is obtained. The passive single-photon imaging 3D camera and the shooting method do not need to adopt a high-power light source, have no influence on a human body, adopt the polarized light single-photon sensor to work in all weather, have the detection distance of more than 200 meters, and meet the requirements of high-end application fields such as automatic driving, intelligent robots or unmanned aerial vehicles.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a block diagram of a passive single-photon imaging 3D camera according to an embodiment of the present disclosure.

Figure 2 is a schematic structural diagram of a single photon sensor provided in the embodiments of the present application.

Fig. 3 is an arrangement of a pixel including a polarization device according to an embodiment of the present application.

Fig. 4 is a schematic diagram of bifocal imaging provided in an embodiment of the present application.

Fig. 5 is an image capturing method according to an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described and will be readily apparent to those of ordinary skill in the art without departing from the spirit of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or device that comprises a list of elements does not include only those elements but may include other elements not expressly listed. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of additional like elements in the article or device comprising the element.

Fig. 1 is a block diagram of a passive single-photon imaging 3D camera according to an embodiment of the present application.

Referring to fig. 1, a passive single photon imaging 3D camera provided by the embodiment of the present application includes an optical system, a polarized single photon sensor and an image processor.

The optical system comprises one or more lenses for converging light rays, and the focal length of the camera can be changed by adjusting the physical position of the lenses.

The polarized single photon sensor comprises a polarizing device, a photosensitive cathode, a collecting anode, a reading circuit, an ADC and an interface circuit.

The polarized single photon sensor can receive photoelectron signals excited by single or a small amount of photons reflected by an object to be amplified, and extremely weak photoelectron signals are identified and extracted through technologies such as pulse discrimination, accumulation and the like, so that the ultra-sensitivity limit of photoelectric detection is reached.

Figure 2 is a schematic structural diagram of a single photon sensor provided in the embodiments of the present application.

Referring to fig. 2, specifically, an accelerating voltage 23 is applied between the photosensitive cathode 21 and the collecting anode 22 to accelerate electrons, and after a single photon or multiple photons 24 enter the photosensitive cathode 21 through a transparent panel 25 of the optical system, a photoelectric effect is triggered to generate photoelectrons 26, which are accelerated to fly to the collecting anode 22 under the action of the accelerating voltage 23, and a large number of electron-hole pairs are generated inside under the bombardment of high-energy electrons to realize high-power amplification of photoelectron signals to generate an output signal 27.

Photoelectrons released by the photosensitive cathode move to the collecting anode under the action of an external high voltage. The photosensitive cathode is prepared from one or more materials of Si, Ge, GaAs, InGaAsP, InGaAs or InAs/GaAsSb II type superlattice by MEMS process. By replacing different materials, the photosensitive cathode can realize the reception of electromagnetic waves with wave bands of 300nm-14um, so the spectral response range of the invention is very wide. The vertical distance between the photosensitive cathode and the collecting anode is larger than 1mm, and the setting distance between the photosensitive cathode and the collecting anode is different according to the difference of bias voltage.

The collection anode may include, but is not limited to, a photomultiplier tube (PMT), an Avalanche Photodiode (APD), a Single Photon Avalanche Diode (SPAD), a silicon photomultiplier tube (SIPM), an enhanced photodiode (IPD), an electron multiplying ccd (emccd), and scientific grade sCMOS, etc., and may be fabricated using standard CMOS processes.

After the natural light is irradiated, the amplitudes of the electric vector vertical component and the electric vector parallel component in the reflected light are changed, so that the reflected light is not isotropic natural light any more and is partially polarized light or linearly polarized light, polarization characteristics determined by the properties of the reflected light and the electromagnetic radiation process of an object can be generated, different states (such as roughness, water content, material physical and chemical characteristics and the like) of different objects or the same object often have different polarization states, multi-dimensional characteristic information such as the intensity, polarization and images of the object can be comprehensively obtained by utilizing a single-photon polarization imaging technology of indicating radiation or reflected polarization information by the object, the contrast between the object and the background is effectively improved, the detailed characteristics of the object are highlighted, the object identification effect is enhanced, and the attributes and behaviors of the object are more comprehensively and deeply known.

The polarization device is used for generating polarized light, and can be a metal wire grating array which is integrated on the polarized single photon sensor by adopting an MEMS (micro electro mechanical System) process and contains a specific polarization angle, so that the single photon sensor containing the polarization device can receive polarization characteristic information reflected by an external object.

In one embodiment, referring to fig. 3, every four mutually adjacent pixels are configured into a pixel group, and the polarizing devices corresponding to the four independent pixels in the pixel group have different polarizing angles. Typically, the polarization angles of four individual picture elements within a pixel group may be 0 °, 45 °, 90 ° and 135 °, respectively.

In one embodiment, the pixels are arranged in a diamond-shaped staggered array.

The reading circuit is composed of a plurality of columns, each column of reading circuit is electrically connected with all the collecting anodes in the column, after the collecting anodes receive light rays reflected by an object outwards, a large number of electron-hole pairs are generated inside the collecting anodes under the bombardment of high-energy electrons, the resistance is changed, the row selection switches of two adjacent rows of pixels are switched on each time, the resistance changed by the pixels in the two adjacent rows is read, and the read signals are in an analog signal format.

Each column readout circuit may include: the device comprises a bias voltage, an auxiliary resistor, a bias voltage, a transistor, an integrator, a sampling capacitor, a row selection switch and a collecting anode equivalent resistor.

After the readout circuit outputs the pixel analog signal, it is sent to an analog-to-digital conversion circuit (ADC), which converts the analog signal to a digital signal and outputs it to the image processor via an interface circuit.

The interface circuit is used for transmitting the single-photon polarization image digital signals output by the ADC to the image processor, so that subsequent image analysis processing is facilitated.

The interface circuit may include one of an AER interface, a Mobile Industrial Processor Interface (MIPI), and a parallel interface.

The image processor includes an image reconstruction module and a depth calculation module.

The image processor may more specifically include, but is not limited to, a Graphics Processor (GPU), a Central Processing Unit (CPU), an Arithmetic Logic Unit (ALU), a Digital Signal Processor (DSP), a microcomputer, a Field Programmable Gate Array (FPGA), a system on chip (SoC), a programmable logic unit, a microprocessor, an Application Specific Integrated Circuit (ASIC), and the like.

The image reconstruction module is used for converting original data containing polarization information into a single photon polarization image, and the single photon polarization image has spatial information.

In one embodiment, the reconstruction process reconstructs four adjacent pixels having different polarization angles into one pixel. Specifically, in the image reconstruction module, the original data acquired by the pixel element is translated for multiple times in the image reconstruction module, so that the reconstructed pixel position contains the original data of multiple polarization angles. The Stokes formula is adopted, a Stokes vector of a target is reconstructed according to original data information acquired by the four pixels, the polarization degree of a pixel to be reconstructed is calculated according to the Stokes vector, then, the incident azimuth angle and the incident zenith angle of the target surface corresponding to the pixel to be reconstructed are calculated by utilizing the polarization degree, the surface normal of the reconstructed target can be calculated according to the incident azimuth angle and the incident zenith angle, then, the space information of the target to be reconstructed is calculated according to the surface normal, the real space information of the structure of the target to be reconstructed is calculated through a surface area fraction Frankot-Chellappa algorithm, and a single photon polarization image with the space information is obtained. Through the image reconstruction process, the first single-photon polarization image original data and the second single-photon polarization image original data are reconstructed into a first single-photon polarization image and a second single-photon polarization image respectively, and after the single-photon polarization images are obtained, the first single-photon polarization image and the second single-photon polarization image are transmitted to a depth calculation module for continuous depth calculation.

The depth calculation module is used for calculating the depth information of one or more object points in the single photon polarization image.

In one embodiment, the images are first feature matched. Image matching can be performed by using an algorithm based on SIFT and improved algorithm PCA-SIFT and SURF feature matching. Further, a simplified SURF algorithm can be adopted, which is to iterate based on the fixed direction of the information in the circular neighborhood of the interest point, then construct a square region for the corresponding selected direction, and extract SURF descriptors according to the square region. The image under the monocular bifocal vision condition is approximate to an ideal condition without rotation transformation, and the simplified SURF algorithm can improve the arithmetic efficiency and enhance the practicability of operators.

The method comprises the steps of adopting a two-step matching method, firstly selecting a first single-photon polarization image with a small focal length as a reference image, taking a second single-photon polarization image with a large focal length as an image to be matched to obtain a matching feature point set, then sequentially obtaining the matching feature point set by exchanging the reference image and the image to be matched, and finally taking the intersection of the first two matching feature points as the final matching feature point set.

And after the characteristic points are matched, calculating the distance of the space object points by using the geometric relation.

Fig. 4 is a schematic diagram of bifocal imaging provided in an embodiment of the present application.

Referring to fig. 4, assuming that at least 2 object points M and N exist in the vertical optical axis plane space, imaging at a small focal distance corresponds to the image points M1 and N1, respectively, and imaging at a large focal distance corresponds to the image points M0 and N0. Z is the distance value from the origin to the plane where the object point is located, namely the depth. f1 and f2 represent a smaller focal length and a larger focal length, respectively, and let l1= M0N0 and l2= M1N1 represent the pixel vector size at the corresponding focal length, respectively, and the calculation formula of the depth Z can be derived according to the relationship of the geometric model:

。

and then selecting the intersection of the feature points acquired by two image matching, performing monocular stereoscopic vision depth estimation, substituting the intersection pixel vector of the feature points into an equation, calibrating by using a camera to acquire internal parameters to obtain the depth value of the feature point pair, wherein the depth value basically accords with an ideal depth value, the depth condition of the space object point can be reflected, and the 3D image is finally acquired by combining the polarized image reconstruction process.

Fig. 5 is an image capturing method according to an embodiment of the present application.

Referring to fig. 5, the method includes: s501, acquiring original data of a first single-photon polarization image at a first focal length; s502, acquiring original data of a second single-photon polarization image at a second focal length, wherein the first focal length is smaller than the second focal length; s503, respectively reconstructing the original data of the first single-photon polarization image and the original data of the second single-photon polarization image to obtain a first single-photon polarization image and a second single-photon polarization image; s504, performing feature matching on the first single-photon polarization image and the second single-photon polarization image, calculating depth information, and obtaining a 3D image.

The passive single-photon imaging 3D camera and the shooting method can be installed in electronic equipment with an image sensing function and/or a light sensing function. For example, the passive single photon imaging 3D video camera may be installed in electronic devices such as smart phones, cameras, internet of things (IoT) devices, tablet Personal Computers (PCs), Personal Digital Assistants (PDAs), Portable Multimedia Players (PMPs), navigation devices, drones, and Advanced Driver Assistance Systems (ADAS). In addition, the passive single photon imaging 3D camera can be provided as an element in vehicles, furniture, manufacturing equipment, various measuring equipment, and the like.

The foregoing is only a preferred embodiment of the present invention, and although the present invention has been disclosed in the preferred embodiments, it is not intended to limit the present invention. Those skilled in the art can make numerous possible variations and modifications to the present teachings, or modify equivalent embodiments to equivalent variations, without departing from the scope of the present teachings, using the methods and techniques disclosed above. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.