阅读说明：本技术 一种基于并行单像素成像的多次反射光分离方法 (Multiple reflected light separation method based on parallel single-pixel imaging ) 是由 姜宏志 闫雍靖 赵慧洁 李旭东 于 2021-07-27 设计创作，主要内容包括：本发明设计一种基于并行单像素成像的多次反射光分离方法,将并行单像素成像应用至多次反射光表面中混叠光线的分解中,能够完成多次反射光线中一次、二次和三次及以上反射光的分离。首先通过投射器投射正弦灰度条纹到被测多次反射光场景,相机对场景中的调制信息进行采集；其次,根据相机采集到的图像结果做相位相减得到傅里叶系数,对傅里叶系数做傅里叶逆变换可以得到投射器视角下的光传输系数图像；随后分别根据极线约束和立体三维模型分离一次反射光和二次反射光线,因此实现了多次反射光表面的分离。这种方法是传统光照渲染的逆过程,可以直接通过相机-投射器系统获得场景信息的复合光照结果并对混叠光线完成分离,对推动光学测量、计算机视觉和计算机图形学的发展有重要作用。(The invention designs a multiple reflected light separation method based on parallel single-pixel imaging, which applies the parallel single-pixel imaging to the decomposition of aliasing light rays in multiple reflected light surfaces and can complete the separation of primary, secondary, tertiary and more than three reflected light rays in multiple reflected light rays. Firstly, projecting sine gray stripes to a scene of multiple reflected lights to be detected through a projector, and acquiring modulation information in the scene through a camera; secondly, carrying out phase subtraction according to image results acquired by a camera to obtain a Fourier coefficient, and carrying out inverse Fourier transform on the Fourier coefficient to obtain an optical transmission coefficient image under the visual angle of the projector; and then separating the primary reflected light and the secondary reflected light according to the polar constraint and the three-dimensional model, so that the separation of the multiple reflected light surfaces is realized. The method is the inverse process of the traditional illumination rendering, can directly obtain a composite illumination result of scene information through a camera-projector system and complete separation of aliasing rays, and plays an important role in promoting the development of optical measurement, computer vision and computer graphics.)

1. A multiple reflected light separation method based on parallel single-pixel imaging is characterized in that: the separation process comprises the following steps:

(1) before a projector and a camera are placed in a detected scene, the projector projects sine stripes required by parallel single-pixel imaging, the camera acquires a mixed image of the stripes and the scene, and the light intensity acquired by image pixels is the mixture of primary, secondary and multiple reflected light components;

(2) performing a parallel single-pixel imaging algorithm on each pixel in the camera to obtain an optical transmission coefficient image under the visual angle of the projector corresponding to the pixel, wherein a plurality of light spots in the optical transmission coefficient image reflect component information of primary, secondary and multiple reflected lights;

(3) separating primary reflected light (direct reflected light) from mixed illumination information by adopting an epipolar constraint principle according to the light transmission coefficient image, adding the primary reflected light intensities of all pixels to obtain a result image of primary reflected light separation, establishing a three-dimensional model through pixel points and primary reflected light points, and reconstructing to obtain a three-dimensional point cloud model of a detected scene;

(4) according to the characteristics of the secondary reflected light separation model and the propagation path, decomposing the secondary reflected light in the indirect reflected light component by adopting a secondary reflected light separation algorithm, and adding the intensities to obtain a secondary reflected light separation result image;

(5) in the light transmission coefficient image, the primary and secondary reflected light components are removed, and the remaining light spots represent the three or more reflected light components, and the intensities thereof are added to obtain a result image separated by three or more times.

2. A method according to claim 1, characterized in that: the expression of the multiple reflected light aliasing ray mentioned in the step (1) is as follows:

wherein, I_out(x, y) is the multiple reflection alias intensity received by a certain camera pixel (x, y), h (x, y; m, n) is the light transmission coefficient of the light from the projector pixel (m, n) to the camera pixel (x, y), I_in(m, n) is the intensity of the outgoing light from the projector pixel (m, n), I_e(x, y) is the ambient light intensity.

3. The method of claim 1, wherein: in the step (3), because the primary reflected light of the camera belongs to a component which is directly emitted from a certain pixel of the projector and is directly received by the camera through the primary reflection, the epipolar constraint condition in the stereoscopic vision is met, in the light transmission coefficient image, the epipolar threshold is set by calculating the epipolar line of the camera pixel corresponding to the visual angle of the projector, so that the light spot in the threshold range can be determined, namely the light spot represents the primary reflected light component, the intensity of the light spot is added, the pixel value of the primary reflected light component can be obtained, and the processing is carried out on each pixel, so that the result image of the primary reflected light component can be obtained.

4. The method of claim 1, wherein: the principle expression of the secondary reflected light separation algorithm involved in the step (4) for separating the secondary reflected light is as follows:

wherein K_LAnd K_GRepresenting the reflectivity of a lambertian surface and a shiny surface, respectively, ranging from 0 to 1, the parameter s describing the roughness of the curved surface.

Due to the uniqueness of the secondary reflection light propagating in space, the secondary reflection occurs on the curved surface of the multiple reflection material, and the secondary reflection occurs as shown in the figure. For a certain point Y on the curved surface, the incident light is emitted by a certain pixel (m, n) of the projector, reflected by the point Y, reaches another point X, and is finally received by the camera pixel (X, Y), i.e., the secondary reflected light. Let L be the light incident on Y and R be the light reflected by Y, where i (m, n; X, Y) and R (m, n; X, Y) are the incident angle and the reflection angle of the light at point Y, respectively, and the above two equations represent the angular relationship between the twice reflected light from a certain pixel (m, n) of the projector to a certain pixel (X, Y) of the camera when the first reflection occurs. For each camera pixel, after removing the primary reflected light component, a light transmission coefficient image containing only the indirect reflected light component can be reconstructed, candidate points in the light transmission coefficient image are traversed, and according to the formula, the image meeting the threshold condition can be determined as secondary reflected light.

Technical Field

The invention relates to a method for separating multiple reflected lights based on parallel single-pixel imaging, which uses the parallel single-pixel imaging in the field of light transmission and photogrammetry and combines the traditional three-dimensional measurement technology to complete the separation of primary, secondary and multiple reflected light components on the surface of multiple reflected lights. The invention mainly belongs to the field of computational imaging and computer vision.

Background

The multiple reflected light is actually the aliasing of the light in the space, which causes the multiple reflected light to be mixed together, and when the light is collected by the optical detector, the light intensity information received by the optical detector is actually the result of the multiple reflected light mixed together. In the field of photogrammetry, the decomposition of multiple reflected light rays can help to reconstruct real three-dimensional scene information, and the measurement precision is improved; in the field of computer graphics, because a virtual scene is often rendered through a rendering equation, the propagation process of light from a light source to the scene is simulated, illumination information such as multiple reflected light is added in the propagation process, so that the scene information is more real, and for the separation of the multiple reflected light, the separation is actually a reverse process of rendering, the separation of the reflected light can help to verify the rendering authenticity, and theoretical basis and real measurement data are provided for the separation.

For example, a parallel single-pixel imaging method disclosed in chinese patent application publication No. cn110264540a studies light intensity information received by a single detector pixel, and analyzes the light intensity information to obtain illumination information corresponding to the pixel, and multiple reflected light is actually a part of the illumination information, so that from a theoretical standpoint, parallel single-pixel imaging should be able to complete identification and separation of various components in the multiple reflected light.

Disclosure of Invention

The invention provides a method for separating multiple reflected lights based on parallel single-pixel imaging, which expands the basic principle of the parallel single-pixel imaging to the problem of separation of aliasing rays, and realizes the identification and separation of multiple reflected light components under the complex illumination condition through the uniqueness of a parallel single-pixel algorithm. The flow chart of the multiple reflected light separation method based on parallel single-pixel imaging is shown in the attached figure 1.

The basic principle of the invention is that parallel single-pixel imaging is transplanted to a traditional array camera, each camera pixel is taken as a single-pixel detector, Fourier inverse transformation processing is carried out on light intensity information acquired by the single-pixel detector to obtain an optical transmission coefficient image, direct reflection points are identified in the optical transmission coefficient image through polar line constraint, and then secondary reflection points are identified and separated through a three-dimensional model to complete separation of primary, secondary and multiple reflection light components.

The method is different from other separated reflected light rays in that the method realizes the reverse reduction from measured data to a multi-reflected light component image, and is characterized in that the method does not acquire the surface material information and the space three-dimensional structure of a detected scene in advance, and the multi-reflected light component is identified and separated aiming at a static unknown scene, which is the most unique and direct advantage of the method.

The technical solution of the invention is as follows: firstly, generating sine gray stripes suitable for the method according to a Fourier transform parallel single-pixel imaging method, projecting the modulated gray stripes to a detected scene by using a projector according to frequency transformation, and shooting and sampling a mixed scene of the scene and the modulated stripes by using a calibrated camera; after shooting is finished, each single pixel on a camera pixel is subjected to phase subtraction and inverse Fourier transform to obtain an optical transmission coefficient image under the projector visual angle corresponding to the pixel; polar line constraint processing is carried out on the light transmission coefficient, and the primary reflection light spots are identified and separated; and reconstructing according to a stereoscopic vision principle to obtain a three-dimensional point cloud, establishing a light propagation model, and identifying and separating secondary reflected light components according to a secondary reflected light criterion. It mainly comprises the following steps:

(1) before the projector and the camera are placed in a detected scene, the projector should generate sine gray stripes needing to be projected in advance, light rays projected by the projector are subjected to aliasing on a propagation path, and an aliasing light ray intensity image is acquired by the camera.

(2) According to the multi-step phase shift principle, phase subtraction is carried out on the four phase images under each frequency to obtain a Fourier coefficient, and inverse Fourier transform processing is carried out on the Fourier coefficient;

(3) all the camera pixels are processed in the step (2), and each pixel point can obtain an optical transmission coefficient image;

(4) decomposing primary reflected light (directly reflected light) and indirect reflected light by adopting an epipolar constraint principle according to the optical transmission coefficient image;

(5) and reconstructing a three-dimensional point cloud data model according to a stereoscopic vision principle, decomposing indirect reflected light components according to a secondary reflected light separation model, and decomposing to obtain secondary reflected light and three or more reflected light.

The expression of the multiple reflected light aliasing ray mentioned in the step (1) is as follows:

wherein, I_out(x, y) is the multiple reflection alias intensity received by a certain camera pixel (x, y), h (x, y; m, n) is the light transmission coefficient of the light from the projector pixel (m, n) to the camera pixel (x, y), I_in(m, n) is the intensity of the outgoing light from the projector pixel (m, n), I_e(x, y) is the ambient light intensity.

The light transmission coefficient image mentioned in the step (3) is located under the visual angle of the projector, the resolution is consistent with that of the projector, a plurality of light spots generally exist on the surface of the multiple-reflection light, and for a certain pixel (m, n), the pixel represents that the light emitted from the point on the projector array reaches the current camera pixel (x, y) after being reflected for a plurality of times in space, and the corresponding relation of the light transmission coefficient image and the current camera pixel (x, y) is just reflected in the light transmission coefficient image.

In the step (4), the primary reflected light of the camera belongs to a component which is directly emitted from a certain pixel of the projector and is directly received by the camera through the primary reflection, so that the epipolar constraint condition in the stereoscopic vision is met, in the light transmission coefficient, only the primary reflected light meets the condition by calculating the epipolar line of the camera pixel corresponding to the visual angle of the projector, and therefore, the separation of the primary reflected light can be completed through the method.

The principle expression of the three-dimensional model designed in the step (5) for separating the secondary reflected light from the third reflected light and above is as follows:

δ(m,n；x,y)≤θ_d

due to the uniqueness of the secondary reflected light in the space, the secondary reflected light occurs on the curved surface of the material of multiple reflected light and can be reflected twice as shown in the figure, and the propagation path of the secondary reflected light is consistent with that of the primary reflected light after the secondary reflection to form aliasing light, so that the secondary reflected light is separated before. For a certain point Y on the curved surface, after the incident light is emitted by a certain pixel (m, n) of the projector, the light is reflected by the point Y on the curved surface to reach another point X on the curved surface, and then is reflected to a point (X, Y) on the camera pixel to form secondary reflected light. Let L be the light incident on Y and R be the light reflected by Y, where i (m, n; X, Y) and R (m, n; X, Y) are the incident angle and the reflection angle of the light at point Y, respectively, and the above two equations represent the angular relationship between the twice reflected light from a certain pixel (m, n) of the projector to a certain pixel (X, Y) of the camera when the first reflection occurs. According to the steps, a certain pixel value acquired by the camera can be subjected to inverse Fourier transform to obtain an optical transmission coefficient image, primary reflected light can be screened out by an epipolar constraint method, the value of a primary reflected light spot is set to be 0 on the optical transmission coefficient image, and indirect reflected light components are reserved; meanwhile, according to the matching relation between the primary reflected light and the camera pixels, the corresponding three-dimensional coordinate of each camera pixel in the space can be obtained through calculation. Therefore, for each camera pixel, a light transmission coefficient image containing only indirect reflected light components can be reconstructed, all points in the light transmission coefficient image are traversed, and according to the above formula, the light which satisfies the threshold condition can be determined as secondary reflected light.

The invention has the advantages that:

(1) the invention aims at a three-dimensional scene with a multi-time reflected light surface, and can complete the separation work of a primary component, a secondary component and a third component or more in the multi-time reflected light on the premise of not knowing the surface material and the three-dimensional shape, which is different from the traditional separation research. The decomposition of the secondary reflection light can help the accurate establishment of BRDFs and BSSDFs models and the verification of rendering effect in the fields of optics and computer graphics.

(2) A visual representation of light from the light source to the collector is provided. By introducing parallel single-pixel imaging, the invention directly reconstructs the light transmission coefficient of the light from the projector to the camera, and the representation result composed of the reflected light in each order can be directly observed in the two-dimensional image of the light transmission coefficient. This means that the propagation of the light can be directly visualized in the optical transmission coefficient, providing a basis for the decomposition of the individual components.

(3) And completing the separation of secondary reflected light of the unknown scene. On the premise of unknown scene morphology and surface material, the difficulty is that multiple reflected lights are decomposed. The method starts from parallel single-pixel imaging, light transmission coefficient images are reconstructed through the parallel single-pixel imaging, and after a three-dimensional model is built, secondary reflection light meeting judgment conditions can be screened out in a reflection light lobe range according to a propagation path of the secondary reflection light and a Phong model.

In a word, the invention provides a method for separating multiple reflected light surfaces based on parallel single-pixel imaging, wherein a light transmission coefficient is reduced through the parallel single-pixel imaging, each component can be separated one by one in the light transmission coefficient according to the respective characteristics of multiple reflected light components of one time, two times and more than three times, the separation of the reflected light components on the multiple reflected light surfaces is realized, and the separation result is displayed by a two-dimensional image.

Drawings

FIG. 1 is a flow chart of the present invention.

Fig. 2 is a schematic diagram of the propagation of light from a projector array to a camera array. In the figure, 1 is a primary reflection light propagation path, 2 is a secondary reflection light path, 3 is a tertiary reflection light path, 4 is a stripe projector, 5 is a certain pixel representation on a projector array, 6 is a measured multiple reflection light scene, 7 is a multiple reflection light aliasing ray received by a camera, 8 is a certain pixel representation on a camera array, and 9 is the camera.

Fig. 3 is a diagram illustrating the light transmission coefficient obtained by the pixel. In the figure, 8 is a pixel on the camera array, 9 is a camera, 10 is an image with optical transmission coefficient, and 11 is an epipolar line corresponding to the pixel on the image.

Fig. 4 is a schematic diagram of secondary reflected light separation. In the figure, 1 is a primary reflection light propagation path, 2 is a secondary reflection light path, 4 is a stripe projector, 7 is a multiple reflection light aliasing ray received by a camera, 8 is a certain pixel representation on a camera array, 9 is the camera, 12 is a computer, 13 is a curved surface representation of a measured multiple reflection light scene, 14 is a gray scale sine stripe emitted by the projector, 15 is a normal line of a reflection surface 13 at a certain point, and 16 and 17 represent an incident angle and a reflection angle of the secondary reflection light at the point respectively.

Detailed Description

The technical solution of the present invention is further explained with reference to the accompanying drawings and the detailed description.

The invention provides a multiple reflected light separation method based on parallel single-pixel imaging, which is implemented by the method according to a flow chart shown in figure 1, gray scale sine stripes are projected by a stripe projector, the stripes are reflected by multiple reflected light scenes and received by camera pixels, a two-dimensional image of a light transmission coefficient is obtained by Fourier inverse transformation, primary reflected light is screened out by polar line constraint on the basis of the two-dimensional image, a stereoscopic vision relation is established between the primary reflected light and pixel points of the projector to obtain three-dimensional reconstruction data, a separation model of secondary reflected light is established on the basis, the secondary reflected light is separated from indirect light components by a secondary reflected light criterion, and finally, the separation of the primary, secondary and tertiary reflected light components and more than primary reflected light components is completed. A schematic diagram of the formation of aliased rays in space by multiple reflections is shown in fig. 2. The method specifically operates as follows:

1. and calibrating the stripe projector and the camera to obtain a spatial pose relationship between the stripe projector and the camera, generating gray sine stripes with the parallel single-pixel characteristic, and ensuring that a modulation pattern projected by the projector can be collected by the camera. The projector projects a modulation pattern, and the camera acquires images of scenes by triggering to obtain a series of scene images combined by the scenes and the stripes.

2. For a two-dimensional image acquired by a camera, a fourier coefficient can be obtained after phase subtraction processing, and an optical transmission coefficient as shown in fig. 3 can be obtained by performing inverse fourier transform on the fourier coefficient. The optical transmission coefficient is an aliasing result of multiple reflected light components, and the expression is as follows:

wherein, I_out(x, y) is the light intensity of the mixed light received by the camera pixel, h (x, y; m, n) is the light transmission coefficient of the projector coordinate (m, n) corresponding to the camera array coordinate (x, y), and is shown as a certain bright point in the light transmission coefficient in fig. 3, and the separation principle of the multiple reflected lights is shown as follows:

hⁿ(x,y；m,n)＝h(x,y；m,n)·Mⁿ(x,y；m,n) (2)

wherein M isⁿ(x, y; m, n) is a mask for reflecting light n times in a certain sequence, and the expression is as follows:

cn in formula (3) represents whether the current reflected light belongs to the nth reflected light which is logically required, if yes, the current reflected light is set to be 1, otherwise, the current reflected light is set to be 0, and formulas (2) and (3) calculate the light transmission coefficient of multiple reflected light in mathematical logic. Fig. 3 illustrates the process of reducing the light transmission coefficient from a certain pixel of the camera.

3. The primary reflected light is separated from the light transmission coefficient image. In fig. 3, 9 is a camera, and 8 is a certain pixel on the camera array, which is subjected to inverse fourier transform to obtain an image under the viewing angle of the projector, the size of the image is the resolution of the projector, some bright spots in the image represent the light transmission coefficients of reflected light in different orders, and the characteristics of reflected light compared with indirect light should be searched for in order to separate reflected light once. Because the primary reflected light represents the primary reflection of the light through the scene to be measured only during the propagation processThat is, a camera pixel is reached, which should satisfy the basic constraint relation of stereo vision in space, so in this step, a primary reflection light spot is selected and screened through epipolar constraint, as shown in the figure, 10 is an optical transmission coefficient image, 11 is an epipolar line of the pixel on the optical transmission coefficient image, a light spot located in the epipolar threshold range is judged according to the epipolar line, and the sum obtained by summing the pixel values is the primary reflection light intensity I received by the pixel¹(x, y), and processing all the pixels according to the method to obtain a two-dimensional separation result of the primary reflected light.

4. And obtaining the three-dimensional structure of the measured surface through three-dimensional visual stereo reconstruction by the aid of the primary reflection light spots and the corresponding camera pixel points. With this three-dimensional result, a separation model for separating the secondary reflected light can be established, as shown in fig. 4.

Due to the uniqueness of the secondary reflected light in the space, the secondary reflected light occurs on the curved surface of the material of multiple reflected light and can be reflected twice as shown in the figure, and the propagation path of the secondary reflected light is consistent with that of the primary reflected light after the secondary reflection to form aliasing light, so that the secondary reflected light is separated before. For a certain point Y on the curved surface, after the incident light is emitted by a certain pixel (m, n) of the projector, the light is reflected by the point Y on the curved surface to reach another point X on the curved surface, and then is reflected to a point (X, Y) on the camera pixel to form secondary reflected light. Here, let L be a ray incident on Y and R be a ray reflected from Y to X, and there is the following formula according to the reflection characteristics of light:

where i (m, n; x, Y) and r (m, n; x, Y) are the incident angle and the reflection angle of the light at point Y, respectively, and the above two equations represent the angular relationship between the twice reflected light from a certain pixel (m, n) of the projector to a certain pixel (x, Y) of the camera when the first reflection occurs. According to the steps, a certain pixel value acquired by the camera can be subjected to inverse Fourier transform to obtain an optical transmission coefficient image, primary reflected light can be screened out by an epipolar constraint method, the value of a primary reflected light spot is set to be 0 on the optical transmission coefficient image, and indirect reflected light components are reserved; meanwhile, according to the matching relation between the primary reflected light and the camera pixels, the corresponding three-dimensional coordinate of each camera pixel in the space can be obtained through calculation. Thus, for each camera pixel, an image of the light transmission coefficient can be reconstructed that contains only indirect reflected light components, including indirect light from some pixels on the projector, with all the spot coordinates being substituted into the following equation:

δ(m,n；x,y)≤θ_d (7)

wherein, K_LAnd K_GRespectively, represent the reflectance of a lambertian surface and a smooth surface, ranging from 0 to 1, which respectively represent the reflectance characteristics of a non-smooth surface and a smooth surface. The parameter s represents the roughness of the current surface, the numerical value of the metal material is larger than that of other materials, once the final calculation result delta (m, n; x, y) is smaller than a given threshold value, the light from (m, n) to (x, y) is judged to belong to secondary reflection light, each camera pixel can be screened in sequence, the corresponding secondary reflection light is selected from the reverse tracking angle of the light, after screening, the light spot pixel values meeting the conditions in the light transmission coefficient images are reserved, the rest is set to be 0, and then the numerical values of each light transmission coefficient image are added to obtain the final secondary reflection light separation result.