阅读说明：本技术 轴向扫描光场数据的重建与拼接方法及装置 (Reconstruction and splicing method and device for axial scanning light field data ) 是由 戴琼海 张亿 季向阳 吴嘉敏 于 2020-02-18 设计创作，主要内容包括：本发明公开了一种轴向扫描光场数据的重建与拼接方法及装置,其中,该方法包括：利用轴向扫描光场系统,按照设定的轴向间隔进行采集得到光场图栈,对光场图栈进行重排得到数据图栈；通过计算机模拟轴向扫描光场系统的光路前传过程,得到子孔径点扩散函数；根据数据图栈和子孔径点扩散函数以及重建算重建目标场景的三维信息。该方法利用轴向扫描光场数据联合重建,得到大轴向范围、高分辨率、轴向连续的场景三维重建结果。(The invention discloses a method and a device for reconstructing and splicing axial scanning light field data, wherein the method comprises the following steps: acquiring by using an axial scanning optical field system according to a set axial interval to obtain an optical field image stack, and rearranging the optical field image stack to obtain a data image stack; simulating a light path forwarding process of an axial scanning light field system by a computer to obtain a sub-aperture point diffusion function; and reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation. The method utilizes the joint reconstruction of axial scanning light field data to obtain a scene three-dimensional reconstruction result with large axial range, high resolution and axial continuity.)

1. A reconstruction and splicing method of axial scanning light field data is characterized by comprising the following steps:

s1, acquiring to obtain a light field map stack according to a set axial interval by using an axial scanning light field system, and rearranging the light field map stack to obtain a data map stack;

s2, simulating a light path forwarding process of the axial scanning light field system through a computer to obtain a sub-aperture point diffusion function;

and S3, reconstructing the three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation.

2. The method for reconstructing and stitching axial scan light field data according to claim 1, wherein the S1 further comprises:

and when the data volume of the data map stack is greater than a preset number value and the density of a single optical field map stack is greater than a preset density value, performing axial down-sampling on the data map stack.

3. The method for reconstructing and stitching axial scan light field data according to claim 1, wherein the S2 further comprises:

in the axial scanning light field system, starting from a point light source, simulating a light path forward process by a computer, calculating a plurality of light fields on the front surface of a micro lens array, obtaining the light intensity distribution of the point light source on the surface of a sensor through the phase modulation of the micro lens array and a propagation process, and obtaining the sub-aperture point diffusion function corresponding to each depth according to the light intensity distribution.

4. The method for reconstructing and stitching axial scan light field data according to claim 2, wherein the S2 further comprises:

and carrying out axial non-equal-interval downsampling on the sub-aperture point spread function, wherein the downsampling proportion is the same as that of the data map stack.

5. The method for reconstructing and stitching axial scan light field data according to claim 4, wherein the S3 further comprises:

s31, specifying the reconstruction data of the data diagram stack, and numbering the diagrams in the data diagram stack;

s32, determining the Volume range to be reconstructed, carrying out all-0 initialization, and selecting the graph to be reconstructed;

s33, extracting a corresponding layer in the Volume, using a preset proportion plus all 1 as a reconstruction initial value, and reconstructing a single light field by using an R L deconvolution method to obtain a corresponding Xguss;

s34, interpolating Xgauge to obtain upXgauge;

s35, using a sigmoid function as a fusion proportion, and updating a corresponding layer of the Volume by upXgusess;

s36, selecting the next image, and repeating the steps S32-S35 until all the data in the data graph stack are calculated.

6. The utility model provides a rebuild of axial scanning light field data and splicing apparatus which characterized in that includes:

the acquisition module is used for acquiring the optical field image stack according to a set axial interval by using an axial scanning optical field system, and rearranging the optical field image stack to obtain a data image stack;

the calculation module is used for simulating a light path forwarding process of the axial scanning light field system through a computer to obtain a sub-aperture point diffusion function;

and the reconstruction module is used for reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction algorithm.

7. The apparatus for reconstructing and stitching axial scan light field data according to claim 6, wherein the acquiring module is further configured to perform axial down-sampling on the data map stack when the data amount of the data map stack is greater than a predetermined amount value and the density of a single light field map stack is greater than a predetermined density value.

8. The apparatus for reconstructing and splicing data of an axial scanning light field according to claim 6, wherein the computing module is specifically configured to, in the axial scanning light field system, simulate a light path forwarding process by a computer starting from a point light source, compute a plurality of light fields on a front surface of the microlens array, obtain a light intensity distribution of the point light source on a sensor surface through a phase modulation of the microlens array and a propagation process, and obtain a sub-aperture point spread function corresponding to each depth according to the light intensity distribution.

9. The apparatus for reconstructing and stitching axial scan light-field data according to claim 7, wherein the computing module is further configured to down-sample the sub-aperture point spread function at axially non-equal intervals, and the down-sampling ratio is the same as that of the data map stack.

10. The apparatus for reconstructing and stitching axial scan light field data as recited in claim 9, wherein the reconstruction module is specifically configured to reconstruct and stitch the axial scan light field data

Defining reconstruction data of the data graph stack, and numbering graphs in the data graph stack;

determining the range of the Volume to be reconstructed, carrying out all-0 initialization, and selecting a graph to be reconstructed;

extracting a corresponding layer in the Volume, using a preset proportion plus all 1 as a reconstruction initial value, and reconstructing a single light field by using an R L deconvolution method to obtain a corresponding Xgauge;

interpolating the Xgauge to obtain upXgauge;

using a sigmoid function as a fusion proportion, and updating a corresponding layer of the Volume by upXgusess;

and selecting the next image, and repeating the above processes for calculation until all the data in the data graph stack are calculated.

Technical Field

The invention relates to the technical field of light field data reconstruction, in particular to a method and a device for reconstructing and splicing axial scanning light field data.

Background

The light field technology is a fast volume imaging technology, and can simultaneously record four-dimensional information of a light field. After the four-dimensional information is processed, a three-dimensional target scene can be reconstructed. Because the light field information can be recorded in a single frame, compared with the imaging technology which needs axial dense scanning such as confocal technology, the axial sampling of the light field is sparser, and the imaging speed is faster.

The lateral and axial resolutions of a three-dimensional scene reconstructed by a single light field are reduced along with the increase of the defocusing distance. The axial reconstruction range can be enlarged by adopting axial scanning light field shooting, and the reconstruction resolution can be ensured. However, the axial scanning light field data reconstruction has the following problems: 1. the adoption of the method of separately reconstructing and splicing leads to discontinuity of axial three-dimensional information. 2. When the point spread function is separately reconstructed, the axial range of the point spread function is small, and the background is not modeled, so that the energy calculation of the defocusing plane is inaccurate, and the splicing is obviously discontinuous. 3. When the point spread function is separately reconstructed, the axial range of the point spread function is large, the calculated amount is increased, and the reconstruction speed is obviously reduced.

Disclosure of Invention

The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, one purpose of the present invention is to provide a method for reconstructing and splicing axial scanning light field data, which utilizes the joint reconstruction of the axial scanning light field data to obtain a large axial range, high resolution and axially continuous three-dimensional reconstruction result of a scene.

The invention also aims to provide a device for reconstructing and splicing the axial scanning light field data.

In order to achieve the above object, an embodiment of the present invention provides a method for reconstructing and splicing axial scanning light field data, including:

s1, acquiring to obtain a light field map stack according to a set axial interval by using an axial scanning light field system, and rearranging the light field map stack to obtain a data map stack;

s2, simulating a light path forwarding process of the axial scanning light field system through a computer to obtain a sub-aperture point diffusion function;

and S3, reconstructing the three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation.

The method for reconstructing and splicing the axial scanning light field data comprises the steps of acquiring a light field image stack according to a set axial interval by using an axial scanning light field system, and rearranging the light field image stack to obtain a data image stack; simulating a light path forwarding process of an axial scanning light field system by a computer to obtain a sub-aperture point diffusion function; and reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation. Therefore, three-dimensional reconstruction results with large axial range, high resolution and axial continuity can be obtained.

In addition, the method for reconstructing and stitching axial scanning light field data according to the above embodiment of the present invention may further have the following additional technical features:

further, in an embodiment of the present invention, the S1 further includes:

and when the data volume of the data map stack is greater than a preset number value and the density of a single optical field map stack is greater than a preset density value, performing axial down-sampling on the data map stack.

Further, in an embodiment of the present invention, the S2 further includes:

in the axial scanning light field system, starting from a point light source, simulating a light path forward process by a computer, calculating a plurality of light fields on the front surface of a micro lens array, obtaining the light intensity distribution of the point light source on the surface of a sensor through the phase modulation of the micro lens array and a propagation process, and obtaining the sub-aperture point diffusion function corresponding to each depth according to the light intensity distribution.

Further, in an embodiment of the present invention, the S2 further includes:

and carrying out axial non-equal-interval downsampling on the sub-aperture point spread function, wherein the downsampling proportion is the same as that of the data map stack.

Further, in an embodiment of the present invention, the S3 further includes:

s31, specifying the reconstruction data of the data diagram stack, and numbering the diagrams in the data diagram stack;

s32, determining the Volume range to be reconstructed, carrying out all-0 initialization, and selecting the graph to be reconstructed;

s33, extracting a corresponding layer in the Volume, using a preset proportion plus all 1 as a reconstruction initial value, and reconstructing a single light field by using an R L deconvolution method to obtain a corresponding Xguss;

s34, interpolating Xgauge to obtain upXgauge;

s35, using a sigmoid function as a fusion proportion, and updating a corresponding layer of the Volume by upXgusess;

s36, selecting the next image, and repeating the steps S32-S35 until all the data in the data graph stack are calculated.

In order to achieve the above object, an embodiment of another aspect of the present invention provides an apparatus for reconstructing and splicing data of an axial scanning light field, including:

the acquisition module is used for acquiring the optical field image stack according to a set axial interval by using an axial scanning optical field system, and rearranging the optical field image stack to obtain a data image stack;

the calculation module is used for simulating a light path forwarding process of the axial scanning light field system through a computer to obtain a sub-aperture point diffusion function;

and the reconstruction module is used for reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction algorithm.

The axial scanning light field data reconstruction and splicing device provided by the embodiment of the invention utilizes an axial scanning light field system to acquire light field image stacks according to set axial intervals, and rearranges the light field image stacks to obtain data image stacks; simulating a light path forwarding process of an axial scanning light field system by a computer to obtain a sub-aperture point diffusion function; and reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation. Therefore, three-dimensional reconstruction results with large axial range, high resolution and axial continuity can be obtained.

In addition, the device for reconstructing and splicing axial scanning light field data according to the above embodiment of the present invention may further have the following additional technical features:

further, in an embodiment of the present invention, the acquisition module is further configured to perform axial down-sampling on the data map stack when a data amount of the data map stack is greater than a preset quantity value and a density of a single optical field map stack is greater than a preset density value.

Further, in an embodiment of the present invention, the calculation module is specifically configured to, in the axial scanning light field system, simulate a light path forwarding process by a computer starting from a point light source, calculate a plurality of light fields on the front surface of the microlens array, obtain light intensity distribution of the point light source on the sensor surface through phase modulation of the microlens array and a propagation process, and obtain the sub-aperture point spread function corresponding to each depth according to the light intensity distribution.

Further, in an embodiment of the present invention, the calculating module is further configured to perform axial non-equidistant downsampling on the sub-aperture point spread function, where a downsampling ratio is the same as that of the data map stack.

Further, in an embodiment of the invention, the reconstruction module is specifically configured to

Defining reconstruction data of the data graph stack, and numbering graphs in the data graph stack;

determining the range of the Volume to be reconstructed, carrying out all-0 initialization, and selecting a graph to be reconstructed;

extracting a corresponding layer in the Volume, using a preset proportion plus all 1 as a reconstruction initial value, and reconstructing a single light field by using an R L deconvolution method to obtain a corresponding Xgauge;

interpolating the Xgauge to obtain upXgauge;

using a sigmoid function as a fusion proportion, and updating a corresponding layer of the Volume by upXgusess;

and selecting the next image, and repeating the above processes for calculation until all the data in the data graph stack are calculated.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a flow chart of a method for reconstructing and stitching axial scan light field data according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method for reconstructing and stitching axial scan light field data according to an embodiment of the present invention;

FIG. 3 is a diagram of a light field and a rearrangement diagram according to an embodiment of the present invention;

FIG. 4 is a schematic view of a light field system according to one embodiment of the present invention;

FIG. 5 is a flow chart of a three-dimensional reconstruction and stitching algorithm according to one embodiment of the present invention;

fig. 6 is a schematic structural diagram of an apparatus for reconstructing and stitching axial scanning light field data according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

The following describes a method and an apparatus for reconstructing and stitching axial scanning light field data according to an embodiment of the present invention with reference to the drawings.

First, a method for reconstructing and stitching axial scanning light field data according to an embodiment of the present invention will be described with reference to the drawings.

Fig. 1 is a flowchart of a method for reconstructing and stitching axial scanning light field data according to an embodiment of the present invention.

As shown in fig. 1, the method for reconstructing and stitching axial scanning light field data includes the following steps:

and step S1, acquiring to obtain a light field map stack according to a set axial interval by using an axial scanning light field system, and rearranging the light field map stack to obtain a data map stack.

Further, in an embodiment of the present invention, S1 further includes: and when the data volume of the data map stack is greater than a preset number value and the density of a single optical field map stack is greater than a preset density value, performing axial down-sampling on the data map stack.

And step S2, simulating the light path forwarding process of the axial scanning light field system by a computer to obtain the sub-aperture point spread function.

Further, in an embodiment of the present invention, S2 further includes: in an axial scanning light field system, starting from a point light source, simulating a light path forward process by a computer, calculating a plurality of light fields on the front surface of a micro lens array, carrying out a propagation process through phase modulation of the micro lens array to obtain light intensity distribution of the point light source on the surface of a sensor, and obtaining sub-aperture point diffusion functions corresponding to all depths according to the light intensity distribution.

Further, in the embodiment of the present invention, the method further includes: and carrying out axial non-equal-interval downsampling on the sub-aperture point spread function, wherein the downsampling proportion is the same as that of the data graph stack.

And step S3, reconstructing the three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation.

It can be understood that the data is acquired by using an axial scanning light field system to obtain light field image stacks with certain axial intervals, the image stacks are arranged according to depth, the image stacks are reconstructed according to a certain sequence, and a reasonable initial value and reconstruction splicing method is adopted to obtain a three-dimensional reconstruction result with a large axial range, high resolution and axial continuity.

Referring to fig. 2, an embodiment of the present invention first acquires a light field stack RAWDATA stack at a set axial interval dz using an axial light field acquisition system. In the dz range, high-resolution three-dimensional reconstruction results can be obtained by using a single light field. The single light field map is rearranged to obtain 4D data, and the RAWDATA stack data is rearranged to obtain 5D data image stack, wherein the fifth dimension is the position of the current data in the stack. For example, if the size of the captured photoplethysmogram is 1300 × 1300 pixels, and the number of pixels corresponding to each microlens in the photoplethysmogram is 13 × 13, then the rearrangement can yield 4D data of 1300 × 13 pixels. If a total of 100 light field patterns are collected at dz intervals, the size of the raw stack is 1300 × 100 pixels, and the size of the 5D data after the rearrangement preprocessing is 1300 × 13 × 100 pixels. When the data volume of the 5D data image stack is large and the collection of a single light field is dense, the data can be transversely down-sampled. Meanwhile, the reconstruction axial range is large, and the intensity and contrast of the actual sampling point diffusion function cannot meet the requirements on a plane with a large defocus distance, so that the simulated sub-aperture point diffusion function psf can be used. If down-sampling is performed when the image stack is calculated, the lateral direction of the sub-aperture point spread function psf is also down-sampled proportionally. Meanwhile, when the defocus distance is large, the reconstruction resolution is reduced, and axial unequal-interval downsampling can be performed on the psf in order to accelerate the reconstruction speed. The psf and the image stack are used as input of a reconstruction algorithm, and a large-axial-range, high-resolution and axially continuous reconstruction result can be obtained.

Specifically, a stack of optical field patterns, RAWDATA stack, is acquired using an axial scanning optical field system. The light field map and the rearranged light field map are shown in fig. 3, the light field map obtains the angular resolution by sacrificing the spatial resolution, and because different angle information is obtained, three-dimensional information of the target scene can be reconstructed by a single light field map. And sampling and rearranging the light field image, and then interpolating to obtain a sub-aperture image corresponding to each sub-aperture. For example, the light field original size is 1300 × 1300 pixels, 4D data of 100 × 13 pixels can be obtained after rearrangement, 4D data of 1300 × 13 can be obtained after interpolation, the first 2 dimensions are space coordinates, and the last 2 dimensions are sub-aperture numbers. The right side of fig. 3 is a sub-aperture image corresponding to the sub-aperture (5, 6). When the spatial sampling of the optical field is relatively large, for accelerating the reconstruction process, the re-arranged data may be down-sampled, for example, the 4D data of 1300 × 13, and the spatial down-sampling may be performed by a factor of 2, so as to obtain a re-arranged image of 650 × 13 pixels. After the above operation is performed on all the data in the raw data stack, a 5-dimensional image stack can be obtained.

Starting from a point light source, simulating the light path forwarding process by a computer, calculating a complex light field on the front surface of a microlens array M L, and performing a propagation process through phase modulation of the microlens array to obtain light intensity distribution of the point light source on a sensor plane, so that sub-aperture point diffusion functions corresponding to various depths can be calculated.

And after the data image stack and the sub-aperture point diffusion function are obtained through the processes, reconstructing three-dimensional information of the target scene through a splicing algorithm. The process is shown in fig. 5, and specifically includes the following steps:

1) first, the order of rebuilding the image stack is defined, and the images in the stack are numbered. For example, the order of depth to lightness can be selected for numbering the stacks.

2) The Volume range to be reconstructed is determined and all 0's are initialized. An image that needs to be reconstructed is selected.

3) And extracting a corresponding layer in the Volume, adding all 1 in a certain proportion as a reconstruction initial value, and reconstructing a single light field by using methods such as R L deconvolution and the like to obtain the corresponding Xguss.

4) Interpolation is performed on Xgauge to obtain upXgauge.

5) And updating the corresponding layer of the Volume by upXgusess by using a sigmoid function as a fusion proportion.

6) And selecting the next image, and repeating the steps 2-5 until all the data in the image stack are calculated.

According to the method for reconstructing and splicing the axial scanning light field data, which is provided by the embodiment of the invention, an axial scanning light field system is utilized, the acquisition is carried out according to the set axial interval to obtain a light field image stack, and the light field image stack is rearranged to obtain a data image stack; simulating a light path forwarding process of an axial scanning light field system by a computer to obtain a sub-aperture point diffusion function; and reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation. Therefore, three-dimensional reconstruction results with large axial range, high resolution and axial continuity can be obtained.

Next, a reconstruction and concatenation apparatus for axial scanning light field data according to an embodiment of the present invention will be described with reference to the drawings.

Fig. 6 is a schematic structural diagram of an apparatus for reconstructing and stitching axial scanning light field data according to an embodiment of the present invention.

As shown in fig. 6, the apparatus for reconstructing and stitching axial scan light field data includes:

the acquisition module 100 is configured to acquire an optical field stack according to a set axial interval by using an axial scanning optical field system, and rearrange the optical field stack to obtain a data stack.

And the calculating module 200 is configured to simulate a light path forwarding process of the axial scanning light field system by using a computer to obtain a sub-aperture point spread function.

And a reconstruction module 300, configured to reconstruct three-dimensional information of the target scene according to the data map stack and the sub-aperture point spread function and the reconstruction algorithm.

Further, in an embodiment of the present invention, the acquisition module is further configured to perform axial down-sampling on the data map stack when the data amount of the data map stack is greater than a preset number value and the density of a single optical field map stack is greater than a preset density value.

Further, in an embodiment of the present invention, the calculation module is specifically configured to, in an axial scanning light field system, simulate a light path forwarding process by a computer starting from a point light source, calculate a plurality of light fields on the front surface of the microlens array, obtain light intensity distribution of the point light source on the sensor surface through phase modulation of the microlens array and a propagation process, and obtain a sub-aperture point spread function corresponding to each depth according to the light intensity distribution.

Further, in an embodiment of the present invention, the calculation module is further configured to perform axial non-equidistant downsampling on the sub-aperture point spread function, where a downsampling ratio is the same as that of the data map stack.

Further, in an embodiment of the invention, the reconstruction module is specifically adapted for

The method comprises the steps of specifying reconstructed data of a data diagram stack, and numbering diagrams in the data diagram stack;

determining the range of the Volume to be reconstructed, carrying out all-0 initialization, and selecting a graph to be reconstructed;

extracting a corresponding layer in the Volume, using a preset proportion plus all 1 as a reconstruction initial value, and reconstructing a single light field by using an R L deconvolution method to obtain a corresponding Xgauge;

interpolating the Xgauge to obtain upXgauge;

using a sigmoid function as a fusion proportion, and updating a corresponding layer of the Volume by upXgusess;

and selecting the next image, and repeating the above processes for calculation until all the data in the data graph stack are calculated.

It should be noted that the foregoing explanation on the embodiment of the method for reconstructing and stitching axial scanning light field data is also applicable to the apparatus of this embodiment, and is not repeated herein.

According to the device for reconstructing and splicing the axial scanning light field data, which is provided by the embodiment of the invention, an axial scanning light field system is utilized, the acquisition is carried out according to the set axial interval to obtain a light field image stack, and the light field image stack is rearranged to obtain a data image stack; simulating a light path forwarding process of an axial scanning light field system by a computer to obtain a sub-aperture point diffusion function; and reconstructing three-dimensional information of the target scene according to the data map stack, the sub-aperture point spread function and the reconstruction calculation. Therefore, three-dimensional reconstruction results with large axial range, high resolution and axial continuity can be obtained.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.