阅读说明：本技术 一种拟合深度图像的几何运动细节重建方法及装置 (Geometric motion detail reconstruction method and device for fitting depth image ) 是由 徐枫 郑成伟 于 2021-07-01 设计创作，主要内容包括：本申请提出一种拟合深度图像的几何运动细节重建方法和装置,其中,方法包括：构造含有数据项的势能函数,根据势能函数计算位置偏移量大小；其中,数据项用于拟合深度数据,约束几何表面点在添加位置偏移量之后与输入的深度图像接近；根据位置偏移量大小,基于粗糙基础几何运动在几何表面点添加位置偏移量,位置偏移量在空间中的方向为沿相机视线方向；根据位置偏移量对细节几何运动进行重建。能够基于粗糙几何运动计算物体表面点的准确偏移量,从而重建物体的细节几何运动且不同表面点的计算过程完全并行进行,运行效率极高。(The application provides a geometric motion detail reconstruction method and device for fitting a depth image, wherein the method comprises the following steps: constructing a potential energy function containing a data item, and calculating the position offset according to the potential energy function; wherein the data items are used for fitting depth data, and the constrained geometric surface points are close to the input depth image after adding the position offset; adding position offset on the geometric surface point based on the rough basic geometric motion according to the position offset, wherein the position offset is in the direction along the sight line of the camera in the space; and reconstructing the detail geometric motion according to the position offset. The accurate offset of the surface points of the object can be calculated based on the rough geometric motion, so that the detailed geometric motion of the object is reconstructed, the calculation processes of different surface points are completely performed in parallel, and the operation efficiency is extremely high.)

1. A method of geometric motion detail reconstruction fitting a depth image, the method comprising the steps of:

constructing a potential energy function containing a data item, and calculating the position offset according to the potential energy function; wherein the data items are used to fit depth data, constraining the geometric surface points to approximate the input depth image after adding the positional offset;

adding a position offset at a geometric surface point based on the rough basic geometric motion according to the position offset, wherein the position offset is in the direction along the sight line of the camera in the space;

and reconstructing the detailed geometric motion according to the position offset.

2. The method of reconstructing geometric motion details of a fitted depth image of claim 1, wherein said constructing a potential energy function containing data items further comprises: the term of regularization is used to describe the term,

the regularization term constrains the amount of displacement of a geometric surface point to be close to the average of the amounts of displacement of the geometric surface point of the previous frame.

3. The geometric motion detail reconstruction method of fitting depth images according to claim 1, wherein said adding a position offset at a geometric surface point based on a coarse basis geometric motion comprises:

adding a position offset to each geometric surface point in the form of a grid and representing the three-dimensional spatial position coordinates of said each geometric surface point.

4. The geometric motion detail reconstruction method of fitting depth image according to claim 3, wherein the offset of each mesh vertex is calculated, and the magnitude of the offset is calculated by the following formula:

E_geo(d)＝E_fit(d)+w_regE_reg(d)

wherein d is the magnitude of the offset, E_fitFor data items, E_regBeing a regularization term, w_regIs a predefined weight of the regularization term.

5. Method for geometric motion detail reconstruction of fitted depth images according to claim 4, characterized in that said data item E_fitIs calculated by the following formula:

E_fit(d)＝(D_base+d-D_in)²

wherein D is_baseIs the distance of the base geometrical vertex from the camera after the base movement is acted on, D_inThe depth value of the input image in the frame corresponding to the vertex.

6. The geometric motion detail reconstruction method of fitting depth image according to claim 4, wherein the regularization term E_regCalculated by the following formula:

where N represents the neighborhood set.

7. The method of reconstructing geometric motion detail of a fitted depth image according to claim 1, further comprising: a set of voxels is created and the average offset of grid vertices within each voxel is stored.

8. A geometric motion detail reconstruction apparatus that fits a depth image, comprising:

the construction module is used for constructing a potential energy function containing a data item and calculating the position offset according to the potential energy function; wherein the data items are used to fit depth data, constraining the geometric surface points to approximate the input depth image after adding the positional offset;

the adding module is used for adding position offset on the basis of rough basic geometric motion at a geometric surface point according to the position offset, and the direction of the position offset in the space is along the sight line direction of the camera;

and the reconstruction module is used for reconstructing the detailed geometric motion according to the position offset.

9. The apparatus according to claim 8, wherein the construction module is further configured to constrain the geometric surface point shift amount to approach an average of the geometric surface point shift amounts of the previous frame by a regularization term.

10. The apparatus for reconstructing geometric motion details of a fitted depth image according to claim 8, further comprising: and the voxel set module is used for establishing a voxel set and storing the average offset of grid vertexes in each voxel.

Technical Field

The invention relates to the technical field of computer vision and computer graphics, in particular to a geometric motion detail reconstruction method for a fitting depth image.

Background

Three-dimensional geometric reconstruction has been a very important and widely used research in computer graphics and vision. With the development of depth cameras, three-dimensional geometric reconstruction techniques have been widely applied to professional-level applications and daily lives of general users. The reconstruction of geometric motion details of moving objects is crucial to obtain high quality reconstruction results. The reconstruction of the geometric motion details brings obvious promotion to the applications of rendering image sequences with strong reality sense, object editing based on geometry, relighting and the like.

Most of the existing methods focus on reconstruction of large-scale motion and neglect reconstruction of geometric motion details, and some existing detail reconstruction methods are difficult to achieve high reconstruction quality and high operation efficiency at the same time. For reconstructing a motion sequence of a dynamic object, the invention provides a geometric motion detail reconstruction method for fitting a depth image, which is characterized in that the high-quality reconstruction of the geometric motion detail of a dynamic object is efficiently completed by means of the input of a depth camera, the depth data of the depth camera is fitted and a regular term is added.

Disclosure of Invention

The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, the first objective of the present invention is to provide a method for reconstructing geometric motion details of a fitted depth image, which rapidly calculates the detail position offset of a reconstructed geometric surface point by a method for fitting depth data on the basis of rough geometric and motion reconstruction, so as to model the geometric motion details of an object surface.

A second object of the present invention is to provide a geometric motion detail reconstruction apparatus that fits a depth image.

In order to achieve the above object, an embodiment of a first aspect of the present invention provides a method for reconstructing details of geometric motion of a fitted depth image, including:

s1, constructing a potential energy function containing a data item, and calculating the position offset according to the potential energy function; wherein the data items are used to fit depth data, constraining the geometric surface points to approximate the input depth image after adding the positional offset;

s2, adding a position offset at a geometric surface point based on the geometric motion of a rough foundation according to the position offset, wherein the position offset is in the direction along the sight line of the camera in the space;

and S3, reconstructing the detail geometric movement according to the position offset.

In addition, the geometric motion detail reconstruction method for fitting a depth image according to the above embodiment of the present invention may further have the following additional technical features:

further, in an embodiment of the present invention, the constructing a potential energy function containing data items further includes: a regularization term that constrains a geometric surface point offset to be close to an average of the geometric surface point offsets of a previous frame.

Further, in an embodiment of the present invention, the adding a position offset at the geometric surface point based on the rough base geometric motion includes: adding a position offset to each geometric surface point in the form of a grid and representing the three-dimensional spatial position coordinates of said each geometric surface point.

Further, in one embodiment of the present invention, the offset of each mesh vertex is calculated, and the offset is calculated by the following formula:

E_geo(d)＝E_fit(d)+w_regE_reg(d)

wherein d is the magnitude of the offset, E_fitFor data items, E_regBeing a regularization term, w_regIs a predefined weight of the regularization term.

Further, in one embodiment of the present invention, the data item E_fitIs calculated by the following formula:

E_fit(d)＝(D_base+d-D_in)²

wherein D is_baseIs the distance of the base geometrical vertex from the camera after the base movement is acted on, D_inThe depth value of the input image in the frame corresponding to the vertex.

Further, in one embodiment of the present invention, the regularization term E_regCalculated by the following formula:

where N represents the neighborhood set.

Further, in an embodiment of the present invention, the method further includes: a set of voxels is created and the average offset of grid vertices within each voxel is stored.

According to the geometric movement detail reconstruction method for the fitting depth image, disclosed by the embodiment of the invention, the position offset is calculated according to the potential energy function by constructing the potential energy function containing the data item, wherein the data item is used for fitting depth data, the geometric surface point is constrained to be close to the input depth image after the position offset is added, the position offset is added to the geometric surface point based on the rough basic geometric movement according to the position offset, the direction of the position offset in the space is along the sight line direction of the camera, and the detail geometric movement is reconstructed according to the position offset. The method is based on the rough basic geometry and the movement which are obtained through reconstruction, the offset of one position is added to each surface point, the accurate offset of the surface point of the object can be calculated based on the rough geometric movement, so that the detailed geometric movement of the object is reconstructed, the calculation processes of different surface points are completely carried out in parallel, the operation efficiency is extremely high, and the rough basic geometry is allowed to have the change on the mesh topology.

To achieve the above object, a second embodiment of the present invention provides a geometric motion detail reconstruction apparatus for fitting a depth image, including:

the construction module is used for constructing a potential energy function containing a data item and calculating the position offset according to the potential energy function; wherein the data items are used to fit depth data, constraining the geometric surface points to approximate the input depth image after adding the positional offset;

the adding module is used for adding position offset on the basis of rough basic geometric motion at a geometric surface point according to the position offset, and the direction of the position offset in the space is along the sight line direction of the camera;

and the reconstruction module is used for reconstructing the detailed geometric motion according to the position offset.

According to the geometric movement detail reconstruction device for fitting the depth image, disclosed by the embodiment of the invention, the position offset is calculated according to the potential energy function by constructing the potential energy function containing the data item, wherein the data item is used for fitting the depth data, the geometric surface point is constrained to be close to the input depth image after the position offset is added, the position offset is added to the geometric surface point based on the rough basic geometric movement according to the position offset, and the direction of the position offset in the space is along the sight line direction of the camera. And reconstructing the detail geometric motion according to the position offset. The method is based on the rough basic geometry and the movement which are obtained through reconstruction, the offset of one position is added to each surface point, the accurate offset of the surface point of the object can be calculated based on the rough geometric movement, so that the detailed geometric movement of the object is reconstructed, the calculation processes of different surface points are completely carried out in parallel, the operation efficiency is extremely high, and the rough basic geometry is allowed to have the change on the mesh topology.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

fig. 1 is a flowchart of a geometric motion detail reconstruction method of fitting a depth image according to an embodiment of the present invention.

Fig. 2 is a schematic structural diagram of a geometric motion detail reconstruction apparatus for fitting a depth image according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

The following describes a geometric motion detail reconstruction method and apparatus for fitting a depth image according to an embodiment of the present invention with reference to the drawings.

According to the geometric movement detail reconstruction method for the fitting depth image, based on the rough basic geometry and movement obtained through reconstruction, the offset of one position is added to each surface point, the offset enables the surface points of the object to have more accurate three-dimensional space position coordinates, and therefore the reconstruction of geometric details is more accurate.

Fig. 1 is a flowchart of a geometric motion detail reconstruction method for fitting a depth image according to an embodiment of the present invention.

As shown in fig. 1, the method for reconstructing geometric motion details of a fitted depth image includes:

step S1, constructing a potential energy function containing data items, and calculating the position offset according to the potential energy function; wherein the data items are used to fit depth data, constraining the geometric surface points to approximate the input depth image after adding the positional offset.

The magnitude of the offset is calculated by using a method of minimizing a potential energy function, and the potential energy function proposed in this embodiment includes two terms, a data term and a regular term. Wherein the data items are used to fit depth data that constrains the object surface points to be close to the input depth image after the offset is added. The offset is added to the reconstructed coarse moved base geometry. The regular term enables the reconstruction result to be smoother in space and time, and the offset of a certain surface point is restrained to be close to the average value of the offsets of nearby surface points in the previous frame, so that the reconstruction is facilitated to obtain a smoother result. .

Specifically, the following energy functions are constructed for calculating the magnitude of the offset

E_geo(d)＝E_fit(d)+w_regE_reg(d)

Wherein d is the magnitude of the offset, E_fitFor data items, E_regBeing a regularization term, w_regIs a predefined weight of the regularization term.

Data item E_fitFitting the input depth image:

E_fit(d)＝(D_base+d-D_in)²

where D is_baseIs the distance of the base geometric vertex from the camera after the base motion is applied, and D_inIs the depth value of the input image in the frame corresponding to the vertex. The increased offset is in the camera line of sight direction.

Regularization term E_regExpressed as:

where N represents the neighborhood set.

And S2, adding a position offset at the geometric surface point based on the rough basic geometric motion according to the position offset, wherein the position offset is in the direction along the sight line of the camera in the space.

It is understood that the direction of the offset added to the surface point in the space in the present embodiment is along the camera sight line direction, and therefore, the magnitude of the offset only needs to be determined in calculation.

It is understood that the coarse after-motion basic geometry reconstructed in the present embodiment can be expressed by using a general form of mesh, and based on the expression, the offset of each mesh vertex is calculated to reconstruct the detailed geometric motion.

Specifically, an offset of one position is added to each surface point of the object, and the offset enables the surface point of the object to have more accurate three-dimensional space position coordinates.

And S3, reconstructing the detail geometric movement according to the position offset.

It will be appreciated that the present invention computes the offset of each mesh vertex to reconstruct the detailed geometric motion, which is added to the reconstructed coarse moved base geometry, while the mesh topology between different frames is allowed to change.

It can be understood that if the rough basic geometry uses a general mesh expression, the mesh topology may change between different frames, which makes it difficult to find the corresponding relationship of the same mesh vertex between different frames. Thus, a voxel set is created and used to store the average offset of the surface points in the voxels of the previous frame, and the voxel set is located in the motion pose of the reference frame, i.e. the first frame of the sequence, so that the same surface point also corresponds to the same voxel of the voxel set in different motion poses in different frames.

Since the topology of the underlying mesh may change during the reconstruction process, a set of voxels is constructed in the reference frame, i.e. the first frame of the sequence, to store the average offset of the vertices in each voxel of the previous frame. Thus, the same surface point also corresponds to the same voxel of the voxel set under different motion postures in different frames, and the offset in the neighborhood of the previous frame can be directly obtained from the voxel set, wherein each neighbor corresponds to a voxel. The voxel will then be updated for the solution of the next frame.

Based on these designs, the above-described detail fitting method can compute the offset of each vertex independently and completely in parallel, which enables the system to efficiently reconstruct high-quality geometry without the need to solve large-scale linear problems. Also, the method can be conveniently integrated into other real-time systems without affecting the real-time performance of the systems.

It should be noted that the data items used in the present invention to fit the depth input are only valid when the vertices correspond to valid depth values. If a vertex corresponds to a pixel without depth data, only the regularization term is used to calculate its offset.

Fig. 2 is a schematic structural diagram of a geometric motion detail reconstruction apparatus for fitting a depth image according to an embodiment of the present invention.

As shown in fig. 2, the geometric motion detail reconstruction apparatus 10 for fitting a depth image includes:

a construction module 100, an addition module 200, and a reconstruction module 300.

A construction module 100, configured to construct a potential energy function including a data item, and calculate a position offset according to the potential energy function; wherein the data items are used for fitting depth data, and the constrained geometric surface points are close to the input depth image after adding the position offset;

the adding module 200 is used for adding position offset on the geometric surface point based on the rough basic geometric motion according to the position offset, wherein the direction of the position offset in the space is along the sight line direction of the camera;

and a reconstruction module 300, configured to reconstruct the detailed geometric motion according to the position offset.

Further, the construction module 100 is further configured to constrain a geometric surface point offset to approach an average of geometric surface point offsets of a previous frame by a regularization term.

Further, the apparatus 10 further comprises: a voxel set module 400 for establishing a voxel set and storing the average offset of the grid vertices within each voxel.

According to the geometric movement detail reconstruction device for fitting the depth image, provided by the embodiment of the invention, the position offset is calculated according to the potential energy function by constructing the potential energy function containing the data item, wherein the data item is used for fitting the depth data, the geometric surface point is constrained to be close to the input depth image after the position offset is added, the position offset is added to the geometric surface point based on the rough basic geometric movement according to the position offset, and the direction of the position offset in the space is along the sight line direction of the camera. And reconstructing the detail geometric motion according to the position offset. The method is based on the rough basic geometry and the movement which are obtained through reconstruction, the offset of one position is added to each surface point, the accurate offset of the surface point of the object can be calculated based on the rough geometric movement, so that the detailed geometric movement of the object is reconstructed, the calculation processes of different surface points are completely carried out in parallel, the operation efficiency is extremely high, and the rough basic geometry is allowed to have the change on the mesh topology.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.