阅读说明：本技术 模型优化方法及相关装置、电子设备和存储介质 (Model optimization method and related device, electronic equipment and storage medium ) 是由 李辰宸 项骁骏 余亦豪 于 2021-08-26 设计创作，主要内容包括：本申请公开了一种模型优化方法及相关装置、电子设备和存储介质,其中,模型优化方法包括：获取待优化模型；其中,待优化模型包括若干模型面片,模型面片由模型顶点构成；获取各个模型面片的属性信息和第一法向信息,并获取各个模型顶点的第一位置信息；基于属性信息优化第一法向信息,得到第二法向信息；基于第一法向信息和第二法向信息优化第一位置信息,得到模型顶点的第二位置信息,以对待优化模型进行优化。上述方案,能够提升模型优化质量和效率。(The application discloses a model optimization method, a related device, electronic equipment and a storage medium, wherein the model optimization method comprises the following steps: obtaining a model to be optimized; the model to be optimized comprises a plurality of model patches, and each model patch is composed of a model vertex; acquiring attribute information and first normal information of each model patch, and acquiring first position information of each model vertex; optimizing the first normal information based on the attribute information to obtain second normal information; and optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized. According to the scheme, the model optimization quality and efficiency can be improved.)

1. A method of model optimization, comprising:

obtaining a model to be optimized; the model to be optimized comprises a plurality of model patches, wherein each model patch is composed of model vertexes;

acquiring attribute information and first normal information of each model surface patch, and acquiring first position information of each model vertex;

optimizing the first normal information based on the attribute information to obtain second normal information;

and optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

2. The method of claim 1, wherein the first location information comprises first coordinates and the second location information comprises second coordinates, and wherein optimizing the first location information based on the first normal information and the second normal information to obtain second location information for vertices of the model comprises:

respectively taking each model vertex as a first current vertex, selecting one model vertex with a connecting edge between the model vertex and the first current vertex as a reference vertex, and respectively taking two model patches sharing the connecting edge as a first patch and a second patch;

obtaining a first normal transformation parameter of the first surface patch based on the first normal information and the second normal information of the first surface patch, and obtaining a second normal transformation parameter of the second surface patch based on the first normal information and the second normal information of the second surface patch;

constructing an objective function of the first current vertex based on the first normal transformation parameter and the second normal transformation parameter; wherein the objective function includes a first reference sub-term and a second reference sub-term, the first reference sub-term is a product of a preset reference coefficient and a first coordinate of the first current vertex, and the second reference sub-term is a product of the preset reference coefficient and a second coordinate of the first current vertex;

and solving to obtain a second coordinate of each model vertex based on the objective function of each model vertex.

3. The method of claim 2, wherein constructing the objective function for the first current vertex based on the first normal transformation parameter and the second normal transformation parameter comprises:

acquiring a first diagonal of the connecting edge in the first patch, and acquiring a second diagonal of the connecting edge in the second patch;

constructing a first optimized sub-item based on the first normal transformation parameter and the first diagonal, the second normal transformation parameter and the second diagonal, and the first coordinate of the first current vertex and the first coordinate of the reference vertex;

constructing a second optimized sub-item based on the first diagonal of the first patch, the second diagonal of the second patch, the second coordinate of the first current vertex and the second coordinate of the reference vertex;

and obtaining an objective function of the first current vertex based on the first optimization sub-item, the first reference sub-item, the second optimization sub-item and the second reference sub-item.

4. The method of claim 2, wherein solving for the second coordinate of each of the model vertices based on the objective function of each of the model vertices comprises:

simultaneously establishing a set of equations for second coordinates of each of the model vertices based on the objective function of each of the model vertices;

and solving the equation set to obtain a second coordinate of each model vertex.

5. The method according to any one of claims 1 to 4, wherein before optimizing the first location information based on the first normal information and the second normal information to obtain second location information of vertices of the model for optimizing the model to be optimized, the method further comprises:

detecting that the current optimization stage meets a preset condition, and acquiring the optimization loss of the model surface patch based on the attribute information, the first normal information and the second normal information of the model surface patch;

selecting at least one model patch as a patch to be subdivided in the current optimization stage based on the optimization loss;

and subdividing the model to be optimized based on the patch to be subdivided to obtain the subdivided model to be optimized, so as to update the model patch of the model to be optimized, and re-executing the step of obtaining the attribute information and the first normal information of each model patch and the subsequent steps.

6. The method of claim 5, wherein the selecting at least one of the model patches as the patch to be subdivided of the current optimization stage based on the optimization loss comprises:

sequencing the model patches according to the sequence of the optimization losses from large to small, and selecting the model patches positioned at the front preset sequence position as the patches to be subdivided;

or selecting the model patch with the optimization loss larger than a first threshold value as the patch to be subdivided of the patch to be subdivided.

7. The method of claim 5, wherein the subdividing the model to be optimized based on the patch to be subdivided to obtain a subdivided model to be optimized comprises:

adding at least one model vertex on the edge of the patch to be subdivided;

respectively taking each newly added model vertex as a second current vertex, and acquiring a candidate vertex set of the second current vertex; the candidate vertex set comprises a plurality of candidate vertices, the candidate vertices are positioned in a plurality of candidate patches, and the candidate patches all comprise edges where the second current vertex is positioned;

and selecting at least one candidate vertex from the candidate vertex set, and respectively connecting the at least one candidate vertex and the second current vertex to obtain a subdivided model to be optimized.

8. The method of claim 5, further comprising:

and a step of detecting that the current optimization stage does not meet the preset condition, and executing the step of optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

9. The method of claim 5, wherein before subdividing the model to be optimized based on the patch to be subdivided, the method further comprises:

screening the patches to be subdivided with the patch area not lower than a second threshold value;

the subdividing of the model to be optimized based on the patch to be subdivided comprises the following steps:

and subdividing the model to be optimized based on the patch to be subdivided obtained by screening to obtain the subdivided model to be optimized.

10. The method according to claim 5, wherein the preset conditions include: and the current optimization stage does not reach the preset numerical optimization stages.

11. A model optimization apparatus, comprising:

the model obtaining module is used for obtaining a model to be optimized; the model to be optimized comprises a plurality of model patches, wherein each model patch is composed of model vertexes;

the information acquisition module is used for acquiring attribute information and first normal information of each model surface patch and acquiring first position information of each model vertex;

the normal optimization module is used for optimizing the first normal information based on the attribute information to obtain second normal information;

and the vertex optimization module is used for optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

12. An electronic device comprising a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the model optimization method of any one of claims 1 to 10.

13. A computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement the model optimization method of any one of claims 1 to 10.

Technical Field

The present application relates to the field of computer vision technologies, and in particular, to a model optimization method, a related apparatus, an electronic device, and a storage medium.

Background

In three-dimensional reconstruction applications, the finally generated reconstructed model often lacks a lot of geometric details due to software-level reasons such as reconstruction algorithms or hardware-level reasons such as sensor accuracy. Therefore, it is necessary to optimize the details of the reconstructed model to improve the quality of the model.

At present, an indirect representation form of a three-dimensional model such as an optimized Truncated Signed Distance Field (TSDF) is usually selected in an existing detail optimization mode, a solving process is often complex, an optimized result can be reflected on a final three-dimensional model only through an additional network construction algorithm, additional algorithm overhead is introduced, and a part of details of the network construction algorithm may be lost again. In view of this, how to improve the quality and efficiency of model optimization becomes an urgent problem to be solved.

Disclosure of Invention

The application provides a model optimization method, a relevant device, electronic equipment and a storage medium.

A first aspect of the present application provides a model optimization method, including: obtaining a model to be optimized; the model to be optimized comprises a plurality of model patches, and each model patch is composed of a model vertex; acquiring attribute information and first normal information of each model patch, and acquiring first position information of each model vertex; optimizing the first normal information based on the attribute information to obtain second normal information; and optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

Therefore, the model to be optimized is obtained, the model to be optimized comprises a plurality of model patches, each model patch is composed of model vertexes, the attribute information and the first normal information of each model patch are obtained, the first position information of each model vertex is obtained, the first normal information is optimized based on the attribute information, the second normal information is obtained, the first position information is optimized based on the first normal information and the second normal information, the second position information of each model vertex is obtained, and the model to be optimized is optimized, on one hand, because the model is directly optimized through the normal information before and after optimization without indirect representation forms such as a truncated symbolic distance field and the like, the algorithm overhead can be greatly reduced, the optimization efficiency is improved, on the other hand, because the connection relation between each model patch and each model vertex is considered in the optimization process, therefore, the optimization quality is improved, and the quality and the efficiency of model optimization can be improved.

Wherein, the first position information includes a first coordinate, the second position information includes a second coordinate, the first position information is optimized based on the first normal information and the second normal information, and the second position information of the model vertex is obtained, which includes: respectively taking each model vertex as a first current vertex, selecting one model vertex with a connecting edge between the model vertex and the first current vertex as a reference vertex, and respectively taking two model patches sharing the connecting edge as a first patch and a second patch; obtaining a first normal transformation parameter of the first surface patch based on the first normal information and the second normal information of the first surface patch, and obtaining a second normal transformation parameter of the second surface patch based on the first normal information and the second normal information of the second surface patch; constructing a target function of the first current vertex based on the first normal transformation parameter and the second normal transformation parameter; the target function comprises a first reference subentry and a second reference subentry, wherein the first reference subentry is the product of a preset reference coefficient and a first coordinate of the first current vertex, and the second reference subentry is the product of the preset reference coefficient and a second coordinate of the first current vertex; and solving to obtain a second coordinate of each model vertex based on the objective function of each model vertex.

Therefore, the first position information is set to include a first coordinate, the second position information is set to include a second coordinate, each model vertex is respectively used as a first current vertex, a model vertex most reference vertex with a connecting edge between the model vertex and the first current vertex is selected, two model patches sharing the connecting edge are respectively used as a first patch and a second patch, on the basis, a first normal transformation parameter of the first patch is obtained on the basis of first normal information and second normal information of the first patch, a second normal transformation parameter of the second patch is obtained on the basis of the first normal information and second normal information of the second patch, and accordingly, an objective function of the first current vertex is constructed on the basis of the first normal transformation parameter and the second normal transformation parameter, the objective function comprises a first reference sub-item and a second reference sub-item, the first reference sub-item is a product of a preset reference coefficient and the first coordinate of the first current vertex, the second reference subentry is the product of the preset reference coefficient and the second coordinate of the first current vertex, and then the second coordinate of each model vertex is obtained through solving based on the objective function of each model vertex, so that in the optimization process of the position coordinates of the model vertices, the product of the preset reference coefficient and the first coordinate of the model vertex and the product of the preset reference coefficient and the second coordinate of the model vertex are considered at the same time and are respectively used as subentries of the objective function, the fault tolerance of objective function solving is favorably improved, and the stability of the solving result is improved.

The method for constructing the objective function of the first current vertex based on the first normal transformation parameter and the second normal transformation parameter comprises the following steps: acquiring a first diagonal of a connecting edge in a first patch, and acquiring a second diagonal of a connecting edge in a second patch; constructing a first optimization sub-item based on the first normal transformation parameter, the first opposite angle, the second normal transformation parameter, the second opposite angle, the first coordinate of the first current vertex and the first coordinate of the reference vertex; constructing a second optimization sub-item based on the first diagonal of the first patch, the second diagonal of the second patch, the second coordinate of the first current vertex and the second coordinate of the reference vertex; and obtaining the objective function of the first current vertex based on the first optimization sub-item, the first reference sub-item, the second optimization sub-item and the second reference sub-item.

Therefore, a first object of a connecting edge in a first patch is obtained, a second opposite angle of the connecting edge in a second patch is obtained, a first optimization sub-item is constructed based on a first normal transformation parameter, the first opposite angle, a second normal transformation parameter, a second object, a first coordinate of a first current vertex and a first coordinate of a reference vertex, and a second optimization sub-item is constructed based on the first opposite angle of the first patch, the second opposite angle of the second patch, the second coordinate of the first current vertex and the second coordinate of the reference vertex.

Solving to obtain a second coordinate of each model vertex based on the objective function of each model vertex, wherein the method comprises the following steps: simultaneously establishing a system of equations for second coordinates of each model vertex based on the objective function of each model vertex; and solving the equation set to obtain second coordinates of the vertexes of the models.

Therefore, based on the objective function of each model vertex, the equation sets of the second coordinates of each model vertex are simultaneously established, and the equation sets are solved to obtain the second coordinates of each model vertex, so that the second coordinates of each model vertex can be obtained by combining the equation sets of the second coordinates of each model vertex through one-time solution, and the model optimization efficiency is further improved.

Before optimizing the first position information based on the first normal information and the second normal information to obtain the second position information of the model vertex so as to optimize the model to be optimized, the method further comprises the following steps: detecting that the current optimization stage meets a preset condition, and acquiring the optimization loss of the model patch based on the attribute information, the first normal information and the second normal information of the model patch; based on optimization loss, selecting at least one model patch as a patch to be subdivided in the current optimization stage; and subdividing the model to be optimized based on the patch to be subdivided to obtain the subdivided model to be optimized, so as to update the model patch of the model to be optimized, and re-executing the step of obtaining the attribute information and the first normal information of each model patch and the subsequent steps.

Therefore, before optimization, when it is detected that the current optimization stage meets preset conditions, optimization losses of the model patches are obtained based on the attribute information, the first normal information and the second normal information of the model patches, so that based on the optimization losses, at least one model patch is selected as a patch to be subdivided in the current optimization stage, and based on the patch to be subdivided, the model to be optimized is subdivided to obtain a subdivided model to be optimized, so as to update the model patch of the model to be optimized, and the steps of obtaining the attribute information and the first normal information of each model patch and the subsequent steps are re-executed, that is, in the process of model optimization, the model to be optimized is subdivided according to the patch to be subdivided by detecting the optimization losses of each model patch and selecting the patch to be subdivided according to the optimization losses, so as to further promote the details of the model, and the optimization quality of the model is improved.

Based on optimization loss, selecting at least one model patch as a patch to be subdivided in the current optimization stage, wherein the method comprises the following steps: sequencing the model patches according to the sequence of the optimization losses from large to small, and selecting the model patches positioned at the front preset sequence position as patches to be subdivided; or selecting the model patch with the optimization loss larger than the first threshold value as the patch to be subdivided.

Therefore, the model patches are sorted according to the sequence of the optimization losses from large to small, the model patch positioned at the front preset order position is selected as the model to be optimized, or the model patch with the optimization loss larger than the first threshold is directly selected as the patch to be subdivided, and the model to be optimized can be refined based on the model patch with the large optimization loss, so that the detailed optimization of the part with poor details can be facilitated, and the optimization quality of the model can be improved.

Wherein, based on the facet piece to be subdivided, subdividing the model to be optimized to obtain the subdivided model to be optimized, including: adding at least one model vertex on the edge of the patch to be subdivided; respectively taking each newly added model vertex as a second current vertex, and acquiring a candidate vertex set of the second current vertex; the candidate vertex set comprises a plurality of candidate vertices, the candidate vertices are positioned on a plurality of candidate patches, and the candidate patches comprise edges where the second current vertex is positioned; and selecting at least one candidate vertex from the candidate vertex set, and respectively connecting the at least one candidate vertex and the second current vertex to obtain the subdivided model to be optimized.

Therefore, at least one model vertex is newly added on the edge of the patch to be subdivided, each newly added model vertex is respectively used as a second current vertex, a candidate vertex set of the second current vertex is obtained, the candidate vertex set comprises a plurality of candidate vertices, the candidate vertices are positioned on a plurality of candidate patches, and the candidate patches all comprise the edge where the second current vertex is positioned.

The model optimization method further comprises the following steps: and when the current optimization stage is detected not to meet the preset condition, optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex, so as to optimize the model to be optimized.

Therefore, when the current optimization stage is detected not to meet the preset condition, the first position information is optimized based on the first normal information and the second normal information, and the second position information of the model vertex is obtained, so that the model to be optimized is optimized.

Before subdividing the model to be optimized based on the patch to be subdivided and obtaining the subdivided model to be optimized, the model optimization method further comprises the following steps: screening the patches to be subdivided with the patch area not lower than a second threshold value; based on the to-be-subdivided patch, subdividing the to-be-optimized model to obtain a subdivided to-be-optimized model, comprising: and subdividing the model to be optimized based on the patch to be subdivided obtained by screening to obtain the subdivided model to be optimized.

Therefore, under the condition that the to-be-subdivided surface patches are selected based on optimization loss, the areas of the surface patches are further screened, so that a to-be-optimized model can be refined based on a model surface patch with large optimization loss and large surface patch area, on one hand, because the contribution of the model surface patch with small surface patch area to model optimization is very little, the negative influence on optimization efficiency due to excessive subdivision of the model surface patch with small surface patch area can be avoided, the optimization efficiency of the model is favorably improved, on the other hand, because the position coordinate distinguishing difficulty of the model vertex of the model surface patch with small surface patch area is also large, the stability of the optimization of the model is favorably improved.

Wherein the preset conditions include: the current optimization stage does not reach the preset number of optimization stages.

Therefore, by setting the preset conditions to include: the current optimization stage does not reach the preset number of optimization stages, so that the patches to be subdivided can be selectively subdivided as required, and the optimization quality of the model is improved as much as possible on the premise of controlling the increase of the time overhead.

A second aspect of the present application provides a model optimization apparatus, including: the model optimization system comprises a model acquisition module, an information acquisition module, a normal optimization module and a vertex optimization module, wherein the model acquisition module is used for acquiring a model to be optimized; the model to be optimized comprises a plurality of model patches, and each model patch is composed of a model vertex; the information acquisition module is used for acquiring attribute information and first normal information of each model surface patch and acquiring first position information of each model vertex; the normal optimization module is used for optimizing the first normal information based on the attribute information to obtain second normal information; and the vertex optimization module is used for optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

A third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement the model optimization method in the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium having stored thereon program instructions that, when executed by a processor, implement the model optimization method of the first aspect described above.

According to the scheme, the model to be optimized is obtained and comprises a plurality of model patches, each model patch is composed of model vertexes, the attribute information and the first normal information of each model patch are obtained, the first position information of each model vertex is obtained, the first normal information is optimized based on the attribute information, the second normal information is obtained, the first position information is optimized based on the first normal information and the second normal information, the second position information of each model vertex is obtained, the model to be optimized is optimized, on one hand, the model is directly optimized through the normal information before and after optimization without indirect representation forms such as a truncation symbol distance field and the like, so that the algorithm overhead can be greatly reduced, the optimization efficiency is improved, on the other hand, the connection relation between each model patch and each model vertex is considered in the optimization process, therefore, the optimization quality is improved, and the quality and the efficiency of model optimization can be improved.

Drawings

FIG. 1 is a schematic flow chart diagram of an embodiment of a model optimization method of the present application;

FIG. 2 is a schematic diagram of an embodiment of a model to be optimized;

FIG. 3 is a diagram of an embodiment of a model patch and model vertices;

FIG. 4 is a schematic flow chart diagram of another embodiment of a model optimization method of the present application;

FIG. 5a is a schematic diagram of an embodiment of a patch to be subdivided and its neighbors;

FIG. 5b is a schematic diagram of another embodiment of a patch to be subdivided and its neighbors;

FIG. 5c is a schematic diagram of yet another embodiment of a patch to be subdivided and its neighbors;

FIG. 6 is a schematic diagram of an embodiment of a model to be optimized and the model after optimization;

FIG. 7 is a block diagram of an embodiment of a model optimization apparatus according to the present application;

FIG. 8 is a block diagram of an embodiment of an electronic device of the present application;

FIG. 9 is a block diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two.

Referring to fig. 1, fig. 1 is a schematic flow chart illustrating a model optimization method according to an embodiment of the present application. Specifically, the method may include the steps of:

step S11: and obtaining a model to be optimized.

In the embodiment of the disclosure, the model to be optimized includes a plurality of model patches, and the model patches are formed by model vertices. Referring to fig. 2, fig. 2 is a schematic diagram of an embodiment of a model to be optimized. As shown in fig. 2, the model to be optimized includes a plurality of model patches, where the model patches may specifically include, but are not limited to, triangles, and each model patch is formed by model vertices, for example, when the model patch is a triangle, the model patch is formed by connecting three model vertices. It should be noted that fig. 2 only shows an exemplary possible model in practical application, and the model to be optimized may be obtained by performing model reconstruction on a single object such as a table, a chair, or the like, or may be obtained by performing model reconstruction on a scene including a plurality of objects such as a sand table, a living room, or the like, which is not limited herein. To facilitate the calculation in the model optimization process, the model to be optimized may be located in a world coordinate system. The following expressions such as position information and position coordinates may specifically refer to position information and position coordinates in a world coordinate system, unless otherwise specified.

In one implementation scenario, for a device integrated with an RGB camera and a depth camera, a plurality of key frame data of a target object may be acquired by the device, where the key frame data includes an RGB image and a depth image, and the camera pose is solved by using one or more combinations of an Iterative Closest Point (ICP), an optical flow method, and an Inertial sensor (IMU). On the basis, the depth image of the target object and the reference voxel model can be fused by combining with the camera pose to obtain a reconstructed voxel model of the target object, voxels in the reconstructed voxel model contain voxel information, the voxel information can include a truncation symbol distance, the reconstructed voxel model is used as a new reference voxel model under the condition that a new depth image is obtained, and the step of fusing the depth image of the target object and the reference voxel model to obtain the reconstructed voxel model of the target object is executed again until the scanning is finished. On the basis, the reconstructed voxel model can be processed by adopting modes such as Marching Cubes and the like to obtain a three-dimensional grid serving as a model to be optimized.

In one implementation scenario, for an apparatus integrating only RGB cameras, a plurality of key frame data of a target object may be acquired by the apparatus, where the key frame data includes RGB images, and a Motion recovery Structure (SFM) is used to solve to obtain camera internal parameters and camera pose, then image depth information is recovered by a Multi-view step (MVS), and finally a three-dimensional grid of the target object is generated by a dirough (Delaunay) algorithm as a model to be optimized.

In an implementation scenario, after the model to be optimized is obtained through any one of the above manners, texture mapping may be performed on the model to be optimized, so that a template in the model to be optimized has image information (e.g., color, brightness, etc.), and then the model after texture mapping is used as the model to be optimized for subsequent optimization. For a specific process, reference may be made to the technical details of texture mapping, which are not described herein again.

In an implementation scenario, the embodiments of the present disclosure and the following embodiments of the present disclosure may also be applied to a computer artificial three-dimensional modeling scenario, specifically, a model obtained by the computer artificial three-dimensional modeling may be used as a model to be optimized, and the number of patches of the model may be sparse (i.e., low modulus).

In one implementation scenario, for ease of description, the model to be optimized may be written asWherein T ═ { T ═ T₀，t₁… represents a model patch, V ═ V₀，v₁… represents the model vertices. It should be noted that, as mentioned above, the model to be optimized may be reconstructed on the basis of several frames of images, which may be referred to as "images" for convenience of descriptionWherein each frame of imageSpecifically, the depth image and the visible light image can be a frame_j(e.g., RGB image), as described above, the depth image may be specifically used to blend depth information in the process of performing model reconstruction by using a device integrating an RGB camera and a depth camera, and reference may be specifically made to the foregoing related description, which is not described herein again. In addition, it should be noted that each frame of image has two parameters, namely a camera pose P_jAnd camera internal parameter K_jSo that each frame image F_jCan be described as F_j＝{G_j，P_j，K_j}。

Step S12: and acquiring attribute information and first normal information of each model patch, and acquiring first position information of each model vertex.

In one implementation scenario, the first normal information of model patch t may be represented using a normal vector, then for model patch T e T, any three points can be taken on model patch T, which can be denoted as p1, p2 and p3 for convenience of description, then from the position coordinates in p1 and p2, a first vector connecting p1 and p2 can be obtained, obtaining a second vector connecting p1 and p3 according to the position coordinates in p1 and p3, and obtaining a third vector connecting p2 and p3 according to the position coordinates in p2 and p3, since the normal vector of the model patch t is orthogonal to the first vector, the second vector and the third vector (i.e. the multiplication result of the normal vector and any one of the first vector, the second vector and the third vector is 0), then, according to the relationship, an equation set about the normal vector of the model patch t can be established, and the normal vector of the model patch t can be obtained by solving the equation set. In addition, the normal vector of the model patch t, i.e. the normal vector of the model patch, can also be solved quickly based on the vector outer productWherein the content of the first and second substances,representing the first vector connecting p1 and p2,representing a third vector connecting p2 and p3, | | | | represents a vector modulo operation.

In one implementation scenario, the attribute information may include, but is not limited to: for convenience of description, for the model patch T ∈ T, the luminance information may represent i (T), the hue information may represent c (T), the luminosity information may represent b (T), and the diffuse reflection information may represent a (T).

In a specific implementation scenario, for the ith model patch t in the model to be optimized_iE.g. T, the camera pose P can be used_jAnd camera internal parameter K_jRe-casting the model to be optimizedShadow to jth frame image F_jIn the image coordinate system, after the occlusion judgment is carried out based on the depth buffer algorithm, the model surface patch t can be obtained_iIn the visible light image G_jCalculating the model patch t by the pixel color mean value of the upper cover_iCorresponding to the visible light image G_jLuminance information of_j(t_i) Hue and hue information C_j(t_i). The luminance information and hue information of other model patches can be analogized, and there is no longer a distance.

In a specific implementation scenario, when a model to be optimized passes through a texture map, the luminance information and hue information of the model patch can be obtained directly from texture coordinates and a texture image of the model patch.

In a specific implementation scenario, when it is detected that a model patch does not correspond to any image region, the model patch may be skipped in a subsequent processing procedure. The following luminance information I_j(t_i) Hue and hue information C_j(t_i) All represent model patch t_iHas effective brightness information and hue information.

In a specific implementation scenario, it may be assumed that the target object has a rough surface and the illumination of the environment where the target object is located is mainly global light (i.e. natural light), and based on this, the ambient illumination coefficient I may be estimated. Specifically, a third-order spherical harmonic illumination model can be used to describe ambient light, and the first normal information n (T) and the diffuse reflection information a (T) of the model patch T e T are used to obtain luminosity information b (T):

in the above formula (1), I ═ l₀，l₁，…，l₈Denotes the ambient illumination factor, H_m(n) represents a spherical harmonic illumination basis function, which can be specifically expressed as:

H₀(n)＝1.0......(2)

H₁(n)＝n_y......(3)

H₂(n)＝n_z......(4)

H₃(n)＝n_x......(5)

H₄(n)＝n_xn_y......(6)

H₅(n)＝n_yn_z......(7)

H₆(n)＝-n_xn_x-n_yn_y+2n_zn_z......(8)

H₇(n)＝n_zn_x......(9)

H₈(n)＝n_xn_x-n_yn_y......(10)

in the above formulas (2) to (10), the first normal information n (t) can be specifically represented by a normal vector, and n_x，n_y，n_zThe first normal information n (t) represents a component in the x-axis direction, a component in the y-axis direction, and a component in the z-axis direction, respectively. On the basis, the luminosity information B (t) and the model patch t can correspond to each visible light image G_jLuminance information of_j(t), constructing and obtaining an optimization function related to the ambient illumination coefficient I, and solving the optimization function in a minimized mode to obtain the ambient illumination coefficient I:

and (3) carrying out minimum solving through the formula (11) to obtain an ambient illumination coefficient I, so that the deduced luminosity information can be close to the brightness information of the model patch as much as possible. In addition, in the process of first performing the solution on the ambient light illumination coefficient, the diffuse reflection information a (t) may be initialized, for example, to 1, 0.5, and the like, which is not limited herein. It should be noted that, in addition to describing the ambient light by using the third-order spherical harmonic illumination model, other forms of illumination models may also be used, for example, a higher-order spherical harmonic illumination model may be used, and is not limited herein. After obtaining the ambient illumination coefficient I, the formula (1) can be substituted to obtain the luminosity information b (t) of the model patch t.

Step S13: and optimizing the first normal information based on the attribute information to obtain second normal information.

Specifically, a cost function related to the second normal information can be constructed according to the attribute information and the first normal information, and the cost function is subjected to iterative optimization to obtain the second normal information of the model patch. The cost function may specifically include a first cost sub-term for representing gradient residuals of the photometric and luma information, a second cost sub-term for representing a deviation between the first and second normal information, a third cost sub-term for smoothing the second normal information, and a fourth cost sub-term for avoiding texture copy problems as much as possible.

In one implementation scenario, for a model patch T ∈ T, gradient values of luminosity information b (T) thereof may be obtained, and the model patch T is obtained corresponding to luminance information I in each frame of image_j(t) and obtaining the residual value between the two values, and on the basis, summing the residual values corresponding to each frame image based on the model patch t to obtain a first valence sub-item E_g(t) of (d). Specifically, it can be expressed as:

in the above-mentioned formula (12),a gradient operator is represented. By constraining the sub-terms, the gradient of the photometric information can be made as close as possible to the gradient of the luminance information. Note that, for the model patch T e T, the luminance information I corresponding to each frame image of the model patch T may be directly obtained_jThe difference value between (t) and luminosity information B (t), on the basis, the sum is carried out corresponding to each frame image based on the model patch t, and a first valence sub item E is obtained_g(t), in which case the above equation (12) may be adaptively modifiedObtaining a first value itemIn practical application, the first valence item E can be obtained according to any one of the two ways_g(t), which is not limited herein. It should be noted that the former calculates the first valence term E by gradient compared with the latter_g(t) has better stability.

In an implementation scenario, for the model patch T E T, the deviation between the first normal information and the second normal information can be obtained to obtain the second cost sub-item E_s(t) of (d). Specifically, it can be expressed as:

E_s(t)＝||n(t)-n₀(t)||²......(13)

in the above formula (13), n (t) represents second normal information, n₀(t) represents the first normal information, | | |. | | represents the vector modulo function. By constraining the sub-item, the second normal information can be made to approach the first normal information without deviating too much from the first normal information.

In an implementation scenario, for a model patch T e T, its neighboring patch T' en (T) may be obtained, it should be noted that the neighboring patch of the model patch T and the model patch T have a shared edge, that is, at least one edge is shared between the two. On the basis, the deviation between the second normal information of the model patch t and the second normal information of the adjacent patch t' can be obtained to obtain a third valence sub item E_r(t) of (d). Specifically, it can be expressed as:

E_r(t)＝∑_t′∈N(t)||n(t)-n(t′)||²......(14)

in the above equation (14), n (t) represents the second normal information of the model patch t, n (t') represents the second normal information of the neighboring patch of the model patch t, and | · | represents a vector modulo function. By constraining the sub-item, the second normal information of the model patch t can be close to the second normal information of the adjacent patch t 'without being too far away from the second normal information of the adjacent patch t', so that the adjacent patches can have certain smoothness.

In one implementation scenario, for a model patch te ∈ T, hue information C of the model patch in each frame of image can be obtained_j(t) hue information C of adjacent patches in each frame image_j(t '), obtaining hue weight according to the difference between the hue weight and the model patch t, obtaining the difference between diffuse reflection information A (t) of the model patch t and diffuse reflection information A (t') of adjacent patches, and constructing a fourth cost sub item E according to the difference and the hue weight_a(t) of (d). Specifically, it can be expressed as:

in the above formula (15), phi (c) ═ 1/(1+3| | c |)³Wherein C represents hue information C_j(t) and hue information C_j(t') and phi (c) represents a hue weight calculated from the difference. The hue information C of the model patch t in each frame image_j(t) hue information C of adjacent patch t' in each frame image_j(t') the larger the difference is, the smaller the hue weight is, so that the color of the model patch itself and the texture caused by illumination can be distinguished by restricting the subitem, thereby being beneficial to inhibiting the texture copying problem and improving the optimization quality of the model.

In an implementation scenario, after the first cost sub-item, the second cost sub-item, the third cost sub-item, and the fourth cost sub-item of each model patch are obtained through construction, the cost function can be obtained through construction by combining the first cost sub-item, the second cost sub-item, the third cost sub-item, and the fourth cost sub-item of all the model patches and the respective importance degrees of the first cost sub-item, the second cost sub-item, the third cost sub-item, and the fourth cost sub-item. Illustratively, the degree of importance λ of the first valence sub-item_gCan be set to 1, the degree of importance of the second-valence sub-item λ_sCan be set to 0.005, the degree of importance of the third valence item λ_rCan be set to 0.003, the degree of importance of the fourth cost sub term λ_aCan be set as 2And is not limited herein. So the cost function E can be expressed as:

E＝∑_t∈T(λ_gE_g(t)+λ_sE_s(t)+λ_rE_r(t)+λ_aE_a(t))......(16)

in one implementation scenario, a solving algorithm such as L-BFGS, DFP, etc. may be used to solve the cost function to optimize the second normal information and the diffuse reflection information of each model patch. Specifically, in the solving process, the solving algorithm capable of iteratively executing the preset times is utilized, and the preset times can be set according to the actual application needs. For example, in the case of a high requirement on the solution efficiency, the preset number of times may be set to be appropriately smaller, such as 10, 15, and the like, or in the case of a relatively loose requirement on the solution efficiency, the preset number of times may be set to be appropriately larger, such as 30, 35, and the like, and is not limited herein. It should be noted that the solution algorithm may specifically adopt a quasi-newton algorithm such as the aforementioned L-BFGS, DFP, etc., and compared with other conventional solution algorithms, the solution algorithm can obtain a better result in a shorter time, so that a better result can be obtained in a shorter time, and the optimization efficiency is greatly improved while the quality is basically guaranteed. In addition, under the condition that the requirement on the solving efficiency is relatively loose, a conventional solving algorithm such as a gauss-newton method can be adopted, and the method is not limited herein.

Step S14: and optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

Specifically, the first location information may include a first coordinate, the second location information may include a second coordinate, each model vertex is respectively used as a first current vertex, and a model vertex having a connection edge with the first current vertex is selected as a reference vertex, because an end point of each edge in the model to be optimized is used as the model vertex, an edge using the first current vertex as the end point may be selected as the connection edge in the model to be optimized, and another end point of the selected edge (i.e., the connection edge) is used as the reference vertex, which may be referred to the following description, which is not repeated herein. On the basis, two model patches sharing the connecting edge can be respectively used as a first patch and a second patch, a first normal transformation parameter of the first patch is obtained based on the first normal information and the second normal information of the first patch, and based on the first normal information and the second normal information of the second patch, obtaining a second normal transformation parameter of the second patch, on the basis, an objective function of the first current vertex is constructed based on the first normal transformation parameter and the second normal transformation parameter, and the objective function comprises a first reference subentry and a second reference subentry, the first reference subentry is the product of the preset reference coefficient and the first coordinate of the first current vertex, the second reference subentry is the product of the preset reference coefficient and the second coordinate of the first current vertex, and the second coordinate of each model vertex is obtained by solving based on the objective function of each model vertex. In the mode, in the position coordinate optimization process of the model vertex, the product of the preset reference coefficient and the first coordinate of the model vertex and the product of the preset reference coefficient and the second coordinate of the model vertex are considered at the same time and are respectively used as the sub-items of the objective function, so that the fault tolerance rate of the objective function solving is improved, and the stability of the solving result is improved.

In one implementation scenario, referring to fig. 3 in combination, fig. 3 is a schematic diagram of an embodiment of a model patch and a model vertex. It should be noted that, for the sake of visual presentation, fig. 3 only exemplarily shows a partial region of the model to be optimized. As shown in fig. 3, the model vertex a is the first current vertex, and the edge with the model vertex a as the end point includes: AB. AC, AD, AE, the other end point of these edges may be used as a reference vertex, i.e. the model vertex having a connecting edge with the first current vertex may include: model vertex B, model vertex C, model vertex D, and model vertex E, model vertex E illustratively being selected as a reference vertex, so that two model patches sharing a connecting edge AE, namely model patch AED and model patch AEB, may be respectively considered as a first patch and a second patch. In the case of selecting model vertex B, model vertex C, and model vertex D, the analogy may be repeated, and no one example is given here. That is, for the first current vertex, if there are N model vertices having connecting edges with it, N pairs of the first patch and the second patch can be obtained.

In an implementation scenario, after the first normal information of the first patch is multiplied by the first normal transformation parameter, the second normal information of the first patch can be solved, and accordingly the first normal transformation parameter can be solved; similarly, the second normal information of the second patch can be obtained by multiplying the first normal information of the second patch by the second normal transformation parameter, and the second normal transformation parameter can be solved accordingly.

In one implementation scenario, a first diagonal of a connecting edge in a first patch may also be obtained, and a second diagonal of a connecting edge in a second patch may also be obtained. It should be noted that the connecting edge specifically refers to a connecting edge shared by the first patch and the second patch, that is, a connecting edge formed by connecting the first current vertex and the reference vertex. On the basis, a first optimization sub-item can be constructed based on a first normal transformation parameter, a first opposite angle, a second normal transformation parameter, a second opposite angle, a first coordinate of a first current vertex and a first coordinate of a reference vertex, a second optimization sub-item is constructed based on the first opposite angle of the first patch, the second opposite angle of the second patch, the first coordinate of the first current vertex and the second coordinate of the reference vertex, and an objective function of the first current vertex is obtained based on the first optimization sub-item, the first reference sub-item, the second optimization sub-item and the second reference sub-item. By the method, the incidence relation between the adjacent patches can be fully considered in the optimization process, and the optimization quality is improved.

In a specific implementation scenario, please continue to refer to fig. 3 in combination, after the model patch AED and the model patch AEB are respectively taken as the first patch and the second patch, the opposite angle (i.e., < ADE) of the connection edge AE in the first patch AED can be taken as the first opposite angle, and the opposite angle (i.e., < ABE) of the connection edge AE in the second patch AEB can be taken as the second opposite angle. Other cases may be analogized, and no one example is given here.

In a specific implementation scenario, the product of the cotangent value of the first diagonal and the first normal transformation parameter, and the product of the cotangent value of the second diagonal and the second normal transformation parameter may be summed to obtain a first sum, and a first difference between the first coordinate of the first current vertex and the first coordinate of the reference vertex is obtained, and the product of the first sum and the first difference is obtained to obtain the first optimization sub-term.

In a specific implementation scenario, the cotangent value of the first diagonal and the cotangent value of the second diagonal may be summed to obtain a second sum, a second difference between the second coordinate of the first current vertex and the second coordinate of the reference vertex is obtained, and a product of the second sum and the second difference is obtained to obtain a second optimized sub-item.

In one particular implementation scenario, for ease of description, for any model vertex v_iE.g. V, it can be regarded as the first current vertex, and its first coordinate can be denoted as V_iIts second coordinate may be denoted as v'_i. The reference vertices of the model vertices can be denoted as v_jIts first coordinate can be denoted as v_jIts second coordinate may be denoted as v'_jAnd will share the connecting edge v_iv_jTwo model patches of (2), i.e. model patch v_iv_jv_k、v_iv_jv_lRespectively as a first patch and a second patch, the first patch having a connecting edge v_iv_jCan be said asConnecting edge v in second patch_iv_jCan be noted asFurthermore, the first normal transformation parameter of the first patch can be written asThe second normal transformation parameter of the second patch can be written asTherefore, canTo obtain a first current vertex v_iThe objective function of (2):

in the above equation (17), μ represents a preset reference coefficient, and the specific value may be set according to practical applications, for example, may be set to 0.01. μ v_iA first sub-item of reference is represented,denotes the first optimized sub term,. mu.v'_iA second sub-item of reference is indicated,representing a second optimization sub-item.

In a specific implementation scenario, as described above, if there are N model vertices with connected edges for the first current vertex, N pairs of the first patch and the second patch may be obtained, and accordingly, for the first current vertex, N objective functions may be constructed, and for simplifying the representation, the N objective functions may be expressed as:

in the above formula (18), N (v)_i) Representation and first current vertex v_iThere is a set of connected edge model vertices.

In one implementation scenario, in order to reduce the complexity of constructing the objective function, unlike the foregoing manner, a first diagonal of a connecting edge in the first patch and a second diagonal of a connecting edge in the second patch may not be obtained. On the basis, a first normal transformation parameter, a second normal transformation parameter, a first coordinate of a first current vertex and a first coordinate of a reference vertex can be directly needed to construct a first optimized sub-item, a second optimized sub-item is constructed on the basis of the first coordinate of the first current vertex and the second coordinate of the reference vertex, and then the first optimized sub-item and the second optimized sub-item are constructed on the basis of the first optimized sub-item and the second coordinate of the reference vertexAnd obtaining the objective function of the first current vertex based on the first optimization sub-item, the first reference sub-item, the second optimization sub-item and the second reference sub-item. In this case, the first current vertex v_iThe objective function of (2) can be simplified as:

similarly, for the first current vertex, N objective functions may be constructed, which may also be simplified as:

it should be noted that, the implementation scenario only describes the differences from the foregoing scheme, and the same points can refer to the foregoing related descriptions, which are not described herein again.

In an implementation scenario, after obtaining the objective function of each model vertex, an equation set of the second coordinate of each model vertex may be established in a simultaneous manner based on the objective function of each model vertex, and the second coordinate of each model vertex may be obtained by solving the equation set. By the method, the second coordinates of the model vertexes can be obtained by one-time solving through combining the equation sets of the second coordinates of the model vertexes, and the model optimization efficiency is further improved.

It should be noted that after the second coordinates of each model vertex are obtained, it can be considered that the position in each model vertex in the model to be optimized has been updated, so the surface morphology of the model to be optimized also changes, and since the position information of each model vertex is updated synchronously in the model optimization process, on one hand, the optimization efficiency can be improved, and on the other hand, the relevance between model vertices can be considered, especially the relevance between adjacent vertices with connecting edges exists, which is beneficial to ensuring the quality of the patch after model optimization, greatly reducing the occurrence probability of low-quality patches such as long and narrow, degeneration, reversal and the like, ensuring the stability of the process, and improving the usability of the result.

In addition, the embodiments of the present disclosure and the following embodiments of the present disclosure may be applied to the fields of Augmented Reality (AR), Virtual Reality (VR), and the like, specifically, application scenarios such as games, movies, electronic commerce, education, real estate, home decoration, and the like, which are not limited herein.

According to the scheme, the model to be optimized is obtained and comprises a plurality of model patches, each model patch is composed of model vertexes, the attribute information and the first normal information of each model patch are obtained, the first position information of each model vertex is obtained, the first normal information is optimized based on the attribute information, the second normal information is obtained, the first position information is optimized based on the first normal information and the second normal information, the second position information of each model vertex is obtained, the model to be optimized is optimized, on one hand, the model is directly optimized through the normal information before and after optimization without indirect representation forms such as a truncation symbol distance field and the like, so that the algorithm overhead can be greatly reduced, the optimization efficiency is improved, on the other hand, the connection relation between each model patch and each model vertex is considered in the optimization process, therefore, the optimization quality is improved, and the quality and the efficiency of model optimization can be improved.

Referring to fig. 4, fig. 4 is a schematic flow chart of another embodiment of the model optimization method of the present application. In the embodiment of the present disclosure, in order to further promote the details of the model, a preset number of optimization stages may be performed on the model to be optimized in the model optimization process. Specifically, the method may include the steps of:

step S41: and obtaining a model to be optimized.

In the embodiment of the present disclosure, the model to be optimized includes a plurality of model patches, and the model patches are formed by model vertices, which may specifically refer to the related description in the foregoing embodiment, and are not described herein again.

Step S42: and acquiring attribute information and first normal information of each model patch, and acquiring first position information of each model vertex.

Reference may be made to the related description in the foregoing embodiments, which are not repeated herein.

Step S43: and optimizing the first normal information based on the attribute information to obtain second normal information.

Reference may be made to the related description in the foregoing embodiments, which are not repeated herein. It should be noted that, as described in the foregoing disclosure, in the process of optimizing the first normal information and obtaining the second normal information, the diffuse reflection information of each model patch can also be obtained through optimization.

Step S44: and detecting whether the current optimization stage meets preset conditions, if so, executing the step S45, otherwise, executing the step S49.

Specifically, the preset condition may be set to include: the current optimization stage does not reach the preset number of optimization stages. To facilitate counting, a count parameter n may be initialized at the first detection, which is used to indicate the current optimization stage, e.g. the count parameter may be set to 0 at the first detection. In addition, the preset value may be set according to actual conditions, for example, in a case where the requirement on the optimization efficiency of the model is high, the preset value may be set to be smaller, for example, 2, 3, and the like, or in a case where the requirement on the optimization efficiency of the model is relatively loose, the preset value may be set to be larger, for example, 4, 5, and the like, and is not limited herein.

Step S45: and obtaining the optimization loss of the model surface patch based on the attribute information, the first normal information and the second normal information of the model surface patch.

Under the condition that the current optimization stage meets the preset conditions, the model to be optimized can be refined. Specifically, the optimization loss of the model patch may be obtained based on the attribute information, the first normal information, and the second normal information of the model patch. As described in the foregoing disclosure, the attribute information may include, but is not limited to: the brightness information, hue information, luminosity information and diffuse reflection information of the model patch. For convenience of description, for the model patch T e T, the luminance information may be represented as i (T), the hue information may be represented as c (T), the luminosity information may be represented as b (T), and the diffuse reflection information may be represented as a (T). The optimization penalty of the model patch may include a first cost item for representing a gradient residual of the photometric information and the luma information, a second cost item for representing a deviation between the first normal information and the second normal information, a third cost item for smoothing the second normal information, and a fourth cost item for avoiding a texture copy problem as much as possible.

In an implementation scenario, the calculation manner of each sub-item may refer to the relevant description in the foregoing disclosed embodiment, and is not described herein again.

In an implementation scenario, as described in the foregoing disclosure, after the first cost sub-item, the second cost sub-item, the third cost sub-item, and the fourth cost sub-item of each model patch are constructed, the optimization loss can be constructed by using respective importance degrees of the first cost sub-item, the second cost sub-item, the third cost sub-item, and the fourth cost sub-item. Specifically, the first price sub item, the second price sub item, the third price sub item and the fourth price sub item can be weighted respectively by using the respective importance degrees of the first price sub item, the second price sub item, the third price sub item and the fourth price sub item, so as to obtain the optimized loss. The respective importance degrees of the sub-items can be referred to the related descriptions in the foregoing disclosed embodiments, and are not described herein again.

It should be noted that, different from the sub-item processes in the foregoing disclosed embodiment, when the optimization loss is calculated, the diffuse reflection information used in the process of updating the first normal information last time is the diffuse reflection information obtained by synchronous optimization, that is, the diffuse reflection information obtained by synchronous updating when step S43 is executed last time.

Step S46: and selecting at least one model patch as a patch to be subdivided in the current optimization stage based on the optimization loss.

In an implementation scenario, the model patches may be sorted in an order from a large optimization loss to a small optimization loss, and the model patch located at a front preset order is selected as the patch to be subdivided. The preset ordinal number may be set according to actual application needs, for example, the preset ordinal number may be set to 15%, 20%, 25%, and the like according to a percentage, that is, a model patch located at the top 15%, the top 20%, or the top 25% may be selected as a patch to be subdivided, or the preset ordinal number may also be set to 1000, 5000, 20000, and the like according to a numerical value, that is, a model patch located at the top 1000, the top 5000, or the top 20000 may be selected as a patch to be subdivided, which is not limited herein. By the method, the model to be optimized can be refined based on the model patch with large optimization loss, so that the detail optimization of the part with poor details can be facilitated, and the optimization quality of the model can be improved.

In one implementation scenario, the model patch with the optimization loss greater than the first threshold may also be directly selected as the patch to be subdivided. The first threshold may be set according to the actual application requirement, for example, in the case of a high requirement on the optimization effect of the model, the first threshold may be set to be appropriately lower, or in the case of a relatively loose requirement on the optimization effect of the model, the first threshold may be set to be appropriately higher, which is not limited herein. By the method, the model to be optimized can be refined based on the model patch with large optimization loss, so that the detail optimization of the part with poor details can be facilitated, and the optimization quality of the model can be improved. It should be noted that, if a model patch with an optimization loss greater than the first threshold is not selected, that is, the optimization losses of all model patches are not greater than the first threshold, the current detail effect of the model to be optimized may be considered to be better, and the model subdivision operation may be skipped, and the subsequent optimization of the first position information based on the first normal information and the second normal information may be directly performed to obtain the second position information of the model vertex, so as to optimize the model to be optimized.

In one implementation scenario, to further improve the model optimization quality, a patch to be subdivided may also be selected based on the optimization loss of the model patch and the patch area. Specifically, after at least one model patch is selected as a patch to be subdivided in the current optimization stage based on the optimization loss, further screening patches to be subdivided, the areas of which are not lower than a second threshold, so as to subdivide the model to be optimized based on the patches to be subdivided obtained by screening subsequently, thereby obtaining the subdivided model to be optimized. It should be noted that, if the area to be subdivided of the patch area not less than the second threshold is selected, that is, the patch areas of all the patches to be subdivided are less than the second threshold, it may be considered that the patches to be subdivided do not need to be subdivided, and the model subdivision operation may be skipped, and the subsequent optimization of the first position information based on the first normal information and the second normal information is directly performed, so as to obtain the second position information of the model vertex, so as to optimize the model to be optimized.

In an implementation scenario, after a patch to be subdivided is selected and obtained based on optimization loss, or a patch to be subdivided is selected and obtained by combining optimization loss and a patch area, whether the total number of the patch to be subdivided which is selected and obtained is more than a third threshold value or not can be further detected, if so, a subsequent step of subdividing a model to be optimized based on the patch to be subdivided can be executed, otherwise, the model subdividing operation can be skipped, and the subsequent step of optimizing first position information based on first normal information and second normal information is directly executed to obtain second position information of a model vertex, so that the model to be optimized is optimized. In addition, the specific value of the third threshold may not be limited, and may be set to 0, 100, 500, 2000, or the like, for example.

Step S47: and subdividing the model to be optimized based on the patch to be subdivided to obtain the subdivided model to be optimized so as to update the model patch of the model to be optimized.

In an implementation scenario, at least one model vertex may be newly added on an edge of a patch to be subdivided, each newly added model vertex is respectively used as a second current vertex, a candidate vertex set of the second current vertex is obtained, the candidate vertex set includes a plurality of candidate vertices, the candidate vertices are located on a plurality of candidate patches, the candidate patches all include the edge where the second current vertex is located, at least one candidate vertex is selected from the candidate vertex set, and the candidate vertices are respectively connected with the at least one candidate vertex and the second current vertex, so that a subdivided model to be optimized is obtained. By the aid of the method, the model patches can be subdivided at positions where the details of the model to be optimized are not good enough, finer optimization can be achieved at the positions, and optimization quality of the model is improved.

In a specific implementation scenario, in order to finely subdivide the model to be optimized in an optimization stage, at least one model vertex may be added to each edge of the patch to be subdivided. In addition, in the case where only one model vertex is added to an edge, the added model vertex may be located at the midpoint of the edge. Referring to fig. 5a, fig. 5a is a schematic diagram of an embodiment of a patch to be subdivided and its neighboring patches. It should be noted that, a patch to be subdivided shares a connecting edge with its neighboring patch, as shown in fig. 5a, a patch to be subdivided ABC shares a connecting edge AB with its neighboring patch ABF, shares a connecting edge AC with its neighboring patch ACE, and shares a connecting edge BC with its neighboring patch BCD. In this case, a new model vertex may be added to each edge of the patch ABC to be subdivided, that is, a new model vertex G may be added to the edge AB of the patch ABC to be subdivided, a new model vertex H may be added to the edge AC of the patch ABC to be subdivided, and a new model vertex I may be added to the edge BC of the patch ABC to be subdivided. Other cases may be analogized, and no one example is given here.

In one specific implementation scenario, continuing with fig. 5a, taking model vertex G as the second current vertex, since the second current vertex G is located at edge AB, model patch ABF and model patch ABC including edge AB may be used as candidate patches, and model vertices located on the two candidate patches (i.e., model vertex a, model vertex B, model vertex C, model vertex F, model vertex G, model vertex H, and model vertex I) may be used as candidate vertices. Other cases may be analogized, and no one example is given here. Furthermore, further, in the model to be optimized, the candidate vertex may not have a connecting edge with the second current vertex, and the candidate vertex may not be the second current vertex itself, in this case, taking the model vertex G as the second current vertex as an example, the model vertex C, the model vertex H, the model vertex I, and the model vertex F may be taken as candidate vertices. Other cases may be analogized, and no one example is given here.

In a specific implementation scenario, in order to subdivide the model to be optimized as much as possible during the optimization stage, as many candidate vertices as possible may be selected from the candidate vertex set, and the second current vertex and the candidate vertex are connected to subdivide the model to be optimized. Please continue to refer to fig. 5a, taking the model vertex G as the second current vertex as an example, the candidate vertex H, the candidate vertex I, and the candidate vertex F may be selected, and the second current vertex G and the candidate vertex H are connected (as shown by a dotted line in the figure) to divide the patch ABC to be subdivided into two, and the second current vertex G and the candidate vertex I are connected (as shown by a dotted line in the figure) to divide the patch ABC to be subdivided into three, and the second current vertex G and the candidate vertex F are connected (as shown by a dotted line in the figure) to divide the patch ABF into two. Other cases may be analogized, and no one example is given here. Therefore, by subdividing the model to be optimized, the number of model patches and model vertices is increased, and the original part with poor details can be further optimized.

In an implementation scenario, a new model vertex may be added in the patch to be subdivided, and the new model vertex is used as a second current vertex and is respectively connected with the vertex of the patch to be subdivided and the second current vertex, so as to obtain the subdivided model to be optimized. It should be noted that, a point may be randomly selected from the patch to be subdivided as a newly added model vertex, or the center of gravity of the patch to be subdivided may be selected as a newly added model vertex, which is not limited herein. Referring to fig. 5b, fig. 5b is a schematic diagram of another embodiment of a patch to be subdivided and its neighboring patches. As shown in fig. 5B, the centroid G of the patch ABC to be subdivided may be selected as a new model vertex and used as a second current vertex, and the vertex a, the vertex B, the vertex C, and the second current vertex G of the patch ABC to be subdivided are connected, so as to subdivide the patch ABC to be subdivided into a patch AGC, a patch BGC, and a patch AGB. Other cases may be analogized, and no one example is given here.

In an implementation scenario, a new model vertex may be added in a patch to be subdivided, the model vertex is connected to the new model vertex from the vertex of the patch to be subdivided and continues to extend to the opposite side of the vertex to intersect to obtain a new model vertex, on this basis, all new model vertices may be respectively used as second current vertices, and candidate vertex sets of the second current vertices are obtained, and the candidate vertex sets include a plurality of candidate vertices located in a plurality of candidate patches, and the plurality of candidate patches include the edge where the second current vertex is located, and at least one candidate vertex is selected from the candidate vertex sets and is respectively connected with the at least one candidate vertex and the second current vertex, so as to obtain a subdivided model to be optimized. Referring to fig. 5c, fig. 5c is a schematic diagram of another embodiment of a patch to be subdivided and its neighboring patches. As shown in fig. 5C, the gravity center J of the patch ABC to be subdivided may be selected as a new model vertex, and the new model vertex J may be connected to the vertex a, the vertex B, and the vertex C of the patch ABC to be subdivided, and may continue to extend to the opposite side BC of the vertex a, the opposite side AC of the vertex B, and the opposite side AB of the vertex C, respectively, so as to obtain a new model vertex I, a new model vertex H, and a new model vertex G through respective intersection. On this basis, all the new model vertices G, the new model vertices H, and the new model vertices I may be respectively used as second current vertices, and candidate vertex sets of the second current vertices may be obtained. Therefore, by the subdivision mode, the face ABC to be subdivided into the face AGJ, the face AHJ, the face CHJ, the face CIJ, the face BIJ and the face BGJ, the adjacent face ABF of the face ABC to be subdivided into the face AGF and the face BGF, the adjacent face BCD of the face ABC to be subdivided into the face BID and the face CID, and the adjacent face ACE of the face ABC to be subdivided into the face AHE and the face CHE. Other cases may be analogized, and no one example is given here.

It should be noted that the above-mentioned subdivision schemes are only possible subdivision schemes in practical application, and the embodiments of the present disclosure do not limit the subdivision schemes adopted in practical subdivision, and are not limited herein.

In an implementation scenario, after all the newly added model vertices are connected to their candidate vertices, the optimization stage of the current round may be considered to be completed, and then the parameter n +1 may be counted.

Step S48: step S42 and subsequent steps are re-executed.

It should be noted that, different from the first time of executing step S42, when step S42 is executed for the nth (N >1), the diffuse reflection information in the attribute information of the model patch is the diffuse reflection information obtained by executing step S42 and subsequent steps for the last time (i.e., the nth-1), that is, the latest diffuse reflection information.

Step S49: and optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex so as to optimize the model to be optimized.

Reference may be made to the related description in the foregoing embodiments, which are not repeated herein.

Referring to FIG. 6, FIG. 6 is a diagram of an embodiment of a model to be optimized and a model after optimization. It should be noted that, in order to show the comparison effect before and after optimization, the model shown in fig. 6 is a rendered model, and as shown in fig. 6, the face details of the model corresponding to the "grand toilet sculpture" are richer after optimization; and after the model corresponding to the lion statue is optimized, the bristle position details are richer.

According to the scheme, before optimization, when the current optimization stage meets preset conditions, optimization loss of the model surface patch is obtained based on attribute information, first normal information and second normal information of the model surface patch, so that based on the optimization loss, at least one model surface patch is selected as a surface patch to be subdivided in the current optimization stage, the model to be optimized is subdivided based on the surface patch to be subdivided, and the step of obtaining the attribute information and the first normal information of each model surface patch and subsequent steps are executed again.

Referring to fig. 7, fig. 7 is a block diagram of an embodiment of a model optimization apparatus 70 according to the present application. The model optimization device 70 includes: the model optimization system comprises a model acquisition module 71, an information acquisition module 72, a normal optimization module 73 and a vertex optimization module 74, wherein the model acquisition module 71 is used for acquiring a model to be optimized; the model to be optimized comprises a plurality of model patches, and each model patch is composed of a model vertex; the information acquisition module 72 is configured to acquire attribute information and first normal information of each model patch, and acquire first position information of each model vertex; a normal optimization module 73, configured to optimize the first normal information based on the attribute information to obtain second normal information; and the vertex optimizing module 74 is configured to optimize the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex, so as to optimize the model to be optimized.

According to the scheme, on one hand, the model is directly optimized through normal information before and after optimization, indirect representation forms such as a truncated symbol distance field and the like are not needed, so that algorithm overhead can be greatly reduced, optimization efficiency is improved, on the other hand, connection relations between each model surface patch and each model vertex are considered in the optimization process, optimization quality is improved, and therefore quality and efficiency of model optimization can be improved.

In some disclosed embodiments, the first location information includes a first coordinate, the second location information includes a second coordinate, the vertex optimization module 74 includes a reference selection sub-module, configured to take each model vertex as a first current vertex, select a model vertex having a connecting edge with the first current vertex as a reference vertex, and take two model patches sharing the connecting edge as a first patch and a second patch, respectively; the vertex optimization module 74 includes a normal transformation submodule, configured to obtain a first normal transformation parameter of the first patch based on the first normal information and the second normal information of the first patch, and obtain a second normal transformation parameter of the second patch based on the first normal information and the second normal information of the second patch; the vertex optimization module 74 includes a function construction sub-module for constructing an objective function of the first current vertex based on the first normal transformation parameter and the second normal transformation parameter; the target function comprises a first reference subentry and a second reference subentry, wherein the first reference subentry is the product of a preset reference coefficient and a first coordinate of the first current vertex, and the second reference subentry is the product of the preset reference coefficient and a second coordinate of the first current vertex; the vertex optimization module 74 includes a function solving submodule configured to solve the second coordinates of each model vertex based on the objective function of each model vertex.

Therefore, in the optimization process of the position coordinates of the model vertex, the product of the preset reference coefficient and the first coordinate of the model vertex and the product of the preset reference coefficient and the second coordinate of the model vertex are considered at the same time and are respectively used as the sub-items of the objective function, so that the fault tolerance rate of the objective function solution is improved, and the stability of the solution result is improved.

In some disclosed embodiments, the function construction sub-module includes a diagonal acquisition unit configured to acquire a first diagonal of a connecting edge in the first patch and acquire a second diagonal of a connecting edge in the second patch; the function construction sub-module comprises a first sub-item unit, a second sub-item unit and a function construction sub-module, wherein the first sub-item unit is used for constructing a first optimization sub-item based on a first normal transformation parameter, a first opposite angle, a second normal transformation parameter, a second opposite angle, a first coordinate of a first current vertex and a first coordinate of a reference vertex; the function construction sub-module comprises a second sub-item unit, a second optimization sub-item unit and a second optimization sub-item unit, wherein the second sub-item unit is used for constructing a second optimization sub-item based on the first diagonal of the first patch, the second diagonal of the second patch, the second coordinate of the first current vertex and the second coordinate of the reference vertex; the function construction sub-module comprises a function construction unit, and the function construction unit is used for obtaining an objective function of the first current vertex based on the first optimization sub-item, the first reference sub-item, the second optimization sub-item and the second reference sub-item.

Therefore, a first object of a connecting edge in a first patch is obtained, a second opposite angle of the connecting edge in a second patch is obtained, a first optimization sub-item is constructed based on a first normal transformation parameter, the first opposite angle, a second normal transformation parameter, a second object, a first coordinate of a first current vertex and a first coordinate of a reference vertex, and a second optimization sub-item is constructed based on the first opposite angle of the first patch, the second opposite angle of the second patch, the second coordinate of the first current vertex and the second coordinate of the reference vertex.

In some disclosed embodiments, the function solution submodule includes an equation simultaneous unit for simultaneously establishing a system of equations for the second coordinates of the respective model vertices based on the objective functions of the respective model vertices; and the function solving submodule equation solving unit is used for solving the equation set to obtain second coordinates of the vertexes of the models.

Therefore, based on the objective function of each model vertex, the equation sets of the second coordinates of each model vertex are simultaneously established, and the equation sets are solved to obtain the second coordinates of each model vertex, so that the second coordinates of each model vertex can be obtained by combining the equation sets of the second coordinates of each model vertex through one-time solution, and the model optimization efficiency is further improved.

In some disclosed embodiments, the model optimization device 70 includes a loss calculation module, configured to detect that a current optimization stage meets a preset condition, and obtain an optimization loss of the model patch based on the attribute information, the first normal information, and the second normal information of the model patch; the model optimization device 70 includes a patch selection module, configured to select at least one model patch as a patch to be subdivided in a current optimization stage based on optimization loss; the model optimizing device 70 includes a model subdivision module, configured to subdivide the model to be optimized based on the patch to be subdivided, to obtain the subdivided model to be optimized, so as to update the model patch of the model to be optimized, and to perform the step of obtaining the attribute information and the first normal information of each model patch and subsequent steps again in combination with the information obtaining module 72 and the normal optimizing module 73.

Therefore, in the model optimization process, optimization losses of each model patch can be detected, patches to be subdivided are selected according to the optimization losses, the model to be optimized is subdivided according to the patches to be subdivided, the subdivided model to be optimized is obtained, the model patch of the model to be optimized is updated, model details can be further improved, and the improvement of the model optimization quality is facilitated.

In some disclosed embodiments, the patch selection module includes a sorting sub-module for sorting the model patches in order of decreasing optimization loss; the patch selection module comprises a selection submodule for selecting the model patch positioned at the front preset sequence position as the patch to be subdivided; or, the patch selection module is specifically configured to select a model patch with an optimization loss greater than a first threshold as the patch to be subdivided.

Therefore, the model patches are sorted according to the sequence of the optimization losses from large to small, the model patch positioned at the front preset order position is selected as the model to be optimized, or the model patch with the optimization loss larger than the first threshold is directly selected as the patch to be subdivided, and the model to be optimized can be refined based on the model patch with the large optimization loss, so that the detailed optimization of the part with poor details can be facilitated, and the optimization quality of the model can be improved.

In some disclosed embodiments, the model subdivision module includes a vertex adding submodule, configured to add at least one model vertex to an edge of a patch to be subdivided; the model subdivision module comprises a set acquisition submodule and a set acquisition submodule, wherein the set acquisition submodule is used for respectively taking each newly added model vertex as a second current vertex and acquiring a candidate vertex set of the second current vertex; the candidate vertex set comprises a plurality of candidate vertices, the candidate vertices are positioned on a plurality of candidate patches, and the candidate patches comprise edges where the second current vertex is positioned; the model subdivision module comprises a vertex connection submodule and is used for selecting at least one candidate vertex from the candidate vertex set and respectively connecting the at least one candidate vertex with the second current vertex to obtain a subdivided model to be optimized.

Therefore, at least one model vertex is newly added on the edge of the patch to be subdivided, each newly added model vertex is respectively used as a second current vertex, a candidate vertex set of the second current vertex is obtained, the candidate vertex set comprises a plurality of candidate vertices, the candidate vertices are positioned on a plurality of candidate patches, and the candidate patches all comprise the edge where the second current vertex is positioned.

In some disclosed embodiments, the vertex optimization module 74 is specifically configured to detect that the current optimization stage does not satisfy the preset condition, and perform a step of optimizing the first position information based on the first normal information and the second normal information to obtain second position information of the model vertex, so as to optimize the model to be optimized.

Therefore, when the current optimization stage is detected not to meet the preset condition, the first position information is optimized based on the first normal information and the second normal information, and the second position information of the model vertex is obtained, so that the model to be optimized is optimized.

In some disclosed embodiments, the model optimization apparatus 70 further includes a patch screening module, configured to screen a patch to be subdivided, whose patch area is not lower than a second threshold; the model subdivision module is specifically used for subdividing the model to be optimized based on the patch to be subdivided obtained through screening to obtain the subdivided model to be optimized.

Therefore, under the condition that the to-be-subdivided surface patches are selected based on optimization loss, the areas of the surface patches are further screened, so that a to-be-optimized model can be refined based on a model surface patch with large optimization loss and large surface patch area, on one hand, because the contribution of the model surface patch with small surface patch area to model optimization is very little, the negative influence on optimization efficiency due to excessive subdivision of the model surface patch with small surface patch area can be avoided, the optimization efficiency of the model is favorably improved, on the other hand, because the position coordinate distinguishing difficulty of the model vertex of the model surface patch with small surface patch area is also large, the stability of the optimization of the model is favorably improved.

In some disclosed embodiments, the preset conditions include: the current optimization stage does not reach the preset number of optimization stages.

Therefore, by setting the preset conditions to include: the current optimization stage does not reach the preset number of optimization stages, so that the patches to be subdivided can be selectively subdivided as required, and the optimization quality of the model is improved as much as possible on the premise of controlling the increase of the time overhead.

Referring to fig. 8, fig. 8 is a schematic block diagram of an embodiment of an electronic device 80 according to the present application. The electronic device 80 comprises a memory 81 and a processor 82 coupled to each other, the processor 82 being configured to execute program instructions stored in the memory 81 to implement the steps of any of the above-described embodiments of the model optimization method. In one particular implementation scenario, the electronic device 80 may include, but is not limited to: a microcomputer, a server, and the electronic device 80 may also include a mobile device such as a notebook computer, a tablet computer, and the like, which is not limited herein.

In particular, the processor 82 is configured to control itself and the memory 81 to implement the steps of any of the above-described embodiments of the model optimization method. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The Processor 82 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 82 may be collectively implemented by an integrated circuit chip.

According to the scheme, on one hand, the model is directly optimized through normal information before and after optimization, indirect representation forms such as a truncated symbol distance field and the like are not needed, so that algorithm overhead can be greatly reduced, optimization efficiency is improved, on the other hand, connection relations between each model surface patch and each model vertex are considered in the optimization process, optimization quality is improved, and therefore quality and efficiency of model optimization can be improved.

Referring to fig. 9, fig. 9 is a block diagram illustrating an embodiment of a computer-readable storage medium 90 according to the present application. The computer readable storage medium 90 stores program instructions 91 executable by the processor, the program instructions 91 for implementing the steps of any of the above-described embodiments of the model optimization method.

According to the scheme, on one hand, the model is directly optimized through normal information before and after optimization, indirect representation forms such as a truncated symbol distance field and the like are not needed, so that algorithm overhead can be greatly reduced, optimization efficiency is improved, on the other hand, connection relations between each model surface patch and each model vertex are considered in the optimization process, optimization quality is improved, and therefore quality and efficiency of model optimization can be improved.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like.

The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on network elements. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.