阅读说明：本技术 基于弱监督学习的三维人脸重建模型建立方法及其应用 (Three-dimensional face reconstruction model establishing method based on weak supervised learning and application thereof ) 是由 侯文广 梅少杰 余勤 王毅凡 董静娴 于 2021-09-27 设计创作，主要内容包括：本发明公开了一种基于弱监督学习的三维人脸重建模型建立方法,属于三维人脸重建技术领域,包括：建立用于三维人脸重建的弱监督学习模型,包括：三维人脸重建网络,用于从输入图像提取三维人脸参数；人脸特征提取模型,用于从输入图像提取人脸特征向量；GCN优化解码器,用于根据人脸特征向量对粗纹理参数进行优化,得到精纹理参数；以及三维人脸生成器,用于根据形状参数和精纹理参数生成三维人脸模型；获取人脸图像数据集,对模型进行训练；训练过程中,将模型输出的三维人脸模型渲染为二维人脸图像并计算损失值,对模型进行模型参数优化,以最小化二维人脸图像与输入图像之间的误差。本发明能够提高三维人脸模型的纹理清晰度,可应用于中医面诊。(The invention discloses a method for establishing a three-dimensional face reconstruction model based on weak supervised learning, which belongs to the technical field of three-dimensional face reconstruction and comprises the following steps: establishing a weak supervision learning model for three-dimensional face reconstruction, comprising the following steps: the three-dimensional face reconstruction network is used for extracting three-dimensional face parameters from an input image; the human face feature extraction model is used for extracting a human face feature vector from an input image; the GCN optimization decoder is used for optimizing the coarse texture parameters according to the face feature vectors to obtain fine texture parameters; the three-dimensional face generator is used for generating a three-dimensional face model according to the shape parameters and the fine texture parameters; acquiring a face image data set, and training a model; in the training process, a three-dimensional face model output by the model is rendered into a two-dimensional face image, a loss value is calculated, and model parameters of the model are optimized so as to minimize errors between the two-dimensional face image and an input image. The method can improve the texture definition of the three-dimensional face model, and can be applied to the traditional Chinese medicine facial diagnosis.)

1. A three-dimensional face reconstruction model building method based on weak supervised learning is characterized by comprising the following steps:

establishing a weakly supervised learning model for three-dimensional face reconstruction, wherein the weakly supervised learning model comprises: the three-dimensional face reconstruction network is used for extracting three-dimensional face parameters from an input image, wherein the input image contains a face image, and the three-dimensional face parameters comprise shape parameters and rough texture parameters; the human face feature extraction model is used for extracting a human face feature vector from the input image; the GCN optimization decoder is used for optimizing the coarse texture parameters according to the face feature vectors to obtain fine texture parameters; the three-dimensional face generator is used for generating a three-dimensional face model according to the shape parameters and the fine texture parameters;

acquiring a face image data set, and training the weak supervised learning model; in the training process, rendering a three-dimensional face model output by the weak supervision learning model into a two-dimensional face image, calculating a loss value based on an error between the two-dimensional face image obtained by rendering and an input image, and optimizing model parameters of the model according to the calculated loss value so as to minimize the error between the two-dimensional face image and the input image; and after the training is finished, obtaining a three-dimensional face reconstruction model.

2. The weak supervised learning based three-dimensional human face reconstruction model building method of claim 1, wherein in the training process, the loss function L used is as follows:

L＝ε₁[L_p+ε₂L_i]+ε₃L_v

wherein L is_p、L_iAnd L_vRespectively representing the color loss, the perception loss and the vertex loss of the rendered two-dimensional face image relative to the input image, epsilon₁、ε₂And ε₃Are weight constants used to control several losses.

3. The method for building a three-dimensional human face reconstruction model based on weak supervised learning as claimed in claim 1 or 2, wherein the three-dimensional human face reconstruction network is a pre-trained 3DMM regressor, and the coarse texture parameters comprise texture parameters, expression parameters, illumination parameters and pose parameters.

4. The weak supervised learning based three-dimensional human face reconstruction model building method as claimed in claim 1 or 2, wherein the human face feature extraction model comprises a human face region segmentation module and a feature extraction module;

the human face region segmentation module is used for segmenting a human face region from an input image;

the feature extraction module is used for extracting a face feature vector from the face region.

5. The method for building a three-dimensional human face reconstruction model based on weak supervised learning as claimed in claim 4, wherein the feature extraction module is a model obtained by removing a structure behind a Flatten layer from a pre-trained FaceNet model.

6. The method for building the three-dimensional human face reconstruction model based on the weak supervised learning as claimed in claim 1 or 2, wherein in the training process, a micro-renderer is used for rendering the three-dimensional human face model output by the weak supervised learning model into the two-dimensional human face image.

7. The weak supervised learning based three-dimensional human face reconstruction model building method of claim 6, wherein the micro-renderer is Softras.

8. A three-dimensional face reconstruction method is characterized by comprising the following steps:

inputting the image containing the face into the three-dimensional face reconstruction model established by the weak supervised learning based three-dimensional face reconstruction model establishing method according to any one of claims 1 to 7 so as to reconstruct the corresponding three-dimensional face model.

9. The three-dimensional face reconstruction method of claim 8, used for facial diagnosis in traditional Chinese medicine.

10. A computer-readable storage medium, comprising a stored computer program, which, when executed by a processor, controls an apparatus to perform the method for building a three-dimensional face reconstruction model based on weakly supervised learning according to any one of claims 1 to 7 and/or the method for building a three-dimensional face reconstruction according to any one of claims 8 to 9.

Technical Field

The invention belongs to the technical field of three-dimensional face reconstruction, and particularly relates to a three-dimensional face reconstruction model building method based on weak supervised learning and application thereof.

Background

The diagnosis of traditional Chinese medicine is a systematic medical theory summarized by Chinese medical experts according to clinical experience for thousands of years. The four methods of "inspection and olfaction" are the main ways of understanding and studying the disease in traditional Chinese medicine. As an important part of the diagnosis, the diagnosis has been the focus of research in the field of traditional Chinese medicine diagnostics. The face is the complement of qi and blood of zang-fu organs and the convergence of meridians. The shape, luster, texture and other characteristics of the face of the human face can well reflect the running state of qi and blood of the human body, and represent the health condition of internal organs of the human body. However, the traditional facial diagnosis method depends heavily on professional knowledge and clinical experience accumulated by experts of traditional Chinese medicine for a long time, and lacks objective and quantitative evaluation indexes; on the other hand, the method is limited by external environments such as illumination, noise and the like during the face examination of the doctor, is not beneficial to the reference of the repeated diagnosis and the experience sharing, and seriously hinders the popularization and the promotion of the traditional Chinese medicine diagnostics.

With the wide application of computer and artificial intelligence technology in many fields such as medical image processing, medical information system, etc. and playing an important role, the computer technology is used for processing and analyzing the face image of the human face, so that the automation, objectification and standardization of the traditional Chinese medicine face diagnosis process are realized, and the method is an important research direction and a modern development trend of the traditional Chinese medicine face diagnosis. Compared with a two-dimensional face image, the three-dimensional face image carries richer personalized features, and has important research significance and application prospect.

In the traditional Chinese medicine face image research, the realization of three-dimensional face image reconstruction can realize more personalized feature extraction and more accurate face region positioning so as to analyze the relationship between the face features and corresponding internal organs. However, the existing three-dimensional face reconstruction method focuses on reconstructing the three-dimensional shape of the face, but the reconstructed three-dimensional face model often has the problem of unclear texture, and is not suitable for traditional Chinese medicine face diagnosis.

Disclosure of Invention

Aiming at the defects and improvement requirements of the prior art, the invention provides a three-dimensional face reconstruction model establishing method based on weak supervised learning and application thereof, aiming at effectively solving the technical problems that the texture of the three-dimensional face model reconstructed by the existing three-dimensional face reconstruction method is not clear and is not suitable for application depending on clear face texture information, such as traditional Chinese medicine face diagnosis and the like.

In order to achieve the above object, according to an aspect of the present invention, there is provided a method for building a three-dimensional face reconstruction model based on weak supervised learning, including:

establishing a weak supervision learning model for three-dimensional face reconstruction, wherein the weak supervision learning model comprises the following steps: the three-dimensional face reconstruction network is used for extracting three-dimensional face parameters from an input image, wherein the input image contains a face image, and the three-dimensional face parameters comprise shape parameters and rough texture parameters; the human face feature extraction model is used for extracting a human face feature vector from an input image; the GCN optimization decoder is used for optimizing the coarse texture parameters according to the face feature vectors to obtain fine texture parameters; the three-dimensional face generator is used for generating a three-dimensional face model according to the shape parameters and the fine texture parameters;

acquiring a face image data set, and training a weak supervised learning model; in the training process, rendering a three-dimensional face model output by a weak supervision learning model into a two-dimensional face image, calculating a loss value based on an error between the two-dimensional face image obtained by rendering and an input image, and optimizing model parameters of the model according to the calculated loss value so as to minimize the error between the two-dimensional face image and the input image; and after the training is finished, obtaining a three-dimensional face reconstruction model.

Further, in the training process, the loss function L used is:

L＝ε₁[L_p+ε₂L_i]+ε₃L_v

wherein L is_p、L_iAnd L_vRespectively representing the color loss, the perception loss and the vertex loss of the rendered two-dimensional face image relative to the input image, epsilon₁、ε₂And ε₃Are weight constants used to control several losses.

Further, the three-dimensional face reconstruction network is a pre-trained 3DMM regression device, and the rough texture parameters comprise texture parameters, expression parameters, illumination parameters and posture parameters.

Further, the face feature extraction model comprises a face region segmentation module and a feature extraction module;

the human face region segmentation module is used for segmenting a human face region from an input image;

the feature extraction module is used for extracting face feature vectors from the face region.

Further, the feature extraction module is a model obtained by removing a structure behind a Flatten layer from a pre-trained FaceNet model.

Further, in the training process, a micro renderer is used for rendering the three-dimensional face model output by the weak supervised learning model into a two-dimensional face image.

Further, the micro-renderer is SoftRas.

According to another aspect of the present invention, there is provided a three-dimensional face reconstruction method, including:

the image containing the human face is input into the three-dimensional human face reconstruction model established by the weak supervised learning-based three-dimensional human face reconstruction model establishing method provided by the invention so as to reconstruct the corresponding three-dimensional human face model.

Furthermore, the three-dimensional face reconstruction method provided by the invention is used for the traditional Chinese medicine face diagnosis.

According to still another aspect of the present invention, there is provided a computer-readable storage medium, which includes a stored computer program, and when the computer program is executed by a processor, the computer-readable storage medium controls an apparatus to execute the weak supervised learning based three-dimensional face reconstruction model building method provided by the present invention and/or the three-dimensional face reconstruction method provided by the present invention.

Generally, by the above technical solution conceived by the present invention, the following beneficial effects can be obtained:

(1) according to the invention, after three-dimensional face parameters are extracted from the face image, face characteristic vectors are further extracted from the face image, coarse texture parameters in the three-dimensional face parameters are optimized based on the extracted face characteristic vectors to obtain finer texture parameters, and finally, a three-dimensional face model is generated based on shape parameters in the three-dimensional face parameters and the fine texture parameters obtained after optimization.

(2) In the process of training the model for three-dimensional face reconstruction, the three-dimensional face model output by the model is rendered into the two-dimensional face image, the loss value is calculated based on the error between the two-dimensional face image and the input image, and the input image is directly used as the basis of loss calculation, so that on one hand, the reconstruction precision of the model can be ensured to be higher, on the other hand, a weak supervision training mode is realized, labels do not need to be labeled in advance, and the requirement on a training sample is reduced.

(3) When the method is used for model training, the calculated loss function simultaneously comprises color loss, perception loss and vertex loss of the two-dimensional face image relative to the input image, and the three types of loss respectively reflect color difference of pixel points between the rendering image and the input image, deep characteristic difference between the rendering image and the input image and vertex error in the three-dimensional face model obtained through reconstruction.

(4) The invention firstly segments the face area from the input image, then extracts the face characteristic vector from the face area, and can reduce the interference of redundant information and improve the expression capability of the face characteristic vector through image segmentation.

(5) The method utilizes a 3DMM regressor as a three-dimensional face reconstruction network for extracting three-dimensional face parameters from an input image, wherein the extracted three-dimensional face parameters specifically comprise shape parameters, texture parameters, expression parameters, illumination parameters and attitude parameters, the method uniformly uses other parameters except the shape parameters, namely the texture parameters, the expression parameters, the illumination parameters and the attitude parameters, as rough texture parameters to optimize by using a GCN decoding optimizer, and the obtained fine texture parameters can reflect the influence of external factors such as illumination and the like on texture information, so that the precision and the definition of the texture information in a three-dimensional face model obtained by reconstruction are further improved.

(6) The invention can be applied to the field with higher requirements on the definition of the texture, such as the traditional Chinese medicine facial diagnosis.

Drawings

Fig. 1 is a schematic diagram of a three-dimensional face reconstruction model establishing method based on weak supervised learning according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a conventional GCN decoding optimizer network;

FIG. 3 is a schematic diagram of a conventional micro-renderer;

fig. 4 is a schematic diagram of a three-dimensional face reconstruction effect according to an embodiment of the present invention; the method comprises the following steps of (a) inputting a face image, (b) view of a three-dimensional face model obtained by model reconstruction under different angles, and (e) rendering a three-dimensional face model obtained by model reconstruction to obtain a two-dimensional face image.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

In the present application, the terms "first," "second," and the like (if any) in the description and the drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

The invention provides a three-dimensional face reconstruction model establishing method based on weak supervision learning and application thereof, aiming at solving the technical problems that the texture of a three-dimensional face model reconstructed by the existing three-dimensional face reconstruction method is not clear and is not suitable for application depending on clear face texture information, such as traditional Chinese medicine face diagnosis.

The following are examples.

Example 1:

a three-dimensional face reconstruction model building method based on weak supervised learning comprises the following steps: a model establishing step and a model training step.

In this embodiment, the model establishing step specifically includes:

establishing a weak supervision learning model for three-dimensional face reconstruction, wherein the weak supervision learning model is shown in figure 1 and comprises the following steps: the system comprises a three-dimensional face reconstruction network, a face feature extraction model, a GCN optimization decoder and a three-dimensional face generator, wherein:

the three-dimensional face reconstruction network is used for extracting three-dimensional face parameters from an input image, wherein the input image contains a face image, and the three-dimensional face parameters comprise shape parameters and rough texture parameters; as an optional implementation manner, as shown in fig. 1, in this embodiment, the three-dimensional face reconstruction model is a pre-trained 3DMM regressor, and the extracted three-dimensional face parameters include a shape parameter, a texture parameter, an expression parameter, an illumination parameter, and a pose parameter, and optionally, in this embodiment, the three-dimensional face parameters are expressed as:

the parameters of the shape are represented by,the expression parameters are represented by a plurality of expression parameters,the parameters of the texture are represented by,which is indicative of a parameter of the illumination,representing an attitude parameter; 3DMM is linearly combined with a shape vector and a texture vector based on pca (principal Component analysis) method; the calculation formula of the shape S and the texture T of the three-dimensional face to be reconstructed is as follows:

in the formula (I), the compound is shown in the specification,andrespectively, the average Face shape and texture in the BFM (Basel Face model) database, B_i、B_eAnd B_tPCA base vectors of the shape, expression and texture of the three-dimensional face after standard deviation scaling are respectively; in a traditional three-dimensional face reconstruction method, after the three-dimensional face parameters are extracted, corresponding three-dimensional face models are generated directly based on the parameters, and the three-dimensional face models are reconstructedThe texture of the three-dimensional face model is often unclear; in the embodiment, other parameters except the shape parameter, namely, a texture parameter, an expression parameter, an illumination parameter and a posture parameter are unified as a rough texture parameter, and the rough texture parameter is further optimized by using other structures in the model; it should be noted that the 3DMM regressor is only an optional embodiment of the present invention, and should not be understood as the only limitation to the present invention, and other manners that can extract three-dimensional face parameters from a face image may also be used in the present invention;

the human face feature extraction model is used for extracting a human face feature vector from an input image; as shown in fig. 1, in this embodiment, the face feature extraction model specifically includes a pre-trained face region segmentation module and a feature extraction module, where the face region segmentation module is used to segment a face region from an input image to reduce interference of redundant information; in this embodiment, the face region segmentation module may be implemented by any image segmentation model;

the feature extraction module is configured to extract a face feature vector from the face region, and optionally, in this embodiment, specifically, a network structure after a scatter layer in a pre-trained FaceNet model is removed, and a remaining structure is used as the feature extraction module; in the embodiment, the faceNet pre-training model is a model trained by an official using a CASIA-Webface data set; it should be noted that the feature extraction module constructed here is only an optional implementation, and should not be understood as the only limitation of the present invention, and any other model that can extract a face feature vector from a face region can be used in the present invention;

the GCN optimization decoder is used for optimizing the coarse texture parameters according to the face feature vectors to obtain fine texture parameters; the structure of the GCN optimized decoder is shown in fig. 2, which includes a GCN decoder using 4 spectral residual blocks, each containing two chebyshev convolutional layers and activated using a ReLU, and a GCN optimization; the GCN optimizer uses similar GCN spectrum residual blocks, and an up-sampling layer and a down-sampling layer are respectively added at the top and the bottom; the human face feature vector is used as the input of a GCN decoder, the coarse texture parameter is used as the input of a GCN optimizer, the outputs of the GCN decoder and the GCN optimizer are connected in series and input into a Chebyshev convolution layer, so that the optimization of the coarse texture parameter can be realized, and the fine texture parameter is output;

the three-dimensional face generator is used for generating a three-dimensional face model according to the shape parameters and the fine texture parameters; because the texture parameters are optimized, the three-dimensional face model generated by the three-dimensional face generator is also an optimized three-dimensional face model, and the texture is clearer; as an optional implementation manner, in this embodiment, a pre-trained 3d mm regression is directly used as a three-dimensional face generator, and the optimized three-dimensional face parameters formed by the shape parameters and the fine texture parameters are input into the three-dimensional face generator, so that a corresponding three-dimensional face model can be generated;

in this embodiment, the model training step specifically includes:

acquiring a face image data set, and training a weak supervised learning model; it is easy to understand that, in the face image data set acquired here, the size of the face image should meet the input requirement of the model, optionally, in this embodiment, the size of the face image in the face image data set is specifically 600 × 900, since the size of the image actually acquired by the camera is generally large, for example, 4000 × 6000, at this time, the image needs to be resampled to a specified size; in other embodiments of the present invention, the size of the model input image may be set to other sizes as desired;

in order to quantitatively calculate the reconstruction loss of the model and to measure the training effect of the model, as shown in fig. 1, in the training process, the three-dimensional face model output by the weak supervised learning model is rendered into a two-dimensional face image, a loss value is calculated based on an error between the rendered two-dimensional face image and an input image, and model parameters are optimized according to the calculated loss value to minimize the error between the two-dimensional face image and the input image; after training is finished, a three-dimensional face reconstruction model is obtained;

optionally, in this embodiment, the three-dimensional face model output by the weakly supervised learning model is rendered into a two-dimensional face image by using SoftRas, which is a micro-renderer and has a structure as shown in fig. 3, and the process of micro-rendering the three-dimensional face model is as follows:

the input of the feature extraction part is to optimize the shape parameter S and the texture parameter T of the three-dimensional face model, and meanwhile, the camera parameter P and the illumination parameter L are used as external environment variables to participate in the regulation and control of the rendering process; the shape parameters S generate a normal vector N, a depth Z and corresponding screen coordinates U of the face grid through coordinate transformation; in a specific illumination environment L, calculating the vertex color C of the model through a grid method vector N and a model texture parameter T; the input of the Softras rendering process is a grid depth Z, a screen coordinate U and a vertex color C, the process firstly carries out probability mapping on the screen coordinate U, calculates the pixel probability D of each point, and then uses an aggregation function to fuse the characteristics of the pixel probability D, the grid depth Z and the vertex color C to generate a rendering image I;

it should be noted that SoftRas is only one alternative embodiment of the present invention, and should not be construed as the only limitation to the present invention, and in other embodiments of the present invention, other modes of rendering display may be used.

In order to further ensure the reconstruction effect of the model, the embodiment designs a mixed loss function, where L represents the loss function, and the expression of the mixed loss function is:

L＝ε₁[L_p+ε₂L_i]+ε₃L_v

wherein L is_p、L_iAnd L_vThe method comprises the steps of respectively representing color loss, perception loss and vertex loss of a two-dimensional face image obtained through rendering relative to an input image, wherein the three types of loss respectively reflect color difference of pixel points between the rendered image and the input image, deep feature difference between the rendered image and the input image and vertex error in a three-dimensional face model obtained through reconstruction; epsilon₁、ε₂And ε₃For the weight constant used to control several losses, alternatively, in this embodiment, ε₂Is set to 0.2 epsilon₁And epsilon₃Gradually adjusted from 0 and 1 to 1 and 0 respectively;

specifically, in the mixed loss function L, the three types of losses are calculated as follows:

in order to compare the difference between the rendered image and the input image, the face area is segmented from the input image in advance in the network structure, the interference of redundant information is reduced, and then the pixel loss between the rendered image and the face segmented image is calculated by adopting the following loss function:

where M represents the face region in the projected image, i represents the pixel location, A_iConfidence indicating I pixel location, provided by Mask, I_iAnd l'_iRespectively representing the colors of pixel points of the input image and the projected image;

extracting perception layer information of the face image by adopting a pre-trained faceNet network, and fitting semantic features of the face model by the following perception loss functions:

wherein, f (I) and f' (I) represent the deep features of the input two-dimensional face image and the rendering image respectively, and the value is less than or greater than the inner product;

and calculating the vertex loss according to the difference between the vertex colors of the rendering image of the three-dimensional reconstruction human face model and the input two-dimensional human face image. The loss function is formulated as follows:

wherein | x | represents the error of predicting the key point and the key point label, ω and ε are parameters of a logarithmic function, ω represents the range of nonlinearity, ε represents the curvature, and C is a constant controlled by ω and ε for smoothing the linear and nonlinear parts of the loss function; alternatively, in the present embodiment, ω and ε are empirically set to 10 and 2.

Example 2:

a three-dimensional face reconstruction method comprises the following steps:

the image containing the face is input into the three-dimensional face reconstruction model established by the method for establishing the three-dimensional face reconstruction model based on the weak supervised learning provided by the embodiment 1 of the invention so as to reconstruct the corresponding three-dimensional face model.

The embodiment can be applied to the application with higher requirements on texture information, such as the traditional Chinese medicine surface diagnosis.

Example 3:

a computer-readable storage medium, including a stored computer program, where when the computer program is executed by a processor, the computer-readable storage medium controls an apparatus to execute the method for building a three-dimensional face reconstruction model based on weakly supervised learning according to embodiment 1 of the present invention, and/or the method for building a three-dimensional face reconstruction according to embodiment 2 of the present invention.

The three-dimensional face reconstruction effect of the present invention will be further described with reference to some specific reconstruction results.

A plurality of face images shown in (a) in fig. 4 are respectively used as the input of a three-dimensional face reconstruction model, and the corresponding three-dimensional face models are output by the model, wherein the views of the three-dimensional face models at different angles are respectively shown in (b), (c) and (d) in fig. 4; according to the results shown in (b) to (d) in fig. 4, the three-dimensional face model reconstructed by the method has a good reconstruction effect at each angle, the three-dimensional structural features of the face are obvious, the face texture is clear, the real skin color and texture can be effectively reflected, and the method can be applied to the traditional Chinese medicine face diagnosis.

Further rendering each three-dimensional face model to obtain a corresponding two-dimensional face image, as shown in (e) in fig. 4, comparing (e) in fig. 4 with (a) in fig. 4, it can be seen that the reconstructed rendered image has a higher correlation with the input image, the corresponding features of the face are better reconstructed, the effect of recovering the detail information is obvious, and further the effectiveness of the method provided by the present invention is explained.

It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.