阅读说明：本技术 一种基于神经网络的面部动画绑定加速方法 (Facial animation binding acceleration method based on neural network ) 是由 赵锐 侯志迎 徐正浩 于 2021-11-08 设计创作，主要内容包括：本发明提供一种基于神经网络的面部动画绑定加速方法,包括以下步骤：S1.训练数据生成,以原始绑定文件中的基本面部表情库为基础随机生成训练数据；S2.神经网络搭建,神经网络包括多层依次连接的全连接层和PCA层,PCA层用于计算训练数据的主成分值,得到多个混合变形器,多个全连接层输入为控制器的值,输出为多个混合变形器的系数组成的多维向量；S3.训练网络模型,用于通过建立回归任务,得到损失最小化时的神经网络模型；S4.通过动画软件编写插件,将所述插件与原始绑定文件中的控制器相结合,并利用训练好的网络模型,组合形成新的绑定文件。本发明模拟逼近控制器驱动的面部Mesh变形,代替原始复杂的绑定方法。(The invention provides a facial animation binding acceleration method based on a neural network, which comprises the following steps: s1, generating training data, namely randomly generating the training data on the basis of a basic facial expression library in an original binding file; s2, building a neural network, wherein the neural network comprises a plurality of fully-connected layers and a PCA layer which are sequentially connected, the PCA layer is used for calculating the main component value of training data to obtain a plurality of mixed deformers, and the input of the plurality of fully-connected layers is the value of a controller and the output is a multi-dimensional vector consisting of the coefficients of the plurality of mixed deformers; s3, training a network model, namely obtaining a neural network model when the loss is minimized by establishing a regression task; and S4, compiling the plug-in through animation software, combining the plug-in with the controller in the original binding file, and combining to form a new binding file by utilizing the trained network model. The invention simulates the Mesh deformation of the face driven by an approximation controller, and replaces the original complex binding method.)

1. A facial animation binding acceleration method based on a neural network is characterized in that: the method comprises the following steps:

s1, generating training data, namely randomly generating the training data on the basis of a basic facial expression library in an original binding file, wherein the training data comprise values of a plurality of groups of controllers and values of Mesh vertexes corresponding to the values;

s2, building a neural network, wherein the neural network comprises a plurality of fully-connected layers and a PCA layer which are sequentially connected, the PCA layer is used for calculating the main component value of training data to obtain a plurality of mixed deformers, and the input of the plurality of fully-connected layers is the value of a controller and the output is a multi-dimensional vector consisting of the coefficients of the plurality of mixed deformers;

s3, training a network model, wherein the training network model is used for carrying out error calculation by adopting a cost function through establishing a regression task to obtain a neural network model when loss is minimized, and the cost function adopts a mean square error;

and S4, compiling the plug-in through animation software, combining the plug-in with the controller in the original binding file, and combining to form a new binding file by utilizing the trained network model.

2. The neural network-based facial animation binding acceleration method according to claim 1, characterized in that: in step S1, the training data generation method specifically includes the following steps:

s11, acquiring a face controller list based on the basic facial expression library;

s12, obtaining the value and the value range of each controller;

s13, randomly changing the values of k controllers in the value range of the controllers and repeating the values for n times to generate n groups of new controller combinations;

s14, enabling k = k + 1 and n = n-i, and repeating the step S13 until k is equal to the number of the controllers;

s15, leading all the controller combinations generated after random change into animation software, and obtaining corresponding Mesh vertex coordinates, namely, the values of each group of controllers correspond to the values of the Mesh vertices one to one;

wherein k, n and i are integers greater than 0.

3. The neural network-based facial animation binding acceleration method according to claim 1, characterized in that: in step S2, the PCA layer specifically includes the following steps:

s211, calculating the average value of the controller values of the whole data setValue of each controller in the data set minus the average valueObtaining a data difference value;

s212, adopting the data difference value in the step S211 to carry out principal component analysis, obtaining the characteristic value and the characteristic vector of the principal component, and extracting the characteristic vectors corresponding to the first n characteristic values as n mixed deformersThen the new Mesh can be represented as:wherein, in the step (A),for each mixed deformer coefficient, i.e. the output of the fully connected layer, j is an integer greater than 0 and n is the number of extracted top-ranked eigenvalues.

4. The neural network-based facial animation binding acceleration method according to claim 3, characterized in that: in step S2, the multiple fully-connected layers are used to obtain coefficients of the hybrid deformer, and calculate the predicted Mesh vertex coordinate value, where the input of the first fully-connected layer is a set of controller values, and outputs the characteristics of the hidden layer, the input of the second to mth fully-connected layers is the characteristics of the hidden layer of the first fully-connected layer, and the output is a vector formed by the coefficients of the multiple hybrid deformers, where M is the number of fully-connected layers, and M is an integer greater than 0.

5. The neural network-based facial animation binding acceleration method according to claim 1, characterized in that: in step S3, the value of the controller is used as input, the Mesh vertex coordinate is used as a label, a regression task is established, the cost function uses a mean square error to minimize the error between the value of the Mesh vertex coordinate and the network prediction result, the model parameter with the lowest loss in the training process is stored, and optimization is performed using the adam algorithm.

6. The neural network-based facial animation binding acceleration method according to claim 1, characterized in that: in step S4, the method for combining bound files specifically includes the following steps:

s41, training a pytorech frame, using a libtorech library for forward operation, combining the libtorech with an interface of animation software, and compiling a plug-in, wherein the plug-in is used for inputting a value of a controller and outputting a coordinate value corresponding to a Mesh vertex;

and S42, the controller for reserving the original binding file is combined with the plug-in the step S41 to form a new binding file, and the accelerated replacement of the binding file is completed.

7. The neural network-based facial animation binding acceleration method according to claim 6, characterized in that: the animation software includes maya and UE.

8. The neural network-based facial animation binding acceleration method according to claim 7, characterized in that: the method for writing the plug-in by the animation software by adopting the UE comprises the following steps:

q1. train the use of a pyrtch framework, predict the libtorech interface provided using a pyrtch;

and Q2, integrating the libtorch into the UE, wherein the plug-in operation comprises the following specific steps:

q21, importing a mesh vertex coordinate value in a natural state, namely the mesh vertex coordinate value in a non-expression state, into the UE;

receiving data of a controller by adopting a levellink of the UE;

q23, inputting data into the libtorch library to obtain the coordinates of the mesh vertex;

and Q24, subtracting the coordinates in the natural state from the obtained coordinate values to obtain the displacement of each vertex;

and Q25, realizing deformation in the UE according to the displacement of each vertex.

Technical Field

The invention belongs to the technical field of animation binding, and particularly relates to a facial animation binding acceleration method based on a neural network.

Background

In order to generate realistic-looking character facial animation during high fidelity digital cinema production, a bound artist typically produces extremely complex bound controls and shape-modifiers for the character head.

In the prior art, complex binding controls directly cause that a binding resolving node graph is huge and complex, a scene is too heavy, so that the software interaction speed is greatly reduced, and the efficiency of a subsequent animation production link is influenced. In fact, the speed drop caused by the binding link is always a stubborn pain point in the animation production process. Each animation company attempts to accelerate the binding solution process using various conventional methods, but these attempts have not achieved significant breakthroughs due to the limitations of the DCC software's own architecture. In the field of animation, deep learning is often adopted to perform operations such as facial expression generation, for example, patent publication No. CN112200894A discloses an automatic migration method of digital human facial expression animation based on a deep learning framework, expression migration is performed by using a trained deep learning model, a controller parameter value of one digital human is input into a network model to obtain a controller parameter value corresponding to the same expression of another digital human, and the generated controller parameter value is applied to the digital human model, so as to drive the network 3D space vertex position on the digital human model. The automatic migration replaces manual operation, and the production efficiency of the virtual character animation is greatly improved. The above patent is only applicable to expression migration, i.e. solving the problem of retargeting, but there is still no outstanding improvement of accelerated binding in the field of animation rigger.

Disclosure of Invention

The invention aims to provide a neural network-based facial animation binding acceleration method, which simulates the facial Mesh deformation driven by an approximation controller to replace the original complex binding method.

The invention provides the following technical scheme:

the application provides a facial animation binding acceleration method based on a neural network, which comprises the following steps:

s1, generating training data, namely randomly generating the training data on the basis of a basic facial expression library in an original binding file, wherein the training data comprise values of a plurality of groups of controllers and values of Mesh vertexes corresponding to the values;

s2, building a neural network, wherein the neural network comprises a plurality of fully-connected layers and a PCA layer which are sequentially connected, the PCA layer is used for calculating the main component value of training data to obtain a plurality of mixed deformers, and the input of the plurality of fully-connected layers is the value of a controller and the output is a multi-dimensional vector consisting of the coefficients of the plurality of mixed deformers;

s3, training a network model, wherein the training network model is used for carrying out error calculation by adopting a cost function through establishing a regression task to obtain a neural network model when loss is minimized, and the cost function adopts a mean square error;

and S4, compiling the plug-in through animation software, combining the plug-in with the controller in the original binding file, and combining to form a new binding file by utilizing the trained network model.

Preferably, in step S1, the training data generating method specifically includes the following steps:

s11, acquiring a face controller list based on the basic facial expression library;

s12, obtaining the value and the value range of each controller;

s13, randomly changing the values of k controllers in the value range of the controllers and repeating the values for n times to generate n groups of new controller combinations;

s14, enabling k = k + 1 and n = n-i, and repeating the step S13 until k is equal to the number of the controllers;

s15, leading all the controller combinations generated after random change into animation software, and obtaining corresponding Mesh vertex coordinates, namely, the values of each group of controllers correspond to the values of the Mesh vertices one to one;

wherein k, n and i are integers greater than 0.

Preferably, in step S2, the PCA layer specifically includes the following steps:

s211, calculating the average value of the controller values of the whole data setValue of each controller in the data set minus the average valueObtaining a data difference value;

s212, adopting the data difference value in the step S211 to carry out principal component analysis, obtaining the characteristic value and the characteristic vector of the principal component, and extracting the characteristic vectors corresponding to the first n characteristic values as n mixed deformersThen the new Mesh can be represented as:wherein, in the step (A),for each mixed deformer coefficient, i.e. the output of the fully connected layer, j is an integer greater than 0 and n is the number of extracted top-ranked eigenvalues.

Preferably, in step S2, the multiple fully-connected layers are used to obtain coefficients of the hybrid deformer, calculate predicted Mesh vertex coordinate values, where the input of the first fully-connected layer is a set of controller values, and output characteristics of the hidden layer, the input of the second to mth fully-connected layers is characteristics of the hidden layer of the first fully-connected layer, and the output is a vector formed by the coefficients of the multiple hybrid deformers, where M is the number of fully-connected layers, and M is an integer greater than 0.

Preferentially, in step S3, the value of the controller is used as input, the Mesh vertex coordinate is used as a label, a regression task is established, the cost function adopts a mean square error, the error between the value of the Mesh vertex coordinate and the network prediction result is minimized, the model parameter with the lowest loss in the training process is stored, and the adam algorithm is used for optimization.

Preferably, in step S4, the method for combining binding files specifically includes the following steps:

s41, training a pytorech frame, using a libtorech library for forward operation, combining the libtorech with an interface of animation software, and compiling a plug-in, wherein the plug-in is used for inputting a value of a controller and outputting a coordinate value corresponding to a Mesh vertex;

and S42, the controller for reserving the original binding file is combined with the plug-in the step S41 to form a new binding file, and the accelerated replacement of the binding file is completed.

Preferably, the animation software comprises maya and UE.

Preferably, the method for the animation software to write the plug-in by adopting the UE comprises the following steps:

q1. train the use of a pyrtch framework, predict the libtorech interface provided using a pyrtch;

and Q2, integrating the libtorch into the UE, wherein the plug-in operation comprises the following specific steps:

q21, importing a mesh vertex coordinate value in a natural state, namely the mesh vertex coordinate value in a non-expression state, into the UE;

receiving data of a controller by adopting a levellink of the UE;

q23, inputting data into the libtorch library to obtain the coordinates of the mesh vertex;

and Q24, subtracting the coordinates in the natural state from the obtained coordinate values to obtain the displacement of each vertex;

and Q25, realizing deformation in the UE according to the displacement of each vertex.

The invention has the beneficial effects that:

1. the method comprises the steps of constructing a neural network model, simulating the deformation of a face Mesh driven by an approximation controller, replacing an original complex binding method, accelerating the original binding efficiency, calculating a sample principal component in the neural network, reducing the training difficulty, enabling the model to be more easily converged, and enabling the convergence effect to be better as the Mesh data volume is larger;

2. data are randomly generated by using the original binding file, the artist does not need to additionally provide training data, the model is objectively generated, and the accuracy is higher;

3. the trained neural network model does not depend on animation software, can run in the corresponding animation software through interfaces of various animation software, and can also run independently without the animation software.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:

FIG. 1 is a schematic diagram of a neural network model structure of the present invention.

Detailed Description

As shown in fig. 1, the present application provides a facial animation binding acceleration method based on a neural network, including the following steps:

s1, generating training data, wherein the training data are randomly generated on the basis of a basic facial expression library in an original binding file, and comprise values of a plurality of groups of controllers and values of Mesh vertexes corresponding to the values.

Step S1 specifically includes the following steps:

s11, acquiring a facial controller list on the basis of a basic facial expression library, wherein the basic facial expression library is the basic expression of the animation adjusted by the animator and can be the basic expression approved by the director, such as happy and sad, or only the expression in the original binding file;

s12, obtaining the value and the value range of each controller;

s13, randomly changing the values of k controllers in the value range of the controllers and repeating the values for n times to generate n groups of new controller combinations;

s14, enabling k = k + 1 and n = n-i, and repeating the step S13 until k is equal to the number of the controllers;

s15, leading all the controller combinations generated after random change into animation software, and obtaining corresponding Mesh vertex coordinates, namely, the values of each group of controllers correspond to the values of the Mesh vertices one to one; wherein k, n and i are integers greater than 0.

Specific examples are as follows:

randomly changing the value of one controller respectively on the basis of a basic facial expression library to obtain a new group of controller combinations, repeating the operation a times to form the group controller combinations, randomly changing the values of two controllers, repeating the operation b times to form the group controller combinations, and repeating the previous operation until the values of all the controllers are changed to generate c data, wherein a > b >. the. And (c) importing all the controller combinations (a + b +. and. + c) into animation software through scripts to obtain corresponding character expressions, and acquiring corresponding mesh vertex coordinates. Thus, random data generation is completed. The parameters a, b, c and n are set according to the data volume, the effect is better when the data volume is larger, but the data production time and the training time are increased along with the data volume.

And S2, building a neural network, wherein the neural network comprises a plurality of fully-connected layers and a PCA layer which are sequentially connected, and the PCA layer is used for calculating the main component value of the training data to obtain a plurality of mixed deformers. After the principal component analysis is carried out, the neural network needs to learn PCA coefficients, does not need to learn the coordinate position of each Mesh vertex, reduces the training difficulty, enables the model to be more easily converged, and is suitable for the condition that the number of the Mesh vertices is large.

In step S2, the PCA layer specifically includes the following steps:

s211, calculating the average value of the controller values of the whole data setValue of each controller in the data set minus the average valueObtaining a data difference value;

s212, adopting the data difference value in the step S211 to carry out principal component analysis, obtaining the characteristic value and the characteristic vector of the principal component, and extracting the characteristic vectors corresponding to the first n characteristic values as n mixed deformersThen the new Mesh can be represented as:wherein, in the step (A),for each mixed deformer coefficient, i.e. the output of the fully connected layer, j is an integer greater than 0 and n is the number of extracted top-ranked eigenvalues.

The multiple fully-connected layers are used for obtaining coefficients of the mixed deformer and calculating a predicted Mesh vertex coordinate value, the input of the first fully-connected layer is a value of a group of controllers and outputting characteristics of the hidden layer, the input of the second fully-connected layer to the Mth fully-connected layer is characteristics of the hidden layer of the first fully-connected layer, and the output is a vector formed by the coefficients of the multiple mixed deformers, wherein M is the number of layers of the fully-connected layers, and M is an integer greater than 0. As shown in fig. 1, the neural network is composed of eight fully-connected layers and one PCA layer, the input of the first fully-connected layer is the value of one set of controllers, the characteristic of the hidden layer is output, the input of the second to seventh fully-connected layers is the characteristic of the hidden layer of the previous fully-connected layer, the characteristic of the hidden layer is output and used as the output of the next layer, the input of the eighth fully-connected layer is the characteristic of the hidden layer output by the seventh fully-connected layer, and the output is the vector composed of the coefficients of a plurality of hybrid deformers.

And S3, training a network model, wherein the training network model is used for carrying out error calculation by adopting a cost function through establishing a regression task to obtain a neural network model when the loss is minimized, and the cost function adopts a mean square error.

In step S3, the value of the controller is used as input, the Mesh vertex coordinate is used as a label, a regression task is established, the cost function uses a mean square error to minimize the error between the value of the Mesh vertex coordinate and the network prediction result, the model parameter with the lowest loss in the training process is stored, and optimization is performed using the adam algorithm.

And S4, compiling the plug-in through animation software, combining the plug-in with the controller in the original binding file, and combining to form a new binding file by utilizing the trained network model.

In step S4, the method specifically includes the following steps:

s41, training a pyrtorch frame, using a libtorch library for forward operation, combining the libtorch with an interface of animation software, compiling a plug-in, wherein the plug-in is used for inputting a value of a controller and outputting a coordinate value corresponding to a Mesh vertex. Animation software includes maya and UE.

And S42, the controller for reserving the original binding file is combined with the plug-in the step S41 to form a new binding file, and the accelerated replacement of the binding file is completed.

Taking UE as an example, C + + is adopted in a programming language, and the method for compiling the plug-in by the animation software through the UE comprises the following steps:

q1. train the use of a pyrtch framework, predict the libtorech interface provided using a pyrtch;

and Q2, integrating the libtorch into the UE, wherein the plug-in operation comprises the following specific steps:

q21, importing a mesh vertex coordinate value in a natural state, namely the mesh vertex coordinate value in a non-expression state, into the UE;

receiving data of a controller by adopting a levellink of the UE;

q23, inputting data into the libtorch library to obtain the coordinates of the mesh vertex;

and Q24, subtracting the coordinates in the natural state from the obtained coordinate values to obtain the displacement of each vertex;

and Q25, realizing deformation in the UE according to the displacement of each vertex.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.