Speech synthesis and feature extraction model training method, device, medium and equipment
阅读说明：本技术 语音合成、特征提取模型训练方法、装置、介质及设备 (Speech synthesis and feature extraction model training method, device, medium and equipment ) 是由 殷翔 于 2020-08-03 设计创作，主要内容包括：本公开涉及一种语音合成、特征提取模型训练方法、装置、介质及设备,所述语音合成方法包括：获取待处理的唇动数据；通过声学特征提取模型对所述唇动数据进行处理,以得到所述唇动数据对应的声学特征信息；根据所述声学特征信息进行语音合成,以得到与所述唇动数据对应的音频信息；其中,所述声学特征提取模型是通过将样本唇动数据对应的目标文本信息作为模型约束条件进行训练得到的。由此,可以一定程度上保证确定出的音频信息中语义连续性和准确性。该声学特征提取模型是基于文本信息对模型进行约束训练获得的,可以通过增加文本学习的辅助任务以提高该声学特征提取模型对不同的测试数据的适用,提高该声学特征提取模型的准确性和适用范围。(The present disclosure relates to a method, apparatus, medium, and device for training a speech synthesis and feature extraction model, the speech synthesis method comprising: acquiring lip movement data to be processed; processing the lip movement data through an acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data; performing voice synthesis according to the acoustic characteristic information to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition. Therefore, the semantic continuity and the accuracy in the determined audio information can be ensured to a certain extent. The acoustic feature extraction model is obtained by performing constraint training on the model based on text information, and can improve the applicability of the acoustic feature extraction model to different test data by adding an auxiliary task of text learning, thereby improving the accuracy and the applicability of the acoustic feature extraction model.)
1. A method of speech synthesis, the method comprising:
acquiring lip movement data to be processed;
processing the lip movement data through an acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data;
performing voice synthesis according to the acoustic characteristic information to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
2. The method of claim 1, wherein the acoustic feature extraction model is trained by:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
3. The method according to claim 2, wherein in the training of the neural network model, a loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and text information corresponding to the sample lip movement data output by the neural network model.
4. The method of claim 2, wherein the obtaining of the sample lip movement data, the target acoustic feature information corresponding to the sample lip movement data, and the target text information comprises:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
5. The method of any of claims 2-4, wherein the neural network model comprises a first submodel and a second submodel;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
6. The method of claim 1, wherein the obtaining lip movement data to be processed comprises:
acquiring video data to be processed;
extracting lip region images in a plurality of image frames of the video data, and using the extracted lip region images as the lip movement data.
7. The method of claim 6, further comprising:
and synthesizing the audio information into the video data to generate target video data.
8. A method for training an acoustic feature extraction model, the method comprising:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
9. The method of claim 8, wherein the loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and the text information corresponding to the sample lip movement data output by the neural network model.
10. The method of claim 8, wherein the obtaining of the sample lip movement data, the target acoustic feature information corresponding to the sample lip movement data, and the target text information comprises:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
11. The method of any one of claims 8-10, wherein the neural network model comprises a first submodel and a second submodel;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
12. A speech synthesis apparatus, characterized in that the apparatus comprises:
the first acquisition module is used for acquiring lip movement data to be processed;
the processing module is used for processing the lip movement data through an acoustic feature extraction model so as to obtain acoustic feature information corresponding to the lip movement data;
the first synthesis module is used for carrying out voice synthesis according to the acoustic characteristic information so as to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
13. An acoustic feature extraction model training apparatus, characterized in that the apparatus comprises:
the second acquisition module is used for acquiring sample lip movement data, target acoustic characteristic information corresponding to the sample lip movement data and target text information;
and the training module is used for training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as a model constraint condition to obtain the acoustic feature extraction model.
14. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1-11.
15. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 11.
Technical Field
The present disclosure relates to the field of speech synthesis technologies, and in particular, to a method, an apparatus, a medium, and a device for training a speech synthesis and feature extraction model.
Background
With the development of computer technology, the application of speech synthesis technology is becoming more important. In order to improve the accuracy of speech synthesis, the lip motion data can be used for recognition to obtain the content information corresponding to the lip motion data. In the prior art, a decision tree is usually trained according to lip motion images and voice data based on a statistical model, and then a mapping relation between the lip motion images and the voice data is determined by clustering leaf nodes in the decision tree. And then tracing back to a leaf node according to the category of the lip motion image, and adopting the clustering center mean value as a prediction result so as to determine the content information corresponding to the lip motion image. In the above manner, for the case that the number of the category samples of the lip motion image is not consistent, the problem of inaccurate content information identification is easily caused, and the manner based on the decision tree ignores the correlation between the attributes, so that the continuity between the identified content information is poor.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides a method of speech synthesis, the method comprising:
acquiring lip movement data to be processed;
processing the lip movement data through an acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data;
performing voice synthesis according to the acoustic characteristic information to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
In a second aspect, the present disclosure provides a method for training an acoustic feature extraction model, the method comprising:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
In a third aspect, the present disclosure provides a speech synthesis apparatus, the apparatus comprising:
the first acquisition module is used for acquiring lip movement data to be processed;
the processing module is used for processing the lip movement data through an acoustic feature extraction model so as to obtain acoustic feature information corresponding to the lip movement data;
the first synthesis module is used for carrying out voice synthesis according to the acoustic characteristic information so as to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
In a fourth aspect, the present disclosure provides an acoustic feature extraction model training apparatus, the apparatus comprising:
the second acquisition module is used for acquiring sample lip movement data, target acoustic characteristic information corresponding to the sample lip movement data and target text information;
and the training module is used for training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as a model constraint condition to obtain the acoustic feature extraction model.
In a fifth aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, performs the steps of the method of any one of the first or second aspects.
In a sixth aspect, the present disclosure provides an electronic device comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to implement the steps of the method of any of the first or second aspects.
In the technical scheme, lip movement data to be processed are obtained, the lip movement data are processed through an acoustic feature extraction model, acoustic feature information corresponding to the lip movement data is obtained, and therefore voice synthesis can be carried out according to the acoustic feature information, and audio information corresponding to the lip movement data is obtained. And the acoustic feature extraction model for obtaining the acoustic feature information in the process is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition. Therefore, according to the technical scheme, the acoustic feature information corresponding to the lip movement data is obtained based on the acoustic feature extraction model, on one hand, the content representation of the lip movement data can be quickly and accurately determined based on the acoustic feature extraction model, and on the other hand, because the acoustic feature extraction model is determined based on a large amount of sample data of natural users, the relevance between semantics can be learned in the model training process, so that the continuity of the determined content representation corresponding to the acoustic feature information can be ensured to a certain extent, and the continuity of the semantics in the obtained audio information is ensured. In addition, the acoustic feature extraction model can be restricted based on text information during training, so that the applicability of the acoustic feature extraction model to different test data can be improved by adding an auxiliary task of text learning, the accuracy and the application range of the acoustic feature extraction model are improved, and the accuracy of the obtained audio information is ensured. In addition, the method can be convenient for generating the audio information corresponding to the user with damaged vocal cords, handicapped people and the like, can provide technical support for good interaction of the user, is convenient for the user to use, improves the application range of the voice synthesis method, and improves the use experience of the user.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:
FIG. 1 is a flow diagram of an acoustic feature extraction model training method provided in accordance with one embodiment of the present disclosure;
FIG. 2 is a schematic diagram of an acoustic feature extraction model provided in accordance with one embodiment of the present disclosure;
FIG. 3 is a flow diagram of a method of speech synthesis provided in accordance with an embodiment of the present disclosure;
FIG. 4 is a block diagram of a speech synthesis apparatus provided in accordance with an embodiment of the present disclosure;
FIG. 5 is a block diagram of an acoustic feature extraction model training apparatus provided in accordance with an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device implementing an embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
In order to solve the problems mentioned in the background art, in the embodiments provided in the present disclosure, an acoustic feature extraction model is trained, so that the speech synthesis based on the acoustic feature extraction model is accurate and convenient, and is convenient for a user to use.
First, a training method of the acoustic feature extraction model in the embodiment of the present disclosure will be described below.
Fig. 1 is a flowchart illustrating an acoustic feature extraction model training method according to an embodiment of the present disclosure, and as shown in fig. 1, the method may include:
in step 11, sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information are acquired.
In an embodiment, a user may record a sample video by himself, for example, a camera device may be facing a face of the user, and video data including a face image when the user reads a target text is obtained by the camera device, image data in the video data may be used as the sample lip movement data, and the target text may be used as target text information, and acoustic feature extraction may be performed on audio data in the recorded video data, and the extracted acoustic feature may be determined as the target acoustic feature information. Illustratively, the acoustic feature information may be a mel-frequency spectrum (mel-frequency spectrum) feature.
In another embodiment, in step 11, an exemplary implementation of acquiring sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information is as follows, and the step may include:
the method comprises the steps of obtaining sample video data, wherein the sample video data can be video data recorded by a user, and can also be video data containing face images and audio obtained from videos such as newly broadcasted audios and videos, live videos and movie and television videos.
Then, a plurality of image frames corresponding to the target audio frame in the sample video data are determined. In the sample video data, the audio data and the image data are aligned with each other in time axis, each audio frame of the audio data may be determined first, and by determining the period information corresponding to each audio frame, the image frame corresponding to the period information may be determined as the image frame corresponding to the audio frame, thereby determining a plurality of image frames corresponding to each audio frame in the audio data.
In an embodiment, if the sample video data is video data that always contains a face image, such as a news broadcast video and a live video, each determined audio frame may be used as a target audio frame, or a part of the determined audio frames may be randomly selected as target audio frames. In another embodiment, when the sample video data contains a movie video, since there may be a case where audio exists in the current time period but the image data corresponding to the time period does not contain a face image in such a video, in this embodiment, a ratio of image frames containing a face image to a total image frame in a plurality of image frames corresponding to the audio frame may be determined, and when the ratio of the number exceeds a preset ratio threshold, the audio frame is determined as the target audio frame.
As an example, face detection may be performed on each image frame and a face image including a lip region image may be extracted, and the plurality of face images may be taken as sample lip movement data. As another example, a lip region image in each of the image frames may be extracted, and the extracted lip region images may be used as the sample lip movement data. For example, the lip region image in the image frame may be extracted by a face detection technique and a face image segmentation technique. Optionally, when the sample lip movement data is obtained by the above example, in order to facilitate the convergence speed and accuracy of the training when the sample lip movement data is subsequently used for the acoustic feature extraction model, the image extracted from the image frame may be enlarged or reduced to a preset resolution, and the processed images with multiple preset resolutions are used as the sample lip movement data, so as to facilitate the unification processing on the training sample data.
In the above manner, sample lip movement data can be obtained from a sample video, and then text information corresponding to the target audio frame can be obtained, and the text information corresponding to the target audio frame is determined as the target text information. The target text information may be a text that is read when the user records a video by himself or a text that is obtained by performing speech recognition on the audio data of the sample video data, and the text information corresponding to the target audio frame is determined from the text according to the time period information of the target audio frame and is used as the target text information.
And extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information, so that sample video data can be processed to obtain sample data for language synthesis model training.
Therefore, in the above embodiment, when sample data used for training the acoustic feature extraction model is determined, the sample data corresponding to a plurality of audio frames can be determined according to the sample video data, so that frame-level training can be performed based on the sample data, the division precision of the sample data can be improved, the precision and the application range of the acoustic feature extraction model obtained based on the sample data training can be improved to a certain extent, and the accuracy of an output result of the acoustic feature extraction model is improved.
In step 12, training the neural network model by using the sample lip movement data as model input, the target acoustic feature information as target output of the model and the target text information as model constraint conditions to obtain an acoustic feature extraction model.
The acoustic feature extraction model is used for extracting acoustic feature information of a sample lip motion data from the sample lip motion data, and the acoustic feature information is obtained by combining the acoustic feature information with the sample lip motion data.
In the technical scheme, sample lip movement data, target acoustic feature information corresponding to the sample lip movement data and target text information are obtained, the sample lip movement data is used as model input, the target acoustic feature information is used as target output of a model, and the target text information is used as a model constraint condition, a neural network model is trained, and the acoustic feature extraction model is obtained. Therefore, the acoustic feature extraction model is obtained by training the neural network model, on one hand, the relevance between semantics can be learned in the model training process, so that the continuity of the determined content representation corresponding to the acoustic feature information can be ensured to a certain extent, and the accuracy of the acoustic feature extraction model is improved; on the other hand, the application range of the acoustic feature extraction model can be widened. In addition, in the technical scheme, learning of target output of the neural network model is included, and meanwhile the model can be restrained based on text information, so that the applicability of the acoustic feature extraction model to different test data can be improved by adding an auxiliary task of text learning, the accuracy and the application range of the acoustic feature extraction model are further improved, and the acoustic feature extraction model is convenient for a user to use.
Optionally, the loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and the text information corresponding to the sample lip movement data output by the neural network model.
In this embodiment, the sample lip movement data is input to the neural network model to obtain acoustic feature information and text information corresponding to the sample lip movement data, and a first loss value may be determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and may be calculated, for example, by RMSE root mean square error; and, a second loss value may be calculated from the target text information and the text information corresponding to the sample lip movement data output by the neural network model, for example, the second loss value may be calculated by means of softmax cross entropy. The calculation method of the first loss value and the calculation method of the second loss value may be selected according to an actual usage scenario, and may be the same or different, and this is not limited in this disclosure.
Therefore, in the technical scheme, when the loss value of the neural network model is determined to train the neural network model, the loss of acoustic feature information which is a learning target of the neural network model is considered, and the constraint of the text information is considered, namely the loss value of the text information is obtained at the same time, so that the loss of the neural network model can be comprehensively determined according to the loss value and the learning target of the neural network model, and the neural network model can be converged more quickly and accurately when the model parameters of the neural network model are adjusted based on the loss value of the neural network model subsequently, so that the training efficiency of the neural network model can be improved, and the accuracy of the trained acoustic feature extraction model can be improved. In addition, the loss of acoustic feature information and text information is comprehensively considered in the process, so that the influence of the accuracy of the acoustic feature extraction model when the training data and the test data deviate can be reduced to a certain extent, the generalization of the acoustic feature extraction model obtained by training can be improved, and the application range of the acoustic feature extraction model is widened.
Optionally, as shown in fig. 2, the neural network model includes a first sub-model and a second sub-model;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
The first sub-model is used for feature coding of sample lip movement data, and may include an Embedding layer (i.e., Embedding layer), a preprocessing network (Pre-net) sub-model, and a CBHG (convergence Bank + high-speed network + bidirectional Gated Recurrent Unit, that is, CBHG is composed of a Convolution layer, a high-speed network, and a bidirectional Recurrent neural network) sub-model. The embedding layer can convert the sample lip motion data into a vector, then the vector is input into the Pre-net submodel to carry out nonlinear transformation on the vector, so that the convergence and generalization capability of the model is improved, and finally, corresponding target coding information is obtained through the CBHG submodel according to the vector after the nonlinear transformation. The convolution layer of the CBHG submodel can be a three-dimensional convolution layer so as to carry out convolution operation on time and space dimensions, and when the lip motion characteristics of each frame in the sample lip motion data are obtained, the relevance and the change of adjacent frames along with the time lapse can be expressed so as to ensure the relevance and the continuity of the lip motion characteristics.
Wherein, the second sub-model is used for decoding the target coding information of the sample lip motion data, for example, a gaussian monotonic attention gmm (gaussian Mixture model) unidirectional attribute mechanism can be adopted for decoding to ensure that the decoding obtains the order of the acoustic feature information and obtain the vector representation. The second sub-model may also include a Pre-processing network Pre-network sub-model, an Attention-RNN (current Neural network), a Decoder-RNN, and a post-processing network (postnet), wherein the post-processing network is formed by a plurality of layers of one-dimensional convolution layers. Wherein the structure of Pre-net is the same as that of Pre-net in the first sub-model, and is used for making some non-linear transformations on the input previous frame. The Attention-RNN structure may be one layer of unidirectional zoneout-based LSTM and the Decode-RNN may be two layers of unidirectional zoneout-based LSTM, resulting in a current output value, which is then fed into postnet. postnet may be a network of 5 layers of one-dimensional convolutional web layers, which may be used to predict the output of Decode-RNN to obtain textual information and acoustic feature information, respectively.
Therefore, by the technical scheme, the sample lip motion data is subjected to feature coding through the multi-layer three-dimensional convolution cyclic neural network model, so that the accuracy of extracting the lip motion features of each frame of image in the sample lip motion data can be ensured, the consideration of the relevance between the lip motion features of adjacent frames can be ensured, and the accuracy and the comprehensiveness of the obtained target coding information can be ensured. And based on a monotone attention mechanism, the orderliness of the acoustic characteristic information obtained by decoding can be ensured, so that the accuracy and the intelligibility of the speech synthesis are improved. In addition, acoustic characteristic information and text information can be predicted through the multilayer one-dimensional convolution layer, the neural network model can be constrained based on the text information, and the training efficiency and accuracy of the neural network model are improved.
The present disclosure also provides a speech synthesis method, as shown in fig. 3, the method including:
in step 31, lip movement data to be processed is acquired.
For example, the lip movement data to be processed is acquired, and face image extraction may be performed based on the acquired video data including the face image of the user, so that a plurality of acquired face images are taken as the lip movement data to be processed. As another example, obtaining lip movement data to be processed may be: acquiring video data to be processed; extracting lip region images in a plurality of image frames of the video data, and using the extracted lip region images as the lip movement data. The manner in which the lip region image is extracted is described in detail above.
As another example, after video data is extracted by any of the above manners, the obtained image is enlarged or reduced to a preset resolution, so that the obtained image with the preset resolution is determined as the lip movement data to be processed, so as to ensure the uniformity of the input data of the acoustic feature extraction model.
In step 32, the lip movement data is processed by the acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data. The acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition. Illustratively, the acoustic feature extraction model is trained by: acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data; and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
The specific training mode of the acoustic feature extraction model is described in detail above, and is not described herein again. Illustratively, the acoustic feature information is feature information used for generating speech, such as mel spectrum, sound spectrum, and the like. In this step, the lip movement data may be analyzed through the acoustic feature extraction model, so that acoustic feature information corresponding to the lip movement data may be obtained, that is, the content included in the lip movement data is represented by the acoustic feature information.
In step 33, speech synthesis is performed based on the acoustic feature information to obtain audio information corresponding to the lip movement data.
Illustratively, the audio information may be obtained by performing voice synthesis according to an acoustic feature information input Vocoder (Vocoder). The vocoder may be a neural network vocoder, for example, the acoustic feature information of the sample audio may be extracted by recording the sample audio, so as to perform training of the neural network vocoder based on the extracted acoustic feature information and the recorded sample audio. The vocoder can be Wavenet, Griffin-Lim, a single-layer recurrent neural network model WaveRNN and the like, so as to obtain better tone quality and achieve the tone quality effect close to that of the speech of a real person.
In the technical scheme, lip movement data to be processed are obtained, the lip movement data are processed through an acoustic feature extraction model, acoustic feature information corresponding to the lip movement data is obtained, and therefore voice synthesis can be carried out according to the acoustic feature information, and audio information corresponding to the lip movement data is obtained. And the acoustic feature extraction model for obtaining the acoustic feature information in the process is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition. Therefore, according to the technical scheme, the acoustic feature information corresponding to the lip movement data is obtained based on the acoustic feature extraction model, on one hand, the content representation of the lip movement data can be quickly and accurately determined based on the acoustic feature extraction model, and on the other hand, because the acoustic feature extraction model is determined based on a large amount of sample data of natural users, the relevance between semantics can be learned in the model training process, so that the continuity of the determined content representation corresponding to the acoustic feature information can be ensured to a certain extent, and the continuity of the semantics in the obtained audio information is ensured. In addition, the acoustic feature extraction model can be restricted based on text information during training, so that the applicability of the acoustic feature extraction model to different test data can be improved by adding an auxiliary task of text learning, the accuracy and the application range of the acoustic feature extraction model are improved, and the accuracy of the obtained audio information is ensured. In addition, the method can be convenient for generating the audio information corresponding to the user with damaged vocal cords, handicapped people and the like, can provide technical support for good interaction of the user, is convenient for the user to use, improves the application range of the voice synthesis method, and improves the use experience of the user.
Optionally, in the training of the neural network model, a loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and the text information corresponding to the sample lip movement data output by the neural network model.
Optionally, an exemplary implementation manner of acquiring the sample lip movement data, the target acoustic feature information corresponding to the sample lip movement data, and the target text information is as follows, and this step may include:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
Optionally, the neural network model comprises a first submodel and a second submodel;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
The specific implementation of the above steps is described in detail above, and is not described herein again.
Optionally, the method further comprises: and playing the audio information. For example, the method can be used in a real-time conversation scene of a person with a damaged vocal cord, and then the user can speak in the mouth while facing the camera device, and then lip movement data to be processed can be obtained through the above steps, so that the lip movement data can be processed, and audio information can be obtained. In the scene, the audio information can be output in real time, so that the lip language of the user can be understood by other users, the user who does not understand the lip language can know the speaking content of the user who speaks the lip language through the mode, communication between the users is facilitated, and the user use experience is improved.
Optionally, the method further comprises:
and synthesizing the audio information into the video data to generate target video data.
For example, in one application scenario, when a user with a damaged vocal cord records a video message, the user may need other users to dub the message to complete the recording of the video message. In the application scenario, the lip movement data in the face image recorded by the user is processed, so that corresponding audio information can be obtained according to the lip movement data, and then the audio information is synthesized into the video data, so that voice data of the user can be automatically generated and added into the video data, the recording of the video letter of the user is automatically completed, and the use by the user is facilitated. As an example, the audio information is synthesized into the video data to generate the target video data, the audio information may be added to the video data as a sound track, and original audio information in the video data, such as background music, may be retained to ensure the integrity of the video data.
As another example, in another application scenario, in an application scenario of video dubbing tampering, the audio information is synthesized into the video data to generate target video data, which may be replacing the audio information in the video data with the original audio information in the video data, so as to implement the original content of the character dialog in the video and implement the restoration of the video data.
The audio information is synthesized into the video data to generate target video data, and a user can preset according to an actual use scene, so that the audio information can be synthesized according to the mode set by the user during synthesis, and the generated target video data is guaranteed to be the video data required by the user.
Therefore, according to the technical scheme, new video data can be regenerated based on the generated audio information, the newly generated video data can contain corresponding audio information, and the user is prompted through the audio information about the content information in the video data, so that the user who watches videos can conveniently know the video content on one hand, the user interaction process can be simplified on the other hand, and the user use experience is improved.
The present disclosure also provides a speech synthesis apparatus, as shown in fig. 4, the
a first obtaining
the
a
Optionally, the acoustic feature extraction model is trained by:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
Optionally, in the training of the neural network model, a loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and the text information corresponding to the sample lip movement data output by the neural network model.
Optionally, the acquiring sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information includes:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
Optionally, the neural network model comprises a first submodel and a second submodel;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
Optionally, the first obtaining module includes:
the first acquisition submodule is used for acquiring video data to be processed;
the first extraction submodule is used for extracting lip region images in a plurality of image frames of the video data and taking the plurality of extracted lip region images as the lip movement data.
Optionally, the apparatus further comprises:
and the second synthesis module is used for synthesizing the audio information into the video data so as to generate target video data.
Optionally, the present disclosure further provides an acoustic feature extraction model training apparatus, as shown in fig. 5, where the apparatus 50 includes:
the second obtaining module 51 is configured to obtain sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information;
the training module 52 is configured to train a neural network model by using the sample lip movement data as a model input, using the target acoustic feature information as a target output of the model, and using the target text information as a model constraint condition, so as to obtain the acoustic feature extraction model.
Optionally, the loss value of the neural network model is determined according to a first loss value and a second loss value, wherein the first loss value is determined according to the target acoustic feature information and the acoustic feature information corresponding to the sample lip movement data output by the neural network model, and the second loss value is determined according to the target text information and the text information corresponding to the sample lip movement data output by the neural network model.
Optionally, the second obtaining module includes:
the second obtaining submodule is used for obtaining sample video data;
the first determining submodule is used for determining a plurality of image frames corresponding to a target audio frame in the sample video data;
the second extraction submodule is used for extracting lip region images in each image frame and taking the extracted lip region images as the sample lip movement data;
the third obtaining submodule is used for obtaining the text information corresponding to the target audio frame and determining the text information corresponding to the target audio frame as the target text information;
and the second determining submodule is used for extracting the acoustic characteristic information of the target audio frame and determining the extracted acoustic characteristic information as the target acoustic characteristic information.
Optionally, the neural network model comprises a first submodel and a second submodel;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
Referring now to FIG. 6, a block diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText transfer protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring lip movement data to be processed; processing the lip movement data through an acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data; performing voice synthesis according to the acoustic characteristic information to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. The name of the module does not in some cases constitute a definition of the module itself, for example, the first acquiring module may also be described as a "module that acquires lip movement data to be processed".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example 1 provides a speech synthesis method according to one or more embodiments of the present disclosure, wherein the method includes:
acquiring lip movement data to be processed;
processing the lip movement data through an acoustic feature extraction model to obtain acoustic feature information corresponding to the lip movement data;
performing voice synthesis according to the acoustic characteristic information to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
Example 2 provides the method of example 1, wherein the acoustic feature extraction model is trained by:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
Example 3 provides the method of example 2, wherein, in training the neural network model, a loss value of the neural network model is determined according to a first loss value determined according to the target acoustic feature information and acoustic feature information corresponding to the sample lip movement data output by the neural network model, and a second loss value determined according to the target text information and text information corresponding to the sample lip movement data output by the neural network model.
Example 4 provides the method of example 2, wherein the obtaining sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information, includes:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
Example 5 provides the method of any of examples 2-4, wherein the neural network model comprises a first sub-model and a second sub-model;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
Example 6 provides the method of example 1, wherein the obtaining lip movement data to be processed comprises:
acquiring video data to be processed;
extracting lip region images in a plurality of image frames of the video data, and using the extracted lip region images as the lip movement data.
Example 7 provides the method of example 6, wherein the method further comprises, in accordance with one or more embodiments of the present disclosure:
and synthesizing the audio information into the video data to generate target video data.
Example 8 provides a method of acoustic feature extraction model training, in accordance with one or more embodiments of the present disclosure, wherein the method comprises:
acquiring sample lip movement data, target acoustic characteristic information and target text information corresponding to the sample lip movement data;
and training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as the model constraint condition to obtain the acoustic feature extraction model.
Example 9 provides the method of example 8, wherein a loss value of the neural network model is determined according to a first loss value determined according to the target acoustic feature information and acoustic feature information corresponding to the sample lip movement data output by the neural network model, and a second loss value determined according to the target text information and text information corresponding to the sample lip movement data output by the neural network model.
Example 10 provides the method of example 8, wherein the obtaining sample lip movement data, target acoustic feature information corresponding to the sample lip movement data, and target text information, includes:
acquiring sample video data;
determining a plurality of image frames corresponding to a target audio frame in the sample video data;
extracting lip region images in each image frame, and taking a plurality of extracted lip region images as the sample lip movement data;
acquiring text information corresponding to the target audio frame, and determining the text information corresponding to the target audio frame as the target text information;
and extracting acoustic feature information of the target audio frame, and determining the extracted acoustic feature information as the target acoustic feature information.
Example 11 provides the method of any of examples 8-10, wherein the neural network model comprises a first sub-model and a second sub-model;
the first sub-model is a recurrent neural network model comprising a plurality of layers of three-dimensional convolutional layers and is used for carrying out characteristic coding on the sample lip motion data to obtain target coding information corresponding to the sample lip motion data; the second sub-model is an autoregressive Gaussian mixture neural network model which comprises a plurality of layers of one-dimensional convolution layers and has a monotonous attention mechanism and is used for decoding the target coding information into acoustic characteristic information and text information.
Example 12 provides a speech synthesis apparatus according to one or more embodiments of the present disclosure, wherein the apparatus comprises:
the first acquisition module is used for acquiring lip movement data to be processed;
the processing module is used for processing the lip movement data through an acoustic feature extraction model so as to obtain acoustic feature information corresponding to the lip movement data;
the first synthesis module is used for carrying out voice synthesis according to the acoustic characteristic information so as to obtain audio information corresponding to the lip movement data; the acoustic feature extraction model is obtained by training target text information corresponding to the sample lip movement data as a model constraint condition.
Example 13 provides an acoustic feature extraction model training apparatus according to one or more embodiments of the present disclosure, wherein the apparatus includes:
the second acquisition module is used for acquiring sample lip movement data, target acoustic characteristic information corresponding to the sample lip movement data and target text information;
and the training module is used for training a neural network model by taking the sample lip movement data as model input, the target acoustic feature information as the target output of the model and the target text information as a model constraint condition to obtain the acoustic feature extraction model.
Example 14 provides a computer-readable medium, on which a computer program is stored, according to one or more embodiments of the present disclosure, characterized in that the program, when executed by a processing device, implements the steps of the method of any of examples 1-11 above.
Example 15 provides, in accordance with one or more embodiments of the present disclosure, an electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing said computer program in said storage means to carry out the steps of the method of any of the above examples 1-11
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.