阅读说明：本技术 语音识别方法、装置和电子设备 (Voice recognition method and device and electronic equipment ) 是由 宫一尘 于 2020-10-14 设计创作，主要内容包括：本公开提供了一种语音识别方法、装置和电子设备,上述语音识别方法中,在用户进行语音输入时,可以在采集音频的同时,对用户的唇部进行拍摄,然后基于当前帧图像和历史帧图像的至少一个第一唇部区域,获取用户在当前帧图像中的第二唇部区域；并行地,可以基于当前帧音频和历史帧音频的至少一个第一语音特征,获取当前帧音频的第二语音特征。之后可以根据上述语音特征和上述唇部区域,获取当前帧的音素概率分布,进而可以根据上述音素概率分布,获得当前帧音频的语音识别结果,从而可以实现将视频的唇部区域和音频的语音特征相结合,来进行语音识别,大大提高了语音识别在噪声场景下的识别效果。(The disclosure provides a voice recognition method, a voice recognition device and electronic equipment, wherein in the voice recognition method, when a user inputs voice, the lip of the user can be shot while audio is collected, and then a second lip region of the user in a current frame image is obtained based on at least one first lip region of the current frame image and a historical frame image; in parallel, a second speech feature of the current frame audio may be obtained based on at least one first speech feature of the current frame audio and the historical frame audio. And then, the phoneme probability distribution of the current frame can be obtained according to the speech characteristics and the lip region, and then the speech recognition result of the current frame audio can be obtained according to the phoneme probability distribution, so that the speech recognition can be carried out by combining the video lip region with the speech characteristics of the audio, and the recognition effect of the speech recognition in a noise scene is greatly improved.)

1. A speech recognition method comprising:

acquiring a video stream and an audio stream in a preset time period, wherein the video stream in the preset time period comprises a current frame image and a historical frame image before the current frame image, and the audio stream in the preset time period comprises a current frame audio and a historical frame audio before the current frame audio;

acquiring at least one first lip region of the historical frame image, and determining a second lip region of a user in the current frame image based on the current frame image and the at least one first lip region;

acquiring at least one first voice feature of the historical frame audio, and acquiring a second voice feature of the current frame audio based on the current frame audio and the at least one first voice feature; wherein a second lip region in the current frame image corresponds to the second speech feature;

obtaining phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature;

and obtaining a speech recognition result of the current frame audio according to the phoneme probability distribution.

2. The method of claim 1, wherein before the obtaining the video stream and the audio stream within the preset time period, the method further comprises:

carrying out voice endpoint detection on the collected audio stream;

and dividing the preset time period according to the voice endpoint detection result.

3. The method of claim 2, wherein the dividing the preset time period according to the result of the voice endpoint detection comprises:

acquiring a starting time point of each audio stream in the acquired audio streams and an ending time point corresponding to the starting time point from the result of the voice endpoint detection;

and taking the starting time point as the starting time of the preset time period, and taking the ending time point as the ending time of the preset time period.

4. The method of claim 1, wherein the determining a second lip region of a user in a current frame image based on the current frame image and the at least one first lip region comprises:

carrying out face detection on the current frame image, and positioning the face area of the user;

according to the at least one first lip region, carrying out lip detection on the face region of the user, and positioning a second lip region of the user in the current frame image;

and cutting the second lip region from the current frame image.

5. The method of claim 1, wherein the obtaining the speech recognition result of the current frame audio according to the phoneme probability distribution comprises:

and identifying through a decoder according to the phoneme probability distribution to obtain a voice identification result of the current frame audio.

6. The method according to any of claims 1-5, wherein said obtaining a phoneme probability distribution for a current frame based on the at least one first lip region, the second lip region, the at least one first speech feature, and the second speech feature comprises:

extracting first lip visual features from the at least one first lip region and second lip visual features from the second lip region;

matching and performing feature fusion on the first lip visual feature and the first voice feature in a time dimension, and matching and performing feature fusion on the second lip visual feature and the second voice feature in the time dimension;

and identifying the features obtained after feature fusion, and acquiring the phoneme probability distribution of the current frame.

7. A speech recognition apparatus comprising:

the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a video stream and an audio stream in a preset time period, the video stream in the preset time period comprises a current frame image and a historical frame image before the current frame image, and the audio stream in the preset time period comprises a current frame audio and a historical frame audio before the current frame audio; acquiring at least one first lip region of the historical frame image, and determining a second lip region of a user in the current frame image based on the current frame image and the at least one first lip region; acquiring at least one first voice feature of the historical frame audio, and acquiring a second voice feature of the current frame audio based on the current frame audio and the at least one first voice feature; wherein a second lip region in the current frame image corresponds to the second speech feature;

the recognition module is used for acquiring phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature;

and the decoding module is used for obtaining the voice recognition result of the current frame audio according to the phoneme probability distribution obtained by the recognition module.

8. The apparatus of claim 7, further comprising:

the detection module is used for carrying out voice endpoint detection on the collected audio stream before the acquisition module acquires the video stream and the audio stream within the preset time period;

and the dividing module is used for dividing the preset time period according to the voice endpoint detection result.

9. A computer-readable storage medium, which stores a computer program for executing the speech recognition method according to any one of claims 1 to 6.

10. An electronic device, the electronic device comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is used for reading the executable instructions from the memory and executing the instructions to realize the voice recognition method of any one of the above claims 1-6.

Technical Field

The present disclosure relates to the field of speech recognition technologies, and in particular, to a speech recognition method and apparatus, and an electronic device.

Background

With the progress of data processing technology and the rapid spread of mobile internet, computer technology is widely applied to various fields of society, and with the progress of data processing technology, mass data is generated. Among them, voice data is receiving more and more attention. Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims at converting the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences.

Disclosure of Invention

In the related art, a speech recognition scheme generally includes the steps of: speech signal noise reduction, feature extraction, phoneme classification and decoding, but such a speech recognition scheme is less effective in speech recognition in a high noise (i.e., low signal-to-noise ratio) scenario.

The present disclosure is proposed to solve the above technical problems. The embodiment of the disclosure provides a voice recognition method, a voice recognition device and electronic equipment.

An embodiment of a first aspect of the present disclosure provides a speech recognition method, including: acquiring a video stream and an audio stream in a preset time period, wherein the video stream in the preset time period comprises a current frame image and a historical frame image before the current frame image, and the audio stream in the preset time period comprises a current frame audio and a historical frame audio before the current frame audio; acquiring at least one first lip region of the historical frame image, and determining a second lip region of a user in the current frame image based on the current frame image and the at least one first lip region; acquiring at least one first voice feature of the historical frame audio, and acquiring a second voice feature of the current frame audio based on the current frame audio and the at least one first voice feature; wherein a second lip region in the current frame image corresponds to the second speech feature; obtaining phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature; and obtaining a speech recognition result of the current frame audio according to the phoneme probability distribution.

In the voice recognition method, when a user inputs voice, the user can shoot lips of the user while collecting audio, and then a second lip area of the user in a current frame image is obtained based on at least one first lip area of the current frame image and a historical frame image; in parallel, a second speech feature of the current frame audio may be obtained based on at least one first speech feature of the current frame audio and the historical frame audio. And then, the phoneme probability distribution of the current frame can be obtained according to the speech characteristics and the lip region, and then the speech recognition result of the current frame audio can be obtained according to the phoneme probability distribution, so that the speech recognition can be carried out by combining the video lip region with the speech characteristics of the audio, and the recognition effect of the speech recognition in a noise scene is greatly improved.

An embodiment of a second aspect of the present disclosure provides a speech recognition apparatus, including: the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a video stream and an audio stream in a preset time period, the video stream in the preset time period comprises a current frame image and a historical frame image before the current frame image, and the audio stream in the preset time period comprises a current frame audio and a historical frame audio before the current frame audio; acquiring at least one first lip region of the historical frame image, and determining a second lip region of a user in the current frame image based on the current frame image and the at least one first lip region; acquiring at least one first voice feature of the historical frame audio, and acquiring a second voice feature of the current frame audio based on the current frame audio and the at least one first voice feature; wherein a second lip region in the current frame image corresponds to the second speech feature; the recognition module is used for acquiring phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature; and the decoding module is used for obtaining the voice recognition result of the current frame audio according to the phoneme probability distribution obtained by the recognition module.

An embodiment of a third aspect of the present disclosure provides a computer-readable storage medium storing a computer program for executing the speech recognition method provided by the first aspect.

An embodiment of a fourth aspect of the present disclosure provides an electronic device, including: a processor; a memory for storing the processor-executable instructions; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the speech recognition method of the first aspect.

It should be understood that the embodiments of the second to fourth aspects of the present disclosure are consistent with the technical solutions of the embodiments of the first aspect of the present disclosure, and the beneficial effects obtained by the aspects and the corresponding possible implementation manners are similar and will not be described again.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail embodiments of the present disclosure with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally represent like parts or steps.

FIG. 1 is a flow chart of a speech recognition method provided by an exemplary embodiment of the present disclosure;

FIG. 2 is a flow chart of a speech recognition method provided by another exemplary embodiment of the present disclosure;

FIG. 3 is a flow chart of a speech recognition method provided by yet another exemplary embodiment of the present disclosure;

FIG. 4 is a flow chart of a speech recognition method provided by yet another exemplary embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of a speech recognition apparatus according to an exemplary embodiment of the present disclosure;

fig. 6 is a schematic structural diagram of a speech recognition apparatus according to another exemplary embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed Description

Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.

In order to solve the problem that the voice recognition effect is poor in a high-noise (i.e., low signal-to-noise ratio) scene in the voice recognition scheme provided by the prior art, the embodiment of the disclosure provides a voice recognition method, which constructs an acoustic model by combining dual-channel information of voice and a video sequence as input, and greatly improves the recognition effect of voice recognition in a noise scene.

Fig. 1 is a flowchart of a speech recognition method according to an exemplary embodiment of the present disclosure, and as shown in fig. 1, the speech recognition method may include:

step 101, acquiring a video stream and an audio stream within a preset time period, where the video stream within the preset time period includes a current frame image and a historical frame image before the current frame image, and the audio stream within the preset time period includes a current frame audio and a historical frame audio before the current frame audio.

102, acquiring at least one first lip region of the historical frame image, and determining a second lip region of a user in the current frame image based on the current frame image and the at least one first lip region; and acquiring at least one first voice characteristic of the historical frame audio, and acquiring a second voice characteristic of the current frame audio based on the current frame audio and the at least one first voice characteristic.

The second lip region in the current frame image corresponds to the second voice feature, and specifically, the correspondence between the second lip region and the second voice feature may be: the second lip region is obtained from the current frame image, and the second voice feature is a voice feature of audio collected when the current frame image is captured.

Specifically, based on the current frame image and the at least one first lip region, determining the second lip region of the user in the current frame image may be: the method comprises the steps of carrying out face detection on a current frame image, positioning a face area of a user, carrying out lip detection on the face area of the user according to at least one first lip area, positioning a second lip area of the user in the current frame image, and intercepting the second lip area from the current frame image.

Further, after the second lip region is cut out from the current frame image, the picture size of the second lip region may be interpolated and changed to a specified size, and buffered.

It will be appreciated that when the history frame is the current frame, the at least one first lip region of the history frame image is also obtained in the manner described above.

Specifically, based on the current frame audio and the at least one first speech feature, the obtaining of the second speech feature of the current frame audio may be: and performing signal processing on the current frame audio to reduce the noise of the current frame audio, and then performing feature extraction on the current frame audio according to at least one first voice feature to obtain a second voice feature of the current frame audio.

In specific implementation, the noise reduction algorithm such as an adaptive filter, a spectral subtraction method or a wiener filtering method can be adopted to perform signal processing on the current frame audio, so as to reduce the noise of the current frame audio. During feature extraction, short-time Fourier transform can be performed on waveform data of the current frame audio after noise reduction to obtain spectral features, and then the spectral features can be stored as voice features of the current frame audio; or, the feature extraction may be performed on the spectral feature by using a Mel Frequency Cepstrum Coefficient (MFCC) or a Filter bank (fbanks) algorithm, so as to obtain the speech feature of the current frame audio.

It will be appreciated that when the historical frame is the current frame, the at least one first speech feature of the historical frame audio is also obtained in the manner described above.

Step 103, obtaining a phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature.

Wherein, the phoneme is the minimum voice unit divided according to the natural attribute of the voice. From an acoustic property point of view, a phoneme is the smallest unit of speech divided from a psychoacoustic point of view. From a physiological point of view, a pronunciation action forms a phoneme, for example: [ ma ] includes [ m ] [ a ] two pronunciation actions, and is two phonemes.

In this embodiment, at least one first lip region corresponds to at least one first speech feature, and a second lip region corresponds to a second speech feature, and in a specific implementation, the at least one first lip region and the second lip region may be respectively matched with the at least one first speech feature and the at least one second speech feature in a time dimension after feature extraction, and then the first lip region, the second lip region, the first speech feature, and the second speech feature may be collectively input to a neural network model for prediction, and feature fusion is performed on the first lip region, the second lip region, the first speech feature, and the second speech feature in the neural network model, and features obtained after feature fusion are recognized by using the neural network model, so that a phoneme probability distribution of a current frame may be obtained.

And step 104, obtaining a speech recognition result of the current frame audio according to the phoneme probability distribution.

Specifically, according to the phoneme probability distribution, the speech recognition result of the current frame audio may be obtained as follows: and identifying through a decoder according to the phoneme probability distribution to obtain a speech identification result of the current frame audio.

In a specific implementation, a decoder based on dynamic programming search or a decoder based on beam search may be used for recognition according to the phoneme probability distribution to obtain a speech recognition result of the current frame audio.

In the voice recognition method, when a user inputs voice, the user can shoot lips of the user while collecting audio, and then a second lip area of the user in a current frame image is obtained based on at least one first lip area of the current frame image and a historical frame image; in parallel, a second speech feature of the current frame audio may be obtained based on at least one first speech feature of the current frame audio and the historical frame audio. And then, the phoneme probability distribution of the current frame can be obtained according to the speech characteristics and the lip region, and then the speech recognition result of the current frame audio can be obtained according to the phoneme probability distribution, so that the speech recognition can be carried out by combining the video lip region with the speech characteristics of the audio, and the recognition effect of the speech recognition in a noise scene is greatly improved.

Fig. 2 is a flowchart of a speech recognition method according to another exemplary embodiment of the present disclosure, as shown in fig. 2, in the embodiment shown in fig. 1 of the present disclosure, before step 101, the method may further include:

step 201, performing voice endpoint detection on the collected audio stream.

In particular, Voice Activity Detection (VAD) is generally used to discriminate between the presence of speech (speech presence) and the absence of speech (speech absence) in an audio signal.

In general, a VAD may comprise the steps of:

1) performing framing processing on the audio signal;

2) extracting features from each frame of data;

3) training a classifier on a set of data frames of a known speech and silence signal region;

4) classifying unknown framing data by using the classifier trained in the step 3) according to the features extracted in the step 2), and judging whether the unknown framing data belongs to a voice signal or a silent signal.

Step 202, dividing a preset time period according to the result of the voice endpoint detection.

Specifically, according to the result of the voice endpoint detection, dividing the preset time period may be: obtaining the starting time point of each audio stream in the collected audio stream and the ending time point corresponding to the starting time point from the result of the voice endpoint detection; and taking the starting time point as the starting time of the preset time period, and taking the ending time point as the ending time of the preset time period.

In the embodiment, the voice endpoint detection is carried out on the collected audio stream, and then the preset time period is divided according to the result of the endpoint detection, so that the audio stream and the video stream can be subjected to segment recognition when voice recognition is carried out, the data volume required to be processed by single voice recognition is greatly reduced, and the processing performance of the processor is improved.

Fig. 3 is a flowchart of a speech recognition method according to still another exemplary embodiment of the present disclosure, and as shown in fig. 3, in the embodiment shown in fig. 1 of the present disclosure, step 103 may include:

step 301, extracting a first lip visual feature from at least one first lip region, and extracting a second lip visual feature from a second lip region.

Specifically, the input lip region picture can be processed through a combination of a convolutional neural network and a pooling network, so that the dimension of the input lip region picture is reduced in the spatial dimension, and the dimension of the input lip region picture is increased in the feature dimension, thereby extracting the lip visual features from the lip region.

The lip visual features may be scale-invariant feature transform (SIFT) operator features, features extracted by a convolutional neural network, and/or optical flow features, and the specific type of the lip visual features is not limited in this embodiment.

Step 302, matching and performing feature fusion on the first lip visual feature and the first voice feature in a time dimension, and matching and performing feature fusion on the second lip visual feature and the second voice feature in the time dimension.

Specifically, the lip visual feature and the voice feature may be feature fused through a scheme such as feature concatenation, weighted summation, gate fusion, or attention fusion, and the scheme adopted by the feature fusion is not limited in this embodiment.

Step 303, identifying the features obtained after feature fusion, and obtaining phoneme probability distribution of the current frame.

Specifically, the features obtained after feature fusion may be identified by using a combination of a convolutional neural network and a pooling network, so as to obtain a phoneme probability distribution corresponding to the lip region.

In this embodiment, during speech recognition, feature fusion may be performed on the lip visual features and the speech features, and then speech recognition may be performed using the features obtained after fusion, so that the recognition accuracy of speech recognition in a noise scene may be improved.

Fig. 4 is a flowchart of a speech recognition method according to still another exemplary embodiment of the present disclosure, and as shown in fig. 4, the speech recognition method may include:

step 401, performing voice endpoint detection on the collected audio stream.

Step 402, dividing a preset time period according to the voice endpoint detection result; then, a video stream and an audio stream in a preset time period are obtained, the video stream in the preset time period comprises a current frame image and a historical frame image before the current frame image, and the audio stream in the preset time period comprises a current frame audio and a historical frame audio before the current frame audio.

Specifically, a start time point and an end time point corresponding to the start time point of each audio stream in the collected audio streams may be obtained from the result of the voice endpoint detection; and taking the starting time point as the starting time of the preset time period, and taking the ending time point as the ending time of the preset time period.

Step 403, acquiring at least one first lip region of the historical frame image, and determining a second lip region of the user in the current frame image based on the current frame image and the at least one first lip region.

Step 404, at least one first voice feature of the historical frame audio is obtained, and a second voice feature of the current frame audio is obtained based on the current frame audio and the at least one first voice feature.

In a specific implementation, step 403 and step 404 may be executed in parallel or sequentially, and the execution order of step 403 and step 404 is not limited in this embodiment.

Step 405, obtaining a phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first voice feature and the second voice feature.

Specifically, the at least one first lip region, the second lip region, the at least one first speech feature, and the second speech feature may be input to a combination of a convolutional neural network and a pooling network to obtain a phoneme probability distribution of the current frame.

And step 406, according to the phoneme probability distribution, using a decoder based on dynamic programming search or a decoder based on beam search for recognition, and obtaining a speech recognition result of the current frame audio.

Specifically, in one implementation, the dynamic decoding network simply compiles the dictionary into a state network to form a search space when using a decoder based on dynamic programming search for recognition. The general flow of compilation is: firstly, all words in a dictionary are connected in parallel to form a parallel network; then replacing the word with a phoneme string; then splitting each phoneme into a state sequence according to the context; and finally, connecting the head and the tail of the state network according to the principle that the phoneme contexts are consistent to form a loop. The network compiled in this way is generally called a linear dictionary, and is characterized in that the state sequence of each word is kept strictly independent, and the states of different words are not shared by nodes, so that the memory occupation is large, and the repeated calculation in the decoding process is more.

To overcome these disadvantages, parts of the words with the same pronunciation from beginning to end are combined, which is called a tree dictionary. Dynamic decoding is performed in a search space formed by a tree dictionary, and if an N-Gram language model is used, the identification of a current word can be known only when the search reaches a leaf node of the tree. Thus, the probabilities of the language models can only be integrated after reaching the end state of the Nth word in the N-Gram. To be able to apply dynamic programming criteria, it is common practice to organize the search space in a "tree copy" (tree copy) manner: for each predecessor history, a copy of the lexicon tree is introduced, so that the predecessor history can be known when the word-ending hypothesis occurs during the search.

Decoding search based on tree copy requires the use of Dynamic Programming (DP) algorithm. The main intent of dynamic programming is to decompose the solution of a global optimal problem into small local problems and form recursive connections.

In another implementation, the core idea of recognition by a beam search-based decoder is to track k most likely partial translations (which may be called as hyptheses hypotheses, similar to a pruning idea) at each step of the decoder, where k is the size of the beam, and the size of k may be set by itself in a specific implementation, for example, k may be a number from 5 to 10.

Assume that the decoder is predicting the target sentence that includes the words y1, y2, …, yt. y1, y2, … and yt have scores, namely the logarithmic probabilities of y1, y2, … and yt, the scores are all negative numbers, the higher the score is, the better the score is, the hypothesis with higher score can be found, and the top k translations of each step can be tracked.

According to the voice recognition method provided by the embodiment of the disclosure, when a user inputs voice, the user can shoot the lips of the user while collecting audio, and then the second lip area of the user in the current frame image is obtained based on the current frame image and at least one first lip area of the historical frame image; in parallel, a second speech feature of the current frame audio may be obtained based on at least one first speech feature of the current frame audio and the historical frame audio. And then, the phoneme probability distribution of the current frame can be obtained according to the speech characteristics and the lip region, and then the speech recognition result of the current frame audio can be obtained according to the phoneme probability distribution, so that the speech recognition can be carried out by combining the video lip region with the speech characteristics of the audio, and the recognition effect of the speech recognition in a noise scene is greatly improved.

The speech recognition method provided by the embodiment of the present disclosure may be implemented by using a general processor, may also be implemented based on an embedded edge Artificial Intelligence (AI) chip, and may also be based on a cloud neural network accelerator, for example: graphics Processing Unit (GPU).

Fig. 5 is a schematic structural diagram of a speech recognition apparatus according to an exemplary embodiment of the present disclosure, and as shown in fig. 5, the speech recognition apparatus may include: an acquisition module 51, an identification module 52 and a decoding module 53;

the acquiring module 51 is configured to acquire a video stream and an audio stream within a preset time period, where the video stream within the preset time period includes a current frame image and a historical frame image before the current frame image, and the audio stream within the preset time period includes a current frame audio and a historical frame audio before the current frame audio; acquiring at least one first lip region of a historical frame image, and determining a second lip region of a user in a current frame image based on the current frame image and the at least one first lip region; acquiring at least one first voice feature of the historical frame audio, and acquiring a second voice feature of the current frame audio based on the current frame audio and the at least one first voice feature; wherein the second lip region in the current frame image corresponds to the second speech feature;

a recognition module 52, configured to obtain a phoneme probability distribution of the current frame according to the at least one first lip region, the second lip region, the at least one first speech feature, and the second speech feature;

and the decoding module 53 is configured to obtain a speech recognition result of the current frame audio according to the phoneme probability distribution obtained by the recognition module 52.

The embodiment shown in fig. 5 provides a speech recognition apparatus, which can be used to implement the technical solution of the embodiment of the method shown in fig. 1 of the present disclosure, and the implementation principle and technical effects thereof can be further referred to the related description in the embodiment of the method.

Fig. 6 is a schematic structural diagram of a speech recognition apparatus according to another exemplary embodiment of the present disclosure, which is different from the speech recognition apparatus shown in fig. 5 in that the speech recognition apparatus shown in fig. 6 may further include: a detection module 54 and a partitioning module 55;

a detection module 54, configured to perform voice endpoint detection on the acquired audio stream before the acquisition module 51 acquires the video stream and the audio stream within the preset time period;

a dividing module 55, configured to divide the preset time period according to the result of the voice endpoint detection.

Specifically, the dividing module 55 is specifically configured to obtain, from a result of the voice endpoint detection, a start time point of each audio stream in the acquired audio streams and a termination time point corresponding to the start time point; and taking the starting time point as the starting time of the preset time period, and taking the ending time point as the ending time of the preset time period.

In this embodiment, the obtaining module 51 may include: a face detection submodule 511, a lip detection submodule 512 and a lip interception submodule 513;

the face detection sub-module 511 is configured to perform face detection on the current frame image, and locate the face area of the user;

a lip detection submodule 512, configured to perform lip detection on the face region of the user according to at least one first lip region, and locate a second lip region of the user in the current frame image;

and a lip clipping submodule 513, configured to clip a second lip region from the current frame image.

In this embodiment, the decoding module 53 is specifically configured to perform recognition by a decoder according to the phoneme probability distribution, so as to obtain a speech recognition result of the current frame audio.

In this embodiment, the identification module 52 may include: a visual feature extraction submodule 521, a feature fusion submodule 522 and a feature identification submodule 523;

wherein, the visual feature extraction submodule 521 is used for extracting a first lip visual feature from at least one first lip region and extracting a second lip visual feature from a second lip region;

a feature fusion sub-module 522, configured to match the first lip visual feature and the first voice feature in a time dimension and perform feature fusion, and match the second lip visual feature and the second voice feature in the time dimension and perform feature fusion;

the feature identification submodule 523 is configured to identify features obtained after feature fusion is performed, and obtain phoneme probability distribution of the current frame.

The speech recognition apparatus provided in the embodiment shown in fig. 6 may be used to implement the technical solutions of the method embodiments shown in fig. 1 to fig. 4 of the present disclosure, and the implementation principles and technical effects of the technical solutions may further refer to the related descriptions in the method embodiments.

Next, an electronic apparatus according to an embodiment of the present disclosure is described with reference to fig. 7. Fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure.

As shown in fig. 7, the electronic device 10 may include one or more processors 11 and memory 12.

The processor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, a Read Only Memory (ROM), a hard disk, a flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium and executed by processor 11 to implement the speech recognition methods of the various embodiments of the present disclosure described above and/or other desired functions. Various contents such as an input signal, a signal component, a noise component, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may further include: an input device 13 and an output device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

In the embodiment of the present disclosure, the input device 13 may include a microphone and a camera, where the microphone is used for collecting audio and the camera is used for shooting video.

The input device 13 may also comprise, for example, a keyboard and/or a mouse or the like.

The output device 14 can output various information including a voice recognition result and the like to the outside. The output devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.

Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present disclosure are shown in fig. 7, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 10 may include any other suitable components depending on the particular application.

In addition to the above-described methods and apparatus, embodiments of the present disclosure also provide a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the speech recognition methods according to various embodiments of the present disclosure described in the "exemplary methods" section above of this specification.

The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the speech recognition method according to various embodiments of the present disclosure described in the "exemplary methods" section above of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM) or flash memory, an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.

The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.