阅读说明：本技术 一种基于深度学习的互联网教育视频图像筛选提取法 (Internet education video image screening and extracting method based on deep learning ) 是由 王晓跃 耿晨熙 于 2021-09-03 设计创作，主要内容包括：本发明公开了一种基于深度学习的互联网教育视频图像筛选提取法,属于多媒体数据检索技术领域,该筛选提取方法的具体步骤如下：步骤一、待检测多媒体数据输入；步骤二、特征提取；步骤三、特征融合；步骤四、筛选提取；本发明采用深度学习技术对多模态数据进行特征提取和特征融合,并将其投射到同一公共空间,通过相似度计算实现了多模态数据的跨模态检索,其相较于现有单一模态检索方法而言,其有利于提高多媒体素材筛选精度和提取效率,进而有利于辅助教师进行多媒体教案制作,提高教师在制作多媒体教案时的工作效率。(The invention discloses an internet education video image screening and extracting method based on deep learning, belonging to the technical field of multimedia data retrieval, and specifically comprising the following steps: step one, inputting multimedia data to be detected; step two, feature extraction; step three, feature fusion; step four, screening and extracting; the method adopts the deep learning technology to extract and fuse the characteristics of the multi-modal data, projects the multi-modal data to the same public space, and realizes the cross-modal retrieval of the multi-modal data through similarity calculation.)

1. A method for screening and extracting Internet education video images based on deep learning is characterized by comprising the following specific steps:

inputting multimedia data to be detected, namely inputting the multimedia data to be detected, which needs to be retrieved by a user, wherein the multimedia data to be detected is multi-modal data and specifically comprises text data and image data;

step two, feature extraction, namely acquiring the text data and the image data obtained in the step one, and respectively inputting the text data and the image data into corresponding text models and image models to carry out vector feature extraction so as to respectively obtain text feature vectors and image feature vectors;

step three, feature fusion, namely acquiring the text feature vector and the image feature vector in the step two, constructing corresponding gate features and residual features of the text feature vector and the image feature vector through a fusion algorithm, performing feature fusion by adopting a metric learning method to obtain a fusion feature vector,

and step four, screening and extracting, namely performing vector transformation on multi-modal data in the multimedia teaching plan material library to obtain target data characteristic vectors, projecting the target data characteristic vectors and the fusion characteristic vectors to the same public space, performing similarity measurement calculation, sequencing according to the similarity, and screening and extracting the first N candidate data as retrieval results.

2. The method as claimed in claim 1, wherein in step one, word segmentation is performed on the text word vector before feature extraction, the word segmentation is a statistical-based word segmentation algorithm, and is used for removing stop words and dividing keywords, and the stop words include two types: one group refers to frequently occurring words and the other group refers to certain fictional words including adverb, preposition, conjunctions, and exclamation words, the stop words being replaced with symbols including "()", "-", "/", and "&", and removed from the word segmentation result.

3. The method as claimed in claim 1, wherein the text model and the image model in step two are a BERT text representation model and a VGGNet network model, respectively.

4. The method for screening and extracting video images for internet education based on deep learning as claimed in claim 1, wherein the specific process of feature fusion in step three is as follows:

s1: constructing gate characteristics and residual characteristics according to the text characteristic vectors and the image characteristic vectors by utilizing a corresponding multiplication mode of the same-position elements,

the door characteristic calculation formula is as follows:

f_gate(φ_x，φ_t)＝σ(W_g*ReLU(φ′_t))⊙φ_x (1)

the residual error characteristic calculation formula is as follows:

f_res(φ_x，φ_t)＝σ(W_r*ReLU(φ′_t) (2)

in the formula: σ is sigmoid function, W_gAnd W_rConvolution filters of 3 x 3 each, ReLU being a linear modification unit, and a calculation method of corresponding multiplication of the same-position elements;

s2: and carrying out weight proportioning on the constructed gate characteristic and the residual error characteristic, and carrying out linear combination, wherein the formula is as follows:

s3: and performing weight parameter optimization on the gate features and the residual error features by adopting a depth measurement learning mode to obtain a fusion feature vector.

5. The method as claimed in claim 4, wherein the gate features and residual features require spatial structure unification of text feature vectors and image feature vectors before construction, that is, the text feature vectors are structurally transformed by a 3 × 3 convolution filter, and the formula is as follows:

φ′_t＝W*(φ_x，φ_t) (4)

in the formula: phi'_tFor the feature vectors of the text after the structural transformation, phi_xRepresenting the image feature vector, phi_tRepresenting the text feature vector and W representing a 3 x 3 convolution filter.

6. The method for screening and extracting video images for internet education based on deep learning of claim 1, wherein the similarity measurement calculation in the fourth step is implemented by a cosine distance algorithm, and the specific formula is as follows:

in the formula: x is a fusion feature vector; y is a target data feature vector; cos is a cosine value, the value range of the cos cosine value is [ -1,1], if the cos cosine value is larger, the two vectors are more similar, otherwise, the two vectors are opposite.

Technical Field

The invention relates to the technical field of multimedia data retrieval, in particular to an internet education video image screening and extracting method based on deep learning.

Background

Through retrieval, the Chinese patent No. CN111723111A discloses a method, a device and equipment for extracting data based on video production, although the method screens out the data meeting the requirements through keyword search, the data screening precision and efficiency are relatively low because the input data mode is single; with the development of science and technology, multimedia is increasingly commonly used in teaching and occupies an increasingly important position in teaching; various schools and academic units establish multimedia classrooms or multifunctional halls; the multimedia teaching integrates sound, image, video, characters and other media, can be used as an effective auxiliary teaching means, can visually display the contents to be presented, is convenient to understand, can achieve the purposes of teaching knowledge, developing intelligence and cultivating capability, and can also achieve the purposes of teaching by factors and personalized teaching, thereby being deeply favored by teachers and students and academic institutions; however, at present, multimedia teaching plans are mostly made by teachers through office software, and teachers need to collect and import multimedia teaching plan materials required by contents of all parts in advance before making the multimedia teaching plans, however, in an era that the information is explosively increased, multimodal internet data is often unavailable to some teachers, and the teachers are difficult to accurately find the materials wanted by the teachers in mass internet data, so that the teachers often spend a lot of time on finding the materials; the deep learning technology is a new research direction in the field of machine learning, and has great potential in character, image and voice recognition, so how to combine the deep learning technology to perform multi-modal retrieval has become the focus of research; therefore, the invention of the internet education video image screening and extracting method based on deep learning becomes more important;

most of the existing multimedia data screening and extracting methods realize retrieval through a single mode, and the method has relatively low screening precision and efficiency of multimedia materials due to the fact that the mode of input data is single, and is easy to reduce the working efficiency of teachers during making multimedia teaching plans; therefore, the internet education video image screening and extracting method based on deep learning is provided.

Disclosure of Invention

The invention aims to solve the defects in the prior art and provides an internet education video image screening and extracting method based on deep learning.

In order to achieve the purpose, the invention adopts the following technical scheme:

a method for screening and extracting Internet education video images based on deep learning comprises the following specific steps:

inputting multimedia data to be detected, namely inputting the multimedia data to be detected, which needs to be retrieved by a user, wherein the multimedia data to be detected is multi-modal data and specifically comprises text data and image data;

step two, feature extraction, namely acquiring the text data and the image data obtained in the step one, and respectively inputting the text data and the image data into corresponding text models and image models to carry out vector feature extraction so as to respectively obtain text feature vectors and image feature vectors;

step three, feature fusion, namely acquiring the text feature vector and the image feature vector in the step two, constructing corresponding gate features and residual features of the text feature vector and the image feature vector through a fusion algorithm, performing feature fusion by adopting a metric learning method to obtain a fusion feature vector,

and step four, screening and extracting, namely performing vector transformation on multi-modal data in the multimedia teaching plan material library to obtain target data characteristic vectors, projecting the target data characteristic vectors and the fusion characteristic vectors to the same public space, performing similarity measurement calculation, sequencing according to the similarity, and screening and extracting the first N candidate data as retrieval results.

Further, before feature extraction, word segmentation is required for the text word vector in the step one, wherein the word segmentation is a word segmentation algorithm based on statistics and is used for removing stop words and dividing keywords, and the stop words comprise two types: one group refers to frequently occurring words and the other group refers to certain fictional words including adverb, preposition, conjunctions, and exclamation words, the stop words being replaced with symbols including "()", "-", "/", and "&", and removed from the word segmentation result.

Further, the text model and the image model in the second step are a BERT text representation model and a VGGNet network model respectively.

Further, the specific process of feature fusion in step three is as follows:

s1: constructing gate characteristics and residual characteristics according to the text characteristic vectors and the image characteristic vectors by utilizing a corresponding multiplication mode of the same-position elements,

the door characteristic calculation formula is as follows:

f_gate(φ_x,φ_t)＝σ(W_g*ReLU(φ＇_t))⊙φ_x (1)

the residual error characteristic calculation formula is as follows:

f_res(φ_x,φ_t)＝σ(W_r*ReLU(φ＇_t) (2)

in the formula: σ is sigmoid function, W_gAnd W_rConvolution filters of 3 x 3 each, ReLU being a linear modification unit, and a calculation method of corresponding multiplication of the same-position elements;

s2: and carrying out weight proportioning on the constructed gate characteristic and the residual error characteristic, and carrying out linear combination, wherein the formula is as follows:

s3: and performing weight parameter optimization on the gate features and the residual error features by adopting a depth measurement learning mode to obtain a fusion feature vector.

Further, the gate features and the residual features need to be unified in spatial structure with respect to the text feature vector and the image feature vector before being constructed, that is, the text feature vector is structurally transformed by a 3 × 3 convolution filter, and the formula is as follows:

φ＇_t＝W*(φ_x，φ_t) (4)

in the formula: phi_tFor the feature vectors of the text after the structural transformation, phi_xRepresenting the image feature vector, phi_tRepresenting the text feature vector and W representing a 3 x 3 convolution filter.

Further, the similarity measurement calculation in the fourth step is implemented by using a cosine distance algorithm, and the specific formula is as follows:

in the formula: x is a fusion feature vector; y is a target data feature vector; cos is a cosine value, the value range of the cos cosine value is [ -1,1], if the cos cosine value is larger, the two vectors are more similar, otherwise, the two vectors are opposite.

Compared with the prior art, the invention has the beneficial effects that:

the application provides an internet education video image screening extraction method based on deep learning, adopt the deep learning technique to carry out feature extraction and feature fusion to multimodal data to project it in same public space, the cross-modal retrieval of multimodal data has been realized through similarity calculation, it compares in current single mode retrieval method, it is favorable to improving multimedia material screening precision and extraction efficiency, and then is favorable to assisting the teacher to carry out the preparation of multimedia teaching plan, improves the work efficiency of teacher when making the multimedia teaching plan.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.

Fig. 1 is an overall flowchart of the internet education video image screening and extracting method based on deep learning according to the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.

Example 1

Referring to fig. 1, the embodiment discloses a screening and extracting method for internet education video images based on deep learning, which specifically comprises the following steps:

firstly, inputting multimedia data to be detected, and inputting the multimedia data to be detected, which needs to be retrieved by a user;

specifically, the multimedia data to be detected is multi-modal data, which specifically includes text data and image data, the text data and the image data can be input in a single-modal form to realize cross-modal screening extraction (i.e., cross-modal retrieval of the multimedia data is realized by single text data or single image data), or can be input in a mode combination mode to realize cross-modal screening extraction (i.e., input in a multi-modal form of text data plus image data to realize cross-modal retrieval of the multimedia data), the text word vector needs to be subjected to word segmentation before feature extraction, the word segmentation is based on a statistical word segmentation algorithm, and is used for removing stop words and dividing keywords, and the stop words include two types: one group refers to frequently occurring words, and the other group refers to certain imaginary words including adverb, preposition, conjunctions, and exclamation words, the stop words are replaced with symbols and removed from the word segmentation result, the symbols including "()", "" - ","/", and" & ".

Then, feature extraction is carried out, text data and image data are obtained and are respectively input into a corresponding text model and an image model for vector feature extraction, and a text feature vector and an image feature vector are respectively obtained;

specifically, the text model and the image model are a BERT text representation model and a VGGNet network model respectively, the BERT text representation model is a pre-training language model issued by Google in October 2018, the BERT text representation model shows very strong performance on related tasks in the field of natural language processing, and a BERT-as-service tool is used for processing original text data to obtain text feature vectors; the VGGNet network model is specifically a VGGNet-16 network model, compared with the network structures of the rest of VGGNet at different levels, the VGGNet-16 network model has higher utilization rate, only comprises convolution operation with the size of 3 multiplied by 3 and pooling operation with the size of 2 multiplied by 2 in the whole training process, proves that the network model is simpler and easier to use while taking excellent feature expression into consideration, and can efficiently obtain image feature vectors by taking the network model as a picture feature extractor;

then, feature fusion is carried out to obtain a text feature vector and an image feature vector, corresponding gate features and residual features are constructed through a fusion algorithm, feature fusion is carried out by adopting a metric learning method to obtain a fusion feature vector,

specifically, because the input text feature vector and the image feature vector are respectively output from different network models and the expressions of the text feature vector and the image feature vector on the spatial structure are not consistent, the feature fusion adopts a method for constructing and combining gate features and residual features, the two features are fused to obtain a unified expression mode which keeps consistency on the spatial structure and combines the respective features, namely, the fusion feature vector is formed by taking the original features as the basis;

finally, screening and extracting, namely performing vector transformation on multi-modal data in the multimedia teaching plan material library to obtain target data characteristic vectors, projecting the target data characteristic vectors and the fusion characteristic vectors to the same public space, performing similarity measurement calculation, sequencing according to the similarity, and screening and extracting the first N candidate data as retrieval results;

specifically, the similarity measurement calculation is implemented by a cosine distance algorithm, and the specific formula is as follows:in the formula: x is a fusion feature vector; y is a target data feature vector; cos is cosine value with the range of [ -1,1 []If the cos cosine value is larger, the two vectors are more similar, otherwise, the two vectors are opposite; compared with the existing single-mode retrieval method, the method and the device are beneficial to improving the multimedia material screening precision and the extraction efficiency, and further beneficial to assisting teachers to make multimedia teaching plans, and improving the working efficiency of teachers when making the multimedia teaching plans.

Example 2

Referring to fig. 1, this embodiment discloses a depth learning-based internet education video image screening and extracting method, which specifically describes a specific process of feature fusion, except for the same structure as the above embodiment;

specifically, the specific process of feature fusion is as follows: firstly, unifying the space structure of the text feature vector and the image feature vector, namely, performing structure transformation on the text feature vector through a 3-by-3 convolution filter, wherein the formula is as follows: phi_t＝W*(φ_x，φ_t) In the formula: phi_tFor the feature vectors of the text after the structural transformation, phi_xRepresenting the image feature vector, phi_tRepresenting the text feature vector, W representing a 3 x 3 convolution filter; then, constructing a gate feature and a residual feature by utilizing a corresponding multiplication mode of the same-position elements according to the text feature vector and the image feature vector, wherein a gate feature calculation formula is as follows: f. of_gate(φ_x,φ_t)＝σ(W_g*ReLU(φ＇_t))⊙φ_x(ii) a The residual error characteristic calculation formula is as follows: f. of_res(φ_x,φ_t)＝σ(W_r*ReLU(φ＇_t) In the formula: σ is sigmoid function, W_gAnd W_rConvolution filters of 3 x 3 each, ReLU being a linear modification unit, and a calculation method of corresponding multiplication of the same-position elements; then, weight proportion is carried out on the constructed gate features and residual error features, and linear combination is carried out, wherein the formula is as follows:finally, performing weight parameter optimization on the gate features and the residual error features by adopting a depth measurement learning mode to obtain a fusion feature vector;

in the embodiment, a deep learning technology is adopted to perform feature extraction and feature fusion on multi-modal data, the multi-modal data are projected to the same public space, and cross-modal retrieval of the multi-modal data is realized through similarity calculation.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.