阅读说明：本技术 古诗词生成方法、装置、设备及存储介质 (Ancient poetry generation method, device and equipment and storage medium ) 是由 刘艳鹏 张引 于 2019-10-17 设计创作，主要内容包括：本发明提供的古诗词生成方法、装置、设备及存储介质,通过对多模态信息进行关键词提取,实现了多模态信息的规范化处理,进而能够接受更多种模态信息,支持更广泛的实际应用场景,充分反映用户意图；然后通过各模态信息的概率统计结果,为多模态信息的处理提供了数据基础；利用古诗词映射知识图谱,解决了现代汉语与古汉语之间的映射问题；最后,将得到的预设数目个古诗词关键词确定为古诗词主题词,并利用预先训练好的古诗词生成模型,生成与所述古诗词主题词相对应的目标古诗词,从而在古诗词生成过程中,充分融合了多模态信息,更全面的体现了用户意图,有效提升了古诗词生成质量。(According to the ancient poetry generating method, the ancient poetry generating device, the ancient poetry generating equipment and the ancient poetry generating storage medium, keyword extraction is carried out on multi-modal information, standardized processing of the multi-modal information is achieved, more modal information can be received, more -extensive practical application scenes can be supported, user intentions can be fully reflected, a data base is provided for processing of the multi-modal information through probability statistical results of the modal information, the ancient poetry mapping knowledge graph is used for solving the mapping problem between modern Chinese and ancient Chinese, finally, the obtained ancient poetry keywords are determined to be ancient poetry subject terms, target ancient poetry corresponding to the ancient poetry subject terms is generated through a pre-trained ancient poetry generating model, accordingly, the multi-modal information is fully fused in the ancient poetry generating process, user intentions are more fully reflected, and the ancient poetry generating quality is effectively improved.)

1, ancient poetry generating method, characterized by comprising:

acquiring multi-modal information input by a user, wherein the multi-modal information comprises at least two of images, videos, texts and audios;

extracting keywords from each modal information in the multi-modal information respectively to obtain a modern Chinese keyword corresponding to each modal information respectively as a target modern Chinese keyword;

acquiring the occurrence probability of each modal information according to historical modal information statistical data;

acquiring ancient Chinese keywords with the highest relevance with the target modern Chinese keywords according to a pre-constructed ancient poetry mapping knowledge graph, and taking the ancient Chinese keywords as the target ancient Chinese keywords;

determining the weight of the target ancient Chinese keywords according to the correlation degree between the target ancient Chinese keywords and the target modern Chinese keywords and the occurrence probability of each modal information;

determining a preset number of ancient Chinese keywords with the maximum weight in the target ancient Chinese keywords as ancient poem topic words;

and generating a target ancient poetry corresponding to the ancient poetry theme by utilizing a pre-trained ancient poetry generating model.

2. The method of claim 1, wherein the performing keyword extraction on each modal information in the multi-modal information respectively to obtain a modern chinese keyword corresponding to each modal information respectively, and the performing the target modern chinese keyword comprises:

when the multi-mode information comprises a text, identifying keywords in the text by using a named entity identification technology to obtain modern Chinese keywords corresponding to the text, and using the modern Chinese keywords as target modern Chinese keywords;

when the multi-mode information comprises audio, firstly, converting the voice in the audio into a voice recognition text by using a voice recognition technology, and then, recognizing key words in the voice recognition text by using a named entity recognition technology to obtain modern Chinese key words corresponding to the audio as target modern Chinese key words;

when the multi-mode information comprises an image, firstly identifying the content information of the image by using an image identification technology, and then determining a modern Chinese keyword corresponding to the image as a target modern Chinese keyword according to the content information of the image;

when the multi-mode information comprises a video, identifying a key frame image in the video by using a video understanding technology, identifying content information of the key frame image by using an image identification technology, and determining a modern Chinese keyword corresponding to the video as a target modern Chinese keyword according to the content information of the key frame image.

3. The method according to claim 1, wherein the obtaining the occurrence probability of each modality information according to the statistical data of the historical modality information comprises:

the probability of occurrence P (B) of the target mode information Bi is calculated by the following formula_i)：

P(B_i)＝COUNT(B_i)/COUNT(B)，i∈[1,n]，B＝∑B_i；

Wherein, COUNT (B)_i) Representing target modal information B in the multi-modal information_iNumber of occurrences in historical modality information; count (B) represents the total number of occurrences of all modality information B in the history modality information in the multimodal information.

4. The method of claim 1, wherein the representation of modal information in the multimodal information comprises a combination of single modal information and multimodal information.

5. The method of claim 1, wherein said determining a weight of the target ancient Chinese keyword based on a degree of correlation between the target ancient Chinese keyword and the target modern Chinese keyword and a probability of occurrence of the modal information comprises:

determining the probability corresponding to the target ancient Chinese keywords according to the corresponding relation between the modal information and the target modern Chinese keywords, the incidence relation between the target modern Chinese keywords and the target ancient Chinese keywords and the occurrence probability of the modal information;

and taking the product of the corresponding relevancy and the probability of the target ancient Chinese key words as the weight of the target ancient Chinese key words.

6. The method of claim 1, wherein the determining a preset number of the targeted ancient chinese keywords having a greatest weight as ancient poetry topic words comprises:

when the number of the target ancient Chinese keywords is larger than or equal to a preset number, taking the ancient Chinese keywords with the maximum weight in the ancient Chinese keywords as ancient poem subject terms;

and when the number of the target ancient Chinese keywords is less than the preset number, expanding the target ancient Chinese keywords by using ancient Chinese Word vectors trained in advance by means of Topical Word Embedding technology to obtain the target ancient Chinese keywords with the preset number, and using the target ancient Chinese keywords as ancient poem subject terms.

7. The method of claim 1, wherein the generating of the target ancient poetry corresponding to the ancient poetry subject matter using a pre-trained ancient poetry generating model comprises:

acquiring a pre-trained ancient poetry generating model;

and inputting the subject terms of the ancient poetry into the ancient poetry generating model, and acquiring the target ancient poetry output by the ancient poetry generating model.

8, kind of ancient poetry generating device, its characterized in that includes:

the system comprises an input information acquisition unit, a processing unit and a display unit, wherein the input information acquisition unit is used for acquiring multi-modal information input by a user, and the multi-modal information comprises at least two of images, videos, texts and audios;

the modern vocabulary extraction unit is used for respectively extracting keywords from each modal information in the multi-modal information to obtain a modern Chinese keyword corresponding to each modal information as a target modern Chinese keyword;

the modal probability obtaining unit is used for obtaining the occurrence probability of each modal information according to the statistical data of the historical modal information;

the ancient Chinese vocabulary matching unit is used for acquiring an ancient Chinese keyword with the highest relevance with the target modern Chinese keyword as the target ancient Chinese keyword according to a pre-constructed ancient poetry mapping knowledge graph;

the ancient Chinese weight determining unit is used for determining the weight of the target ancient Chinese key words according to the correlation degree between the target ancient Chinese key words and the target modern Chinese key words and the occurrence probability of each modal information;

a poetry theme determining unit, configured to determine a preset number of ancient Chinese keywords with the largest weight in the target ancient Chinese keywords as ancient poetry theme words;

and the target poetry generating unit is used for generating target ancient poetry corresponding to the subject term of the ancient poetry by utilizing an ancient poetry generating model trained in advance.

9, ancient poetry generating equipment, which is characterized by comprising a processor and a memory;

the memory is used for storing programs;

the processor is used for executing the program stored in the memory to realize the ancient poetry generating method as claimed in any of claims 1-7.

10, kinds of computer-readable storage media, characterized in that the computer-readable storage media have stored therein a program which, when executed by a processor, realizes the ancient poetry generating method according to any of claims 1 to 7.

Technical Field

The invention relates to the technical field of natural language processing, in particular to a method, a device, equipment and a storage medium for generating ancient poems.

Background

Natural Language Generation (NLG) technology is becoming more and more popular in industrial applications, and with the rapid development of Natural Language Processing (NLP) technology, generating coherent natural language utterances from a given topic (structured data) becomes an emerging topic of hot , and it becomes possible to make a computer perform the creation of ancient poems using machine learning.

In addition , in practical application, for the generation of ancient poems, information given by users is usually multi-modal, for example, the users may give various types of information such as texts, images, videos, audio and the like at the same time, and how to improve the ancient poem generation quality by fusing the multi-modal information does not have a good solution at present.

Therefore, there is an urgent need for practical and effective ancient poetry generating schemes to fuse multi-modal information to improve the quality of the ancient poetry generating results.

Disclosure of Invention

In view of this, the invention provides ancient poetry generating methods and devices, so as to solve the technical problems that the ancient poetry generating quality is poor based on single modal information, and the quality of the ancient poetry generating result is difficult to improve by fusing multi-modal information at present.

In order to achieve the purpose, the invention provides the following technical scheme:

ancient poetry generating method, comprising:

acquiring multi-modal information input by a user, wherein the multi-modal information comprises at least two of images, videos, texts and audios;

extracting keywords from each modal information in the multi-modal information respectively to obtain a modern Chinese keyword corresponding to each modal information respectively as a target modern Chinese keyword;

acquiring the occurrence probability of each modal information according to historical modal information statistical data;

acquiring ancient Chinese keywords with the highest relevance with the target modern Chinese keywords according to a pre-constructed ancient poetry mapping knowledge graph, and taking the ancient Chinese keywords as the target ancient Chinese keywords;

determining the weight of the target ancient Chinese keywords according to the correlation degree between the target ancient Chinese keywords and the target modern Chinese keywords and the occurrence probability of each modal information;

determining a preset number of ancient Chinese keywords with the maximum weight in the target ancient Chinese keywords as ancient poem topic words;

and generating a target ancient poetry corresponding to the ancient poetry theme by utilizing a pre-trained ancient poetry generating model.

Preferably, the extracting keywords from each modal information in the multi-modal information to obtain the modern chinese keywords corresponding to each modal information respectively, and the extracting keywords as the target modern chinese keywords include:

when the multi-mode information comprises a text, identifying keywords in the text by using a named entity identification technology to obtain modern Chinese keywords corresponding to the text, and using the modern Chinese keywords as target modern Chinese keywords;

when the multi-mode information comprises audio, firstly, converting the voice in the audio into a voice recognition text by using a voice recognition technology, and then, recognizing key words in the voice recognition text by using a named entity recognition technology to obtain modern Chinese key words corresponding to the audio as target modern Chinese key words;

when the multi-mode information comprises an image, firstly identifying the content information of the image by using an image identification technology, and then determining a modern Chinese keyword corresponding to the image as a target modern Chinese keyword according to the content information of the image;

when the multi-mode information comprises a video, identifying a key frame image in the video by using a video understanding technology, identifying content information of the key frame image by using an image identification technology, and determining a modern Chinese keyword corresponding to the video as a target modern Chinese keyword according to the content information of the key frame image.

Preferably, the obtaining of the occurrence probability of each modal information according to the statistical data of the historical modal information includes:

the probability of occurrence P (B) of the target mode information Bi is calculated by the following formula_i)：

P(B_i)＝COUNT(B_i)/COUNT(B)，i∈[1,n]，B＝∑B_i；

Wherein, COUNT (B)_i) Representing target modal information B in the multi-modal information_iNumber of occurrences in historical modality information; count (B) represents the total number of occurrences of all modality information B in the history modality information in the multimodal information.

Preferably, the representation form of the modal information in the multi-modal information includes single-modal information and multi-modal combined information.

Preferably, the determining the weight of the target ancient chinese keyword according to the degree of correlation between the target ancient chinese keyword and the target modern chinese keyword and the occurrence probability of each modal information includes:

determining the probability corresponding to the target ancient Chinese keywords according to the corresponding relation between the modal information and the target modern Chinese keywords, the incidence relation between the target modern Chinese keywords and the target ancient Chinese keywords and the occurrence probability of the modal information;

and taking the product of the corresponding relevancy and the probability of the target ancient Chinese key words as the weight of the target ancient Chinese key words.

Preferably, the determining the preset number of ancient Chinese keywords with the maximum weight in the target ancient Chinese keywords as the ancient poetry topic words comprises:

when the number of the target ancient Chinese keywords is larger than or equal to a preset number, taking the ancient Chinese keywords with the maximum weight in the ancient Chinese keywords as ancient poem subject terms;

and when the number of the target ancient Chinese keywords is less than the preset number, expanding the target ancient Chinese keywords by using ancient Chinese Word vectors trained in advance by means of Topical Word Embedding technology to obtain the target ancient Chinese keywords with the preset number, and using the target ancient Chinese keywords as ancient poem subject terms.

Preferably, the generating of the target ancient poetry corresponding to the subject term of the ancient poetry by using the pre-trained ancient poetry generating model comprises:

acquiring a pre-trained ancient poetry generating model;

and inputting the subject terms of the ancient poetry into the ancient poetry generating model, and acquiring the target ancient poetry output by the ancient poetry generating model.

ancient poetry generating device, comprising:

the system comprises an input information acquisition unit, a processing unit and a display unit, wherein the input information acquisition unit is used for acquiring multi-modal information input by a user, and the multi-modal information comprises at least two of images, videos, texts and audios;

the modern vocabulary extraction unit is used for respectively extracting keywords from each modal information in the multi-modal information to obtain a modern Chinese keyword corresponding to each modal information as a target modern Chinese keyword;

the modal probability obtaining unit is used for obtaining the occurrence probability of each modal information according to the statistical data of the historical modal information;

the ancient Chinese vocabulary matching unit is used for acquiring an ancient Chinese keyword with the highest relevance with the target modern Chinese keyword as the target ancient Chinese keyword according to a pre-constructed ancient poetry mapping knowledge graph;

the ancient Chinese weight determining unit is used for determining the weight of the target ancient Chinese key words according to the correlation degree between the target ancient Chinese key words and the target modern Chinese key words and the occurrence probability of each modal information;

a poetry theme determining unit, configured to determine a preset number of ancient Chinese keywords with the largest weight in the target ancient Chinese keywords as ancient poetry theme words;

and the target poetry generating unit is used for generating target ancient poetry corresponding to the subject term of the ancient poetry by utilizing an ancient poetry generating model trained in advance.

ancient poetry generating equipment, comprising a processor and a memory;

the memory is used for storing programs;

the processor is used for executing the program stored in the memory so as to realize the ancient poetry generating method.

kinds of computer-readable storage media having stored therein a program that, when executed by a processor, implements the ancient poetry generating method described above.

According to the ancient poetry generating method, the ancient poetry generating device, the ancient poetry generating equipment and the storage medium, the multi-modal information is subjected to keyword extraction, standardized processing of the multi-modal information is achieved, more modal information can be received, more -extensive practical application scenes are supported, the user intention is fully reflected, then a data base is provided for processing of the multi-modal information through probability statistical results of the modal information, the mapping problem between modern Chinese and ancient Chinese is solved by using an ancient poetry mapping knowledge map, finally, the obtained ancient poetry keywords with the preset number are determined as ancient poetry theme words, and target ancient poetry words corresponding to the ancient poetry theme words are generated by using a pre-trained ancient poetry generating model, so that the multi-modal information is fully fused in the ancient poetry generating process, the user intention is more fully embodied, and the ancient poetry generating quality is effectively improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is kinds of flow charts of the method for generating ancient poems according to the embodiment of the present invention;

FIG. 2 is another flowcharts of the method for generating ancient poetry according to the embodiment of the present invention;

FIG. 3 is an exemplary diagram of an ancient poetry mapping knowledge-graph provided by an embodiment of the present invention;

FIG. 4 is a schematic processing flow diagram of an ancient poetry generating system provided by an embodiment of the invention;

fig. 5 is a schematic structural diagram of an ancient poetry generating device provided by the embodiment of the invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only partial embodiments of of the present invention, rather than all embodiments.

The invention provides a technical scheme for generating ancient poetry based on multi-modal information, which can support more modal information as user input information, mainly comprises the normalized processing, the fusion processing and the mapping processing from modern Chinese to ancient Chinese of the multi-modal information, and also provides a subject word expansion scheme which can ensure the conformity of a subject when the subject words are insufficient.

Referring to fig. 1, fig. 1 is a flow chart of methods for generating ancient poems according to an embodiment of the present invention.

As shown in fig. 1, the method for generating ancient poetry of the present embodiment may include:

s101: multi-modal information input by a user is obtained, wherein the multi-modal information comprises at least two of images, videos, texts and audios.

The multi-modal information inputted by the user is an initial material for generating the ancient poetry, and the expression form of the modal information in the multi-modal information may include single-modal information and multi-modal combined information, wherein the separately existing images, videos, texts, audios and the like are all single-modal information, and when different single-modal information are fused with each other at , the multi-modal combined information is a multi-modal combined information, for example, text and image information, text and short video information, audio and image information and the like.

S102: and respectively extracting keywords from each modal information in the multi-modal information to obtain a modern Chinese keyword corresponding to each modal information as a target modern Chinese keyword.

Since the modal information has diversity, it is necessary to normalize the modal information by extracting the modern chinese keywords corresponding to the modal information and using the modern chinese keywords as the normalization result of the modal information.

When performing normalization processing, different modality information needs to be processed, and the modality information mainly related in an actual scene includes images, videos, texts, audios and the like, and mainly includes the following four processing modes a1 to a 4:

a1, when the multi-mode information comprises a text, using a named entity recognition technology to recognize the keywords in the text, and obtaining the modern Chinese keywords corresponding to the text as the target modern Chinese keywords.

For example, for the white language text, named entity recognition is performed to obtain entity keyword information in the white language text and add the entity keyword information to the topic keyword set.

A2, when the multi-mode information comprises audio, firstly, converting the voice in the audio into a voice recognition text by using a voice recognition technology, and then, recognizing the keywords in the voice recognition text by using a named entity recognition technology to obtain the modern Chinese keywords corresponding to the audio as the target modern Chinese keywords.

For example, for audio, the deep learning model may be used to convert the speech in the audio into text, and the subsequent processing is similar to the processing of the white language text, and corresponding entity keyword information is obtained and added to the topic keyword set.

A3, when the multi-mode information includes an image, firstly using an image recognition technology to recognize the content information of the image, and then determining the modern Chinese keywords corresponding to the image according to the content information of the image, wherein the modern Chinese keywords are used as the target modern Chinese keywords.

The content information of the image may include information such as an object and a scene.

For example, for an image, image object and scene recognition is performed, and corresponding modern Chinese keywords and confidence levels are obtained and added to the topic keyword set.

A4, when the multi-mode information comprises a video, firstly identifying a key frame image in the video by using a video understanding technology, then identifying content information of the key frame image by using an image identification technology, and then determining a modern Chinese keyword corresponding to the video according to the content information of the key frame image to be used as a target modern Chinese keyword (or performing content understanding by using other time sequence-based video understanding technologies).

For example, for a video, firstly, key frame recognition is performed, conversion from a dynamic video to a static image is realized, a key frame image and a frame frequency are obtained, then, according to an image processing method, corresponding modern Chinese keywords and confidence coefficients are recognized and added into a topic keyword set.

S103: and acquiring the occurrence probability of each modal information according to the statistical data of the historical modal information.

The probability of the occurrence of each modal information in the historical modal information can be obtained according to the law of large numbers.

S104: and acquiring the ancient Chinese keywords with the highest relevance with the target modern Chinese keywords as the target ancient Chinese keywords according to a pre-constructed ancient poetry mapping knowledge graph.

The method has the advantages that the significance difference gap exists between the ancient Chinese language and the modern Chinese language, and in order to solve the Mapping problem between the ancient Chinese language and the modern Chinese language, the invention constructs a ancient Poetry Mapping Knowledge map (PMKG) in advance, and the Mapping relation between the ancient Chinese keywords and the modern Chinese keywords is provided.

With respect to the ancient poetry mapping knowledge graph, reference may be made to the specific example shown in fig. 3, wherein circles with shading represent modern chinese words and circles with no shading represent ancient chinese words, and the degree of correlation between modern chinese words and ancient chinese words may be calculated using TF-IDF (term frequency-inverse text frequency) technique, it should be noted that fig. 3 shows only examples of the ancient poetry mapping knowledge graph of the present invention, and the ancient poetry mapping knowledge graph provided by the present invention may also cover more vocabulary data and is not limited to the content shown in fig. 3.

According to the ancient poetry mapping knowledge graph established in advance, the ancient Chinese keywords most relevant to the target modern Chinese keywords can be sequentially obtained and used as the target ancient Chinese keywords.

S105: and determining the weight of the target ancient Chinese keywords according to the correlation between the target ancient Chinese keywords and the target modern Chinese keywords and the occurrence probability of each modal information.

In the example, the product of the relevance and the probability corresponding to the target ancient Chinese keyword can be used as the weight of the target ancient Chinese keyword, wherein the probability corresponding to the ancient Chinese keyword is, in effect, the probability of occurrence of the corresponding modal information.

In addition, ancient chinese keywords and their weights may be added to the keyword dictionary.

S106: and determining the preset number of ancient Chinese keywords with the maximum weight in the target ancient Chinese keywords as ancient poem theme words.

The preset number refers to the number of ancient chinese keywords required for generating ancient poems.

S107: and generating a target ancient poetry corresponding to the ancient poetry theme by utilizing a pre-trained ancient poetry generating model.

The ancient poetry generating model can be a Seq2Seq model obtained by training according to ancient poetry theme word samples.

The ancient poetry generating method provided by the embodiment has the advantages that the multi-modal information is subjected to keyword extraction, the multi-modal information is subjected to standardized processing, more modal information can be received, more extensive practical application scenes are supported, the user intention is fully reflected, then a data base is provided for the multi-modal information processing through the probability statistical result of each modal information, the mapping problem between modern Chinese and ancient Chinese is solved by using an ancient poetry mapping knowledge map, finally, the obtained ancient poetry keywords with the preset number are determined as the ancient poetry subject terms, and the target ancient poetry corresponding to the ancient poetry subject terms is generated by using the pre-trained ancient poetry generating model, so that the multi-modal information is fully fused in the ancient poetry generating process, the user intention is more fully embodied, and the ancient poetry generating quality is effectively improved.

Referring to fig. 2, fig. 2 is another flowcharts of the method for generating ancient poetry according to the embodiment of the present invention.

As shown in fig. 2, the method for generating ancient poems according to the embodiment includes:

s201: multimodal information input by a user is obtained.

The multimodal information includes at least two of images, video, text, and audio.

S202: and extracting the target modern Chinese keywords.

And respectively extracting keywords from each modal information in the multi-modal information to obtain a modern Chinese keyword corresponding to each modal information as a target modern Chinese keyword.

Steps S201 to S202 are similar to steps S101 to S102 in the foregoing embodiment , and are not repeated herein.

S203: and calculating the occurrence probability of each modal information.

Specifically, according to the law of large numbers, the probability P (B) of occurrence of the target mode information Bi can be calculated using the following formula_i)：

P(B_i)＝COUNT(B_i)/COUNT(B)，i∈[1,n]，B＝∑B_i；

Wherein, COUNT (B)_i) Representing target modal information B in the multi-modal information_iNumber of occurrences in historical modality informationCounting; count (B) represents the total number of occurrences of all modality information B in the history modality information in the multimodal information. The target modality information B_iIs any modality information in the all modality information B.

S204: and acquiring the target ancient Chinese keywords.

And acquiring the ancient Chinese keywords with the highest relevance with the target modern Chinese keywords as the target ancient Chinese keywords according to a pre-constructed ancient poetry mapping knowledge graph.

Step S204 corresponds to step S104 of the previous embodiment and step , which is not described herein again.

S205: and acquiring the weight of the target ancient Chinese keywords.

Taking the product of the corresponding relevancy and the probability of the target ancient Chinese key words as the weight of the target ancient Chinese key words;

the relevancy corresponding to the target ancient Chinese key words is the relevancy between the target ancient Chinese key words and the target modern Chinese key words; and the probability corresponding to the target ancient Chinese key words is the occurrence probability of the modal information corresponding to the target ancient Chinese key words.

Specifically, before step S205, the method may further include: and determining the probability corresponding to the target ancient Chinese keywords according to the corresponding relation between the modal information and the target modern Chinese keywords, the incidence relation between the target modern Chinese keywords and the target ancient Chinese keywords and the occurrence probability of the modal information. That is, according to the correspondence and the association relationship, the correspondence between the target ancient chinese language keyword and the modal information can be found, and the occurrence probability of the modal information is used as the probability corresponding to the corresponding target ancient chinese language keyword.

S206: judging whether the COUNT (D) is greater than or equal to K, if yes, executing S207; if not, go to step S208.

And judging whether the COUNT (D) is more than or equal to K, namely judging whether the number of the target ancient Chinese keywords is more than or equal to the preset number.

Wherein, count (D) represents the number of the target ancient Chinese keywords, D represents the target ancient Chinese keywords, and K is a preset number. The K value can be set manually or determined automatically according to historical experience, and only the K value is required to meet the vocabulary requirement for generating ancient poems.

S207: and removing redundancy of the target ancient Chinese keywords to obtain the subject terms of the ancient poems.

And taking the ancient Chinese keywords with the maximum weight in the ancient Chinese keywords as ancient poem subject terms.

That is to say, when the number of the target ancient Chinese keywords is greater than or equal to the preset number, the ancient Chinese keywords with the maximum weight in the ancient Chinese keywords are used as the ancient poetry theme words.

In the example, the target ancient Chinese keywords may be sorted according to their weights, and if count (D) is greater than or equal to K, Top-K ancient Chinese keywords in D are selected as ancient poetry topic words, and Top-K represents K Top-ranked words.

Through the step S207, when the ancient poetry theme words are obtained, the redundant target ancient Chinese keywords with lower weight can be effectively filtered, and the redundancy problem of the target ancient Chinese keywords is further solved.

S208: and expanding the target ancient Chinese keywords to obtain ancient poetry theme words.

And expanding the target ancient Chinese keywords by using ancient Chinese Word vectors trained in advance through a topic Word Embedding technology to obtain a preset number of target ancient Chinese keywords as ancient poem topic words.

The topic Word Embedding technology is used for learning Word vectors based on words and topics, so that the generated Word vectors are closer to Word vectors of topics, and the technology can ensure consistency of expanded keywords on the topics.

That is, when the number of the target ancient Chinese keywords is less than the preset number, the target ancient Chinese keywords are expanded by using ancient Chinese Word vectors trained in advance by means of Topical Word Embedding technology to obtain the preset number of target ancient Chinese keywords as ancient poem subject terms.

In the example, the nearest neighbors (count (d) -K) of the obtained target ancient chinese keywords can be found from the ancient chinese Word vectors trained by local Word Embedding, wherein cosine similarity is used as a dimension, so as to expand the target ancient chinese keywords, ensure the theme consistence of the ancient poetry theme words, and further effectively improve the ancient poetry generation quality.

S209: and generating target ancient poems.

And generating a target ancient poetry corresponding to the ancient poetry theme by utilizing a pre-trained ancient poetry generating model.

In the example, the step S209 may specifically include obtaining a pre-trained ancient poetry generating model, inputting the subject terms of the ancient poetry into the ancient poetry generating model, and obtaining target ancient poetry output by the ancient poetry generating model.

Specifically, a Seq2Seq ancient poetry generating model can be trained according to an ancient poetry theme word sample, and then the ancient poetry theme words obtained here are used as subject code input and decoded to generate target ancient poetry.

The method for generating the ancient poetry provided by the embodiment calculates the occurrence probability of each modal information according to a law of large numbers, provides a sufficient data base for processing multi-modal information, respectively executes corresponding redundancy removal processing or expansion processing according to the number of the obtained target ancient Chinese keywords, thereby solving the redundancy problem and the missing problem of the ancient poetry theme words, filters the target ancient Chinese keywords with low weight during the redundancy removal processing, ensures the accuracy of the ancient poetry theme words, and expands the target ancient Chinese keywords by adopting the ancient Chinese word vectors during the expansion processing, thereby ensuring that the expanded ancient Chinese keywords and the target ancient Chinese keywords have higher theme consistency, and effectively improving the quality of the generated ancient poetry.

Referring to fig. 4, fig. 4 is a schematic processing flow diagram of an ancient poetry generating system according to an embodiment of the present invention.

The basic flow of the ancient poetry generating process mainly comprises the following steps: "normalization- > fusion- > mapping- > extension".

As shown in fig. 4, first, multi-modal information is obtained, which includes images, short video, white text and voice.

Secondly, carrying out user intention recognition and standardization processing on the multi-modal information by using an information processing module, wherein the user intention recognition and standardization processing comprises image recognition on an image, image recognition is carried out after key frame recognition is carried out on a short video, named entity recognition is carried out on a white text, and named entity recognition is carried out after text transcription is carried out on voice; by means of the method, the normalized modern Chinese keywords can be obtained and output, and the intention of the user can be reflected accordingly.

Then, the information processing module is used for selecting and organizing the content of the normalized output result, wherein the content comprises keyword fusion and mapping, namely, obtaining the modern Chinese keywords output after normalization, and carrying out ancient Chinese keyword mapping according to the modern Chinese keywords to obtain the ancient Chinese keywords required by generating the ancient poetry, and the content further comprises subject -caused planning, namely, selecting or expanding the ancient Chinese keywords obtained by mapping to obtain the subject words of the ancient poetry.

And finally, generating an ancient poetry corresponding to the ancient poetry theme by using the ancient poetry generating model.

Wherein, when ancient poetry generation model training, can gather ancient poetry data set in advance to carry out the preprocessing process that text washd, text screening, participle, rhythm were drawed to ancient poetry data set, obtain ancient poetry subject matter word sample, then utilize ancient poetry subject matter word sample to train and obtain ancient poetry generation model.

The processing flow of the ancient poetry generating system provided by the embodiment provides a more comprehensive and complete processing flow of ancient poetry generation from a system level, realizes the standardized processing of multi-mode information by extracting keywords from the multi-mode information, can accept more modal information, supports more -wide practical application scenes, provides a data base for the processing of the multi-mode information through the probability statistical result of each modal information, solves the mapping problem between modern Chinese and ancient Chinese through an ancient poetry mapping knowledge map, finally determines a preset number of ancient poetry keywords as ancient poetry theme words, and generates target ancient poetry corresponding to the ancient poetry theme words by utilizing a pre-trained ancient poetry generating model, so that the multi-mode information is fully fused in the ancient poetry generating process, and the quality of the generated ancient poetry is effectively improved.

The embodiment of the invention also provides an ancient poetry generating device, wherein the ancient poetry generating device is used for implementing the ancient poetry generating method provided by the embodiment of the invention, the technical content of the ancient poetry generating device described below can be correspondingly referred to with the technical content of the ancient poetry generating method described above, and the same or similar content is not repeated.

Referring to fig. 5, fig. 5 is a schematic view of structures of an ancient poetry generating device according to an embodiment of the present invention.

As shown in fig. 5, the apparatus includes: the poetry theme recognition method comprises an input information acquisition unit 100, a modern vocabulary extraction unit 200, a modal probability acquisition unit 300, an ancient vocabulary matching unit 400, an ancient weight determination unit 500, a poetry theme determination unit 600 and a target poetry generation unit 700.

The input information acquiring unit 100 is configured to acquire multi-modal information input by a user, where the multi-modal information includes at least two of an image, a video, a text, and an audio.

The expression form of the modal information in the multimodal information comprises single modal information and multimodal combined information.

And a modern vocabulary extracting unit 200, configured to perform keyword extraction on each modal information in the multimodal information, to obtain a modern chinese keyword corresponding to each modal information, as a target modern chinese keyword.

A modal probability obtaining unit 300, configured to obtain occurrence probabilities of the modal information according to the historical modal information statistical data.

And the ancient language vocabulary matching unit 400 is used for acquiring the ancient Chinese keywords with the highest relevance with the target modern Chinese keywords as the target ancient Chinese keywords according to a pre-constructed ancient poetry mapping knowledge graph.

A ancient language weight determining unit 500, configured to determine a weight of the target ancient chinese keyword according to a correlation between the target ancient chinese keyword and the target modern chinese keyword and an occurrence probability of each modal information.

A poetry topic determination unit 600, configured to determine a preset number of ancient Chinese keywords with the largest weight in the target ancient Chinese keywords as ancient poetry topic words.

And the target poetry generating unit 700 is used for generating target ancient poetry corresponding to the subject term of the ancient poetry by utilizing an ancient poetry generating model trained in advance.

In the example, the modern vocabulary extraction unit 200 is specifically configured to:

when the multi-mode information comprises a text, identifying keywords in the text by using a named entity identification technology to obtain modern Chinese keywords corresponding to the text, and using the modern Chinese keywords as target modern Chinese keywords;

when the multi-mode information comprises audio, firstly, converting the voice in the audio into a voice recognition text by using a voice recognition technology, and then, recognizing key words in the voice recognition text by using a named entity recognition technology to obtain modern Chinese key words corresponding to the audio as target modern Chinese key words;

when the multi-mode information comprises an image, firstly identifying the content information of the image by using an image identification technology, and then determining a modern Chinese keyword corresponding to the image as a target modern Chinese keyword according to the content information of the image; the content information of the image may include information such as an object and a scene.

When the multi-mode information comprises a video, a video understanding technology is used for identifying a key frame image in the video, an image identification technology is used for identifying content information of the key frame image, and then a modern Chinese keyword corresponding to the video is determined according to the content information of the key frame image and is used as a target modern Chinese keyword (or other video understanding technologies based on time sequence for content understanding).

In the example, the modality probability obtaining unit 300 is specifically configured to:

the obtaining of the occurrence probability of each modal information according to the statistical data of the historical modal information includes:

the probability of occurrence P (B) of the target mode information Bi is calculated by the following formula_i)：

P(B_i)＝COUNT(B_i)/COUNT(B)，i∈[1,n]，B＝∑B_i；

Wherein, COUNT (B)_i) Representing target modal information B in the multi-modal information_iNumber of occurrences in historical modality information; count (B) represents the total number of occurrences of all modality information B in the history modality information in the multimodal information.

In the example, the ancient language weight determination unit 500 is specifically configured to:

determining the probability corresponding to the target ancient Chinese keywords according to the corresponding relation between the modal information and the target modern Chinese keywords, the incidence relation between the target modern Chinese keywords and the target ancient Chinese keywords and the occurrence probability of the modal information;

and taking the product of the corresponding relevancy and the probability of the target ancient Chinese key words as the weight of the target ancient Chinese key words.

In the example, the verse theme determination unit 600 is specifically configured to:

when the number of the target ancient Chinese keywords is larger than or equal to a preset number, taking the ancient Chinese keywords with the maximum weight in the ancient Chinese keywords as ancient poem subject terms;

and when the number of the target ancient Chinese keywords is less than the preset number, expanding the target ancient Chinese keywords by using ancient Chinese Word vectors trained in advance by means of Topical Word Embedding technology to obtain the target ancient Chinese keywords with the preset number, and using the target ancient Chinese keywords as ancient poem subject terms.

In the example, the target poetry generating unit 700 is specifically configured to:

the method for generating the target ancient poetry corresponding to the ancient poetry theme words by utilizing the pre-trained ancient poetry generating model comprises the following steps:

acquiring a pre-trained ancient poetry generating model;

and inputting the subject terms of the ancient poetry into the ancient poetry generating model, and acquiring the target ancient poetry output by the ancient poetry generating model.

The ancient poetry generating device provided by the embodiment realizes the standardized processing of multi-modal information by extracting keywords from the multi-modal information, can further receive more modal information, supports more extensive practical application scenes, and fully reflects the intention of a user, provides a data base for the processing of the multi-modal information through the probability statistical result of each modal information, solves the mapping problem between modern Chinese and ancient Chinese by using an ancient poetry mapping knowledge map, finally determines the obtained preset number of ancient poetry keywords as the ancient poetry subject terms, and generates the target ancient poetry corresponding to the ancient poetry subject terms by using a pre-trained ancient poetry generating model, thereby fully fusing the multi-modal information in the process of generating the ancient poetry, more comprehensively reflecting the intention of the user, and effectively improving the generating quality of the ancient poetry.

And moreover, during the redundancy removal processing, the target ancient Chinese keywords with low weight are filtered, the accuracy of the ancient poetry theme words is ensured, and the target ancient Chinese keywords are expanded by adopting the ancient Chinese word vectors during the expansion processing, so that the expanded ancient Chinese keywords and the target ancient Chinese keywords have higher theme consistency, and the quality of the generated ancient poetry words is effectively improved.

The ancient poetry generating device provided by the embodiment of the invention comprises a processor and a memory, wherein the input information acquisition unit 100, the modern vocabulary extraction unit 200, the modal probability acquisition unit 300, the ancient vocabulary matching unit 400, the ancient language weight determination unit 500, the poetry theme determination unit 600, the target poetry generating unit 700 and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor comprises a kernel, and the kernel calls corresponding program units from the memory, wherein or more kernels can be set, and the ancient poetry generating method is realized by adjusting kernel parameters.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), including at least memory chips.

An embodiment of the present invention provides computer-readable storage media having stored therein a program that, when executed by a processor, implements the steps of the aforementioned method of generating ancient poems.

The embodiment of the invention provides processors, wherein the processors are used for running a program, and the program runs to execute the steps of the ancient poetry generating method.

The embodiment of the invention provides ancient poetry generating equipment, which comprises a processor, a memory and a program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the ancient poetry generating method.

The device herein may be a server, a PC, a PAD, a mobile phone, etc.

The present application further provides computer program products adapted to execute a program for initializing the steps of the method for generating ancient poetry described previously when executed on a data processing device.

Finally, it should also be noted that, in this document, relational terms such as , , and the like are only used to distinguish entities or operations from another entities or operations without requiring or implying any such actual relationship or order between such entities or operations, and furthermore, the terms "comprise," "include," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises the series elements does not include only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

Based on the understanding that the present disclosure may take the form of a software product, which may be stored on a storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing computer devices (which may be personal computers, servers, or network devices, etc.) to perform the methods described in various embodiments or portions of embodiments of the present disclosure.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.

The principles and embodiments of the present application are described herein using specific examples, which are provided only to help understand the method and the core idea of the present application, and the application scope and the detailed implementation of the method may be changed by persons skilled in the art according to the idea of the present application.