阅读说明：本技术 标题生成方法、装置、电子设备和存储介质 (Title generation method and device, electronic equipment and storage medium ) 是由 朱昆磊 刘佳卉 陈杰 霍小庆 谷伟波 贠挺 于 2019-09-29 设计创作，主要内容包括：本申请公开了标题生成方法、装置、电子设备和存储介质,涉及自然语言处理领域。具体实现方案为：标题生成方法包括：将待处理文本输入语言生成模型,得到待处理文本对应的多个候选标题及其的概率；计算多个候选标题的困惑度；根据多个候选标题的概率及困惑度,生成待处理文本的标题。能够生成困惑度更低、可靠性更高的标题,避免生成的标题语句不通顺或语义不准确。(The application discloses a title generation method, a title generation device, electronic equipment and a storage medium, and relates to the field of natural language processing. The specific implementation scheme is as follows: the title generation method comprises the following steps: inputting a text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles; calculating a perplexity of the plurality of candidate titles; and generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles. The title with lower confusion and higher reliability can be generated, and the generated title sentences are prevented from being not smooth or having inaccurate semantics.)

1. A title generation method, comprising:

inputting a text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles;

calculating a perplexity of the plurality of candidate titles;

and generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles.

2. The title generation method of claim 1, wherein generating the title of the text to be processed according to the probability and the confusion of the candidate titles comprises:

according to preset probability weight and confusion weight, carrying out weighted summation on the probability and the confusion of the candidate titles to obtain scores of the candidate titles;

and determining the candidate title with the highest score as the title of the text to be processed.

3. The title generation method of claim 1, wherein the method further comprises:

inputting the title of the text to be processed into a title availability judgment model;

and identifying whether the title of the text to be processed is available by using the title availability judging model.

4. The title generation method of claim 1, wherein before inputting the text to be processed into the language generation model, further comprising:

obtaining an initialization network structure of a language generation model according to the model for pre-training;

training a language generation model having the initialized network structure using training data.

5. The title generation method of claim 1, wherein the method further comprises:

displaying the title of the text to be processed and prompting a user to input the information of the title;

in the case where a title input by a user is received, the title input by the user is displayed in a predetermined title display area.

6. The title generation method of claim 1, wherein the language generation model is a sequence-to-sequence model; inputting a text to be processed into a language generation model, and obtaining a plurality of candidate titles corresponding to the text to be processed and the probability thereof, wherein the method comprises the following steps:

inputting a text to be processed into a sequence model encoder to obtain an intermediate semantic vector;

inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences;

determining whether the word sequence of the current time step is connected with an end symbol or not according to the word sequence of the current time step and the decoder;

under the condition that the word sequence of the current time step has no connection end symbol, grouping a plurality of word sequences of the current time step;

determining a connecting word of a word sequence according to the scores of the words in the word bank by adopting a cluster search algorithm; wherein if the term appears in other groups, the score of the term is decreased;

updating a plurality of word sequences and the probability of the word sequences at the current time step according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

7. The title generation method of claim 3, wherein the title availability discrimination model comprises a binary model; before the title of the text to be processed is input into a title availability judging model, the method further comprises the following steps:

identifying grammatical components of the title of the text to be processed;

dividing the title of the text to be processed into a plurality of words according to the grammar components;

removing partial words in the title of the text to be processed to obtain an incomplete title;

and taking the title of the text to be processed as a positive example, and taking the incomplete title as a negative example, and training the binary classification model.

8. A title generation apparatus, comprising:

the acquisition module is used for inputting a text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles;

a calculation module for calculating a perplexity of the plurality of candidate titles;

and the generating module is used for generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles.

9. The title generation apparatus of claim 8, wherein the generation module comprises:

the summing unit is used for weighting and summing the probability and the confusion of the candidate titles according to preset probability weight and confusion weight to obtain scores of the candidate titles;

and the determining unit is used for determining the candidate title with the highest score as the title of the text to be processed.

10. The title generation apparatus of claim 8, wherein said apparatus further comprises:

the input module is used for inputting the title of the text to be processed into a title availability judgment model;

and the first identification module is used for identifying whether the title of the text to be processed is available or not by using the title availability judgment model.

11. The title generation apparatus of claim 8, wherein said apparatus further comprises:

the pre-training module is used for obtaining an initialization network structure of the language generation model according to the model used for pre-training;

and the first training module is used for training the language generation model with the initialization network structure by adopting training data.

12. The title generation apparatus of claim 8, wherein said language generation model is a sequence-to-sequence model; the acquisition module comprises:

the encoding unit is used for inputting a text to be processed into an encoder of the sequence model to obtain an intermediate semantic vector;

the decoding unit is used for inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences;

an ending judgment unit, configured to determine whether an ending symbol is concatenated after the word sequence of the current time step according to the word sequence of the current time step and the decoder;

the grouping unit is used for grouping the word sequences of the current time step under the condition that the word sequences of the current time step do not have the connection end characters;

the selecting unit is used for determining a connecting word of the word sequence according to the scores of the words in the word bank by adopting a cluster searching algorithm; wherein if the term appears in other groups, the score of the term is decreased;

the output unit is used for updating a plurality of word sequences of the current time step and the probability thereof according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

13. The title generation apparatus of claim 10, wherein said title availability discrimination model comprises a binary model; the device further comprises:

the second identification module is used for identifying grammatical components of the title of the text to be processed;

the dividing module is used for dividing the title of the text to be processed into a plurality of words according to the grammar components;

the removing module is used for removing partial words in the title of the text to be processed to obtain an incomplete title;

and the second training module is used for training the two classification models by taking the titles of the texts to be processed as positive examples and taking the incomplete titles as negative examples.

14. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

15. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.

Technical Field

The present application relates to the field of data processing, and more particularly, to the field of natural language processing.

Background

With the rise of network platforms, more and more people can write articles to be published on the network. When an author issues a document on a network platform, the problem of difficulty in writing a title often exists. Titles with poor quality not only affect the distribution of articles and user profits, but also affect the ecological quality of the whole content. At present, the method for automatically generating the title has the defects of insufficient smoothness of sentences and inaccurate semantics.

Disclosure of Invention

The embodiment of the application provides a title generation method, a title generation device, electronic equipment and a storage medium, which are used for solving the problems in the related art, and the technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a title generating method, including:

inputting a text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles;

calculating a perplexity of the plurality of candidate titles;

and generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles.

Through the technical scheme, in the title generation process, the title of the text to be processed is determined by combining the probability and the confusion degree of the candidate title. Compared with the method that the title of the text to be processed is generated only according to the probability, the technical scheme can generate the title with lower confusion degree and higher reliability, and the generated title sentence is prevented from being unsmooth or inaccurate in semantics.

In one embodiment, generating a title of a text to be processed according to the probability and the confusion of a plurality of candidate titles comprises:

according to the preset probability weight and the preset confusion weight, carrying out weighted summation on the probability and the confusion of the candidate titles to obtain scores of the candidate titles;

and determining the candidate title with the highest score as the title of the text to be processed.

According to the technical scheme, the probability and the perplexity of the candidate titles are weighted and summed to obtain the score, and the score is used as a basis for determining the title of the text to be processed from the candidate titles. By combining the probability and the confusion degree, the basis for generating the title can be objectively and directly obtained, the adjustability is realized, and the generation effect can be optimized by reasonably setting the threshold value.

In one embodiment, the method further comprises:

inputting the title of the text to be processed into a title availability judgment model;

whether the title of the text to be processed is available is identified by using a title availability discrimination model.

By the technical scheme, the title usability judging model is set after the title is generated, whether the title is usable or not can be identified, the title with quality problems of grammar, semantics or logic and the like is identified, and text quality reduction caused by generation of wrong titles is avoided.

In one embodiment, before inputting the text to be processed into the language generation model, the method further comprises:

obtaining an initialization network structure of a language generation model according to the model for pre-training;

training a language generative model having an initialized network structure using training data.

Through the technical scheme, the pre-training of the language generation model is realized, the initialization network structure of the language generation model is obtained, the language generation model learns the basic grammar knowledge and the expression mode of the title language, and then the language generation model with the initialization network structure is trained, so that the generated title is more in line with the natural language requirement, sentences are smoother, and semantics are more accurate.

In one embodiment, the method further comprises:

displaying the title of the text to be processed and prompting a user to input the information of the title;

in the case where a title input by a user is received, the title input by the user is displayed in a predetermined title display area.

Through the technical scheme, after the language generation model outputs the title of the text to be processed, the title can be provided for a user to refer, the user inputs the title, and if the user inputs the title, the title input by the user is used and displayed in the preset title display area. In this way, incorrect titles can be corrected in time by interacting with the user, and the wrong titles can be prevented from being displayed in the preset title display area.

In one embodiment, the language generation model is a sequence-to-sequence model; inputting a text to be processed into a language generation model, and obtaining a plurality of candidate titles corresponding to the text to be processed and the probability thereof, wherein the method comprises the following steps:

inputting a text to be processed into a sequence model encoder to obtain an intermediate semantic vector;

inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences;

determining whether the word sequence of the current time step is connected with an end symbol or not according to the word sequence of the current time step and the decoder;

under the condition that the word sequence of the current time step has no connection end symbol, grouping a plurality of word sequences of the current time step;

determining a connecting word of a word sequence according to the scores of the words in the word bank by adopting a cluster search algorithm; wherein if the term appears in other groups, the score of the term is decreased;

updating a plurality of word sequences and the probability of the word sequences at the current time step according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

Through the technical scheme, the word sequences are grouped in the Beam Search (Beam Search) process, and when the connecting words of the word sequences are selected at a certain time step, if the candidate connecting words appear in other groups, the word is subjected to division punishment. Thereby realizing diversity of candidate titles so that the generated title is optimal in a wider range.

In one embodiment, the title availability discrimination model comprises a two-classification model; before the title of the text to be processed is input into the title availability judging model, the method further comprises the following steps:

identifying grammatical components of a title of a text to be processed;

dividing the title of the text to be processed into a plurality of words according to the grammatical components;

removing partial words in the title of the text to be processed to obtain an incomplete title;

and taking the title of the text to be processed as a positive example and taking the incomplete title as a negative example, and training the binary model.

By the technical scheme, the incomplete title is used as a negative example, and the complete title is used as a positive example to train the two-classification model. Therefore, the problem of whether the semantic expression is complete or not can be accurately identified by the binary classification model. The accuracy of whether the title is available or not is improved by the title availability judging model.

In a second aspect, an embodiment of the present application further provides a title generating apparatus, including:

the acquisition module is used for inputting the text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles;

a calculation module for calculating a perplexity of the plurality of candidate titles;

and the generating module is used for generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles.

In one embodiment, the generating module comprises:

the summing unit is used for weighting and summing the probability and the confusion of the candidate titles according to the preset probability weight and the confusion weight to obtain the scores of the candidate titles;

and the determining unit is used for determining the candidate title with the highest score as the title of the text to be processed.

In one embodiment, the apparatus further comprises:

the input module is used for inputting the title of the text to be processed into the title availability judgment model;

the first identification module is used for identifying whether the title of the text to be processed is available or not by using the title availability judgment model.

In one embodiment, the apparatus further comprises:

the pre-training module is used for obtaining an initialization network structure of the language generation model according to the model used for pre-training;

the first training module is used for training the language generation model with the initialized network structure by adopting the training data.

In one embodiment, the language generation model is a sequence-to-sequence model; the acquisition module comprises:

the encoding unit is used for inputting a text to be processed into an encoder of the sequence model to obtain an intermediate semantic vector;

the decoding unit is used for inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences;

an ending judgment unit, configured to determine whether an ending symbol is concatenated after the word sequence of the current time step according to the word sequence of the current time step and the decoder;

the grouping unit is used for grouping the word sequences of the current time step under the condition that the word sequences of the current time step do not have the connection end characters;

the selecting unit is used for determining a connecting word of the word sequence according to the scores of the words in the word bank by adopting a cluster searching algorithm; wherein if the term appears in other groups, the score of the term is decreased;

the output unit is used for updating a plurality of word sequences of the current time step and the probability thereof according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

In one embodiment, the title availability discrimination model comprises a two-classification model; the device still includes:

the second identification module is used for identifying grammatical components of the title of the text to be processed;

the dividing module is used for dividing the title of the text to be processed into a plurality of words according to the grammatical composition;

the removing module is used for removing partial words in the title of the text to be processed to obtain an incomplete title;

and the second training module is used for training the two classification models by taking the titles of the texts to be processed as positive examples and taking the incomplete titles as negative examples.

In a third aspect, an embodiment of the present application further provides an electronic device, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method provided by any of the embodiments of the present application.

In a fourth aspect, embodiments of the present application further provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method provided in any of the embodiments of the present application.

One embodiment in the above application has the following advantages or benefits: in the title generation process, the title of the text to be processed is determined according to the probability and the confusion degree of the candidate title. Compared with the method that the title of the text to be processed is generated only according to the probability, the technical scheme can generate the title with lower confusion degree and higher reliability, and the generated title sentence is prevented from being unsmooth or inaccurate in semantics.

Other effects of the above-described alternative will be described below with reference to specific embodiments.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present application;

FIG. 2 is a schematic diagram according to a second embodiment of the present application;

FIG. 3 is a schematic illustration according to a third embodiment of the present application;

FIG. 4 is a schematic illustration according to a fourth embodiment of the present application;

FIG. 5 is a schematic illustration according to a fifth embodiment of the present application;

FIG. 6 is a schematic illustration according to a sixth embodiment of the present application;

FIG. 7 is a schematic illustration according to a seventh embodiment of the present application;

FIG. 8 is a schematic illustration according to an eighth embodiment of the present application;

fig. 9 is a block diagram of an electronic device for implementing the title generation method according to the embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Generating the headlines may be generated based on template mining or may be accomplished using a language generation model. For example, the common title forms are collected to form a template, and the title of the text is formed by extracting the text keywords and filling the slots in the template. This approach has insufficient richness in titles. As another example, a Sequence to Sequence (seq2seq) model is used to generate a title for the text based on the input text to be processed. The title generated in this way may have the problems of discordance of sentences, grammar error, insufficient diversity, no logical relationship between fragments or incomplete semantic expression.

Based on this, the embodiment of the present application provides a title generation method. Illustratively, the title generation method may be triggered to be executed in various scenarios. For example, execution is triggered when a text edit box in a text publication platform is out of focus. As another example, the execution is triggered when the number of text words in the text edit box is greater than a set threshold.

As an exemplary implementation manner, as shown in fig. 1, an embodiment of the present application provides a title generation method, including:

step S101, inputting a text to be processed into a language generation model to obtain a plurality of candidate titles corresponding to the text to be processed and the probability of the candidate titles.

In the embodiment of the present application, the language generation model may include any model capable of generating some candidate texts and probabilities thereof according to the text to be processed, and the candidate texts may be candidate titles of the text to be processed. For example, the language generation model may be a seq2seq model, i.e., a sequence-to-sequence model. The seq2seq model comprises an encoder (encoder) and a decoder (decoder), the text to be processed is input into the encoder, and the encoder outputs an intermediate semantic vector as an input of the decoder. The decoder may search for and output possible word sequences using a variety of algorithms. For example, greedy search algorithm or bundle search algorithm, etc. Illustratively, a plurality of word sequences may be output as a plurality of candidate headings using a bundle search algorithm.

Step S102, calculating the confusion degree of a plurality of candidate titles.

In natural language processing, Perplexity (PPL) is used to indicate the certainty of the model output result. It will be appreciated that here, the degree of confusion is used to indicate the accuracy of the probability. Illustratively, the degree of confusion may be calculated using a statistical language model. For example, the confusion is calculated using a KenLM model or SRILM model.

And step S103, generating the title of the text to be processed according to the probability and the confusion degree of the candidate titles.

Through the technical scheme, in the title generation process, the title of the text to be processed is determined by combining the probability and the confusion degree of the candidate title. Compared with the method that the title of the text to be processed is generated only according to the probability, the technical scheme can generate the title with lower confusion degree and higher reliability, and the generated title sentence is prevented from being unsmooth or inaccurate in semantics.

As an exemplary embodiment, the step S103 of generating the caption of the text to be processed according to the probability and the confusion of the plurality of candidate captions includes:

according to the preset probability weight and the preset confusion weight, carrying out weighted summation on the probability and the confusion of the candidate titles to obtain scores of the candidate titles;

and determining the candidate title with the highest score as the title of the text to be processed.

Illustratively, the confusion weight is a negative value. The higher the confusion, the lower the score of the candidate title.

According to the technical scheme, the probability and the perplexity of the candidate titles are weighted and summed to obtain the score, and the score is used as a basis for determining the title of the text to be processed from the candidate titles. By combining the probability and the confusion degree, the basis for generating the title can be objectively and directly obtained, the adjustability is realized, and the generation effect can be optimized by reasonably setting the threshold value.

In some embodiments, as shown in fig. 2, the method may further comprise:

step S201, inputting the title of the text to be processed into a title availability judgment model.

Step S202, whether the title of the text to be processed is available is identified by using a title availability judging model.

The title availability discrimination model may include one or more of a statistical language model, a neural network language model, or a binary model. The title availability discrimination model can identify whether a title is available by identifying whether the title sentence is smooth, whether the semantic expression is complete, and whether logic exists between title fragments. Whether the title of the text to be processed is available is identified by using a title availability discrimination model, as follows:

the first way, the title availability discrimination model may include a statistical language model. Statistical language models may make use of statistics to determine whether a title is uncommon. For example, some indicator evaluating the generated result is calculated. These indicators may include similarity, confusion, and the like. According to the indexes, whether the sentence is smooth or not and whether the semantic meaning is accurate or not can be identified. Thereby identifying whether a title is available. The statistical language model may be, for example, a 5-gramm model.

In a second mode, the title availability discrimination model may include a neural network language model. The neural network language model may use characters as input to determine whether an OOV (Out of vocubulary) problem occurs. Thereby identifying whether a title is available.

In a third mode, the title availability discrimination model may include a binary model. The classification model may classify the inputted titles into usable titles and unusable titles through training.

In a fourth mode, the title availability discrimination model may include BERT. Because the prediction between the upper sentence and the lower sentence exists during BERT training, the produced title is divided into a plurality of different sentences according to punctuation, and whether the sentence keys have logical relations can be identified by the BERT. Thereby identifying whether a title is available.

By the technical scheme, the title usability judging model is set after the title is generated, whether the title is usable or not can be identified, the title with quality problems of grammar, semantics or logic and the like is identified, and text quality reduction caused by generation of wrong titles is avoided.

Upon identifying whether a title is available, the identification result may be presented to a user editing the text, and the user may make title modifications. Through the participation of interactive users, the problem that the generated title grammar is incorrect can be corrected in time.

As a specific example, the method provided in the embodiment of the present application further includes:

displaying the title of the text to be processed and prompting a user to input the information of the title;

in the case where a title input by a user is received, the title input by the user is displayed in a predetermined title display area.

For example, the information for prompting the user to input the title may be a specific text prompt information, an input box, or: in the display area of the title of the text to be processed, whether the title is editable or not is indicated by a specific identifier (such as color, font format, pointer pattern and the like), so that the user is prompted to input the title. The title of the text to be processed may be displayed and the user may be prompted to enter the title only when the title is recognized as not being available, or may be displayed when the title of the text to be processed is generated regardless of whether it is available. The title of the text to be processed and information prompting the user to input the title may be displayed together with the recognition result of whether the title is available.

Through the technical scheme, after the language generation model outputs the title of the text to be processed, the title can be provided for a user to refer, the user inputs the title, and if the user inputs the title, the title input by the user is used and displayed in the preset title display area. In this way, incorrect titles can be corrected in time by interacting with the user, and the wrong titles can be prevented from being displayed in the preset title display area.

As a specific example, the title availability discrimination model may include a binary model. Referring to fig. 3, before the title of the text to be processed is input into the title availability judging model, the method further includes:

and step S301, identifying grammar components of the title of the text to be processed.

And step S302, dividing the title of the text to be processed into a plurality of words according to the grammar components.

For example, the components are identified and the words are divided according to the dependency syntax and the part-of-speech characteristics.

And step S303, removing partial words in the title of the text to be processed to obtain an incomplete title.

For example, an incomplete title may be constructed by culling one or more words in the title in a random manner. A plurality of incomplete headers may be constructed.

And step S304, taking the title of the text to be processed as a positive example and taking the incomplete title as a negative example, and training a binary classification model.

By the technical scheme, the incomplete title is used as a negative example, and the complete title is used as a positive example to train the two-classification model. The binary classification model can accurately identify whether the semantic expression is complete or not, and the accuracy of judging whether the title is available or not by the title availability identification model is improved.

Similarly, different segments of the title or segments in the text to be processed can be freely combined to construct a negative example for training a model, and the model is used for identifying whether logical or semantic association exists between the title segments.

As an exemplary embodiment, as shown in fig. 4, before the generating a model of the text input language to be processed, the method may further include:

s401, obtaining an initialization network structure of a language generation model according to a model used for pre-training;

and S402, training a language generation model with an initialized network structure by adopting training data.

Illustratively, the pre-trained model may include a BERT (Bidirectional encoder representation of a transformer) model. By the technical scheme, the pre-training of the language generation model is realized by using the BERT model, and the initialization network structure of the language generation model is obtained. BERT masks partial title words in the encoding part and predicts the title words according to the context in the decoding part, so that the language generating model learns the basic grammar knowledge and the expression mode of the title language through pre-training. The initial network structure is obtained by using BERT pre-training, and then the language generation model with the initial network structure is trained, so that the generated title is more in line with the requirements of natural language, sentences are smoother, and semantics are more accurate.

Illustratively, the language generation model may be a sequence-to-sequence (seq2seq) model. The decoder (decoder) in the seq2seq model may be implemented using a beam search algorithm. In a specific example, the step S101 of inputting the text to be processed into the language generation model to obtain a plurality of candidate headings corresponding to the text to be processed and probabilities thereof may include:

1. and inputting the text to be processed into a sequence of an encoder of the sequence model to obtain an intermediate semantic vector.

2. And inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences. Here, the probabilities of each word in the word stock as the initial word may be determined according to the intermediate semantic vector and the parameters of the decoder, and the N words with the highest probabilities may be selected from the probabilities. The N words are a plurality of word sequences of the initial time step; n may be the beam width (beam size) of the beam search.

3. And determining whether the ending symbol is connected after the word sequence of the current time step according to the word sequence of the current time step and the decoder. Here, based on the word sequence of the current time step and the parameters of the decoder, it may be determined whether the word sequence of the current time step is followed by an end-of-line. One or more word sequences may be output if the one or more word sequences are concatenated with the terminator.

4. Under the condition that the word sequence of the current time step has no connection end symbol, grouping a plurality of word sequences of the current time step;

5. determining a connecting word of a word sequence according to the scores of the words in the word bank by adopting a cluster search algorithm; wherein the score of the term is decreased if the term has appeared in other groups.

Here, by way of example, may include: generating a plurality of candidate connecting words of the word sequence of the current time step according to the word sequence of the current time step, a decoder and a cluster searching algorithm; if the candidate connecting words of the word sequence of the current time step appear in other groups, reducing the scores of the candidate connecting words; and selecting the connecting words of each word sequence according to the scores of the candidate connecting words. Specifically, the probability that each word in the word library is linked behind the word sequence can be determined according to the word sequence of the current time step and the decoder, and a plurality of words with the highest probability are extracted as candidate linked words. Wherein, the number of candidate connecting words behind a word sequence can be determined by the beam width (beam size) of the beam search algorithm. Then, a plurality of candidate conjunctions of a word sequence are scored. The basis for scoring may include the probability of this word and the similarity between word sequences, specifically, if some candidate adaptor of a certain word sequence appears in other groups, the penalty is reduced. The other groups herein refer to groups other than the group in which the word sequence is located. The score of the candidate conjunct is determined according to the probability of the candidate conjunct and whether the candidate conjunct appears in other groups, and the conjunct of the word sequence can be selected according to the score, for example, the candidate conjunct with the largest score is selected as the conjunct of the word sequence.

6. Updating a plurality of word sequences and the probability of the word sequences at the current time step according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

Here, the word sequence of the previous time step and its conjunction will constitute the word sequence of the new time step. And determining the probability of the word sequence of the current time step according to the probability of the word sequence of the previous time step and the probability or the score of the connecting word. And at each time step, determining a connecting word of the word sequence according to the steps 3-5 to obtain the word sequence of the next time step, and outputting the word sequence as a candidate title until an end symbol is met.

Through the technical scheme, the word sequences are grouped in the Beam Search (Beam Search) process, and when the connecting words of the word sequences are selected at a certain time step, if the candidate connecting words appear in other groups, the word is subjected to division punishment. Thereby realizing diversity of candidate titles so that the generated title is optimal in a wider range.

In some embodiments, on the basis of the language generation model, for example, on the basis of the seq2seq model, different mechanisms can be added to solve some technical problems:

example one, a Pointer-generator mechanism may be added to solve the OOV problem and the problem that the generated title is not accurate for the detailed description of the text.

Example two, a coverage mechanism or intra-attention mechanism may be added to solve the problem of overlapping words in the title.

Third, a Learning method of operator-critic of DDQN (Double Deep Q-Learning) in Q-Learning may be adopted, for example, a Q2seq generation model is used as an operator model, a Q-net (Q network) model is used as a critic model, and a Q value for selecting each word is obtained at each time step according to a loss function of the operator. The method solves the problems that training evaluation indexes in a seq2seq model are inconsistent with the inference time, and a whole title is unavailable when a word generated at one time step in the decoder process is wrong.

The language generation model disclosed by the embodiment of the application is integrated with various generation mechanisms, and the availability and diversity of the generated titles are improved.

An embodiment of the present application further provides a title generating apparatus, as shown in fig. 5, the apparatus 500 includes:

the obtaining module 501 inputs a text to be processed into a language generation model, and obtains a plurality of candidate titles corresponding to the text to be processed and probabilities thereof;

a calculating module 502 for calculating a perplexity of the plurality of candidate titles;

and a generating module 503, configured to generate a title of the text to be processed according to the probabilities and the puzzles of the multiple candidate titles.

In one embodiment, the generating module 503 includes:

the summing unit is used for weighting and summing the probability and the confusion of the candidate titles according to the preset probability weight and the confusion weight to obtain the scores of the candidate titles;

and the determining unit is used for determining the candidate title with the highest score as the title of the text to be processed.

In one embodiment, as shown in fig. 6, the apparatus 500 further comprises:

an input module 601, configured to input a title of a text to be processed into a title availability discrimination model;

a first identification module 602, configured to identify whether a title of the text to be processed is available using a title availability discrimination model.

In one embodiment, as shown in fig. 7, the apparatus 500 further comprises:

a pre-training module 701, configured to obtain an initialized network structure of a language generation model according to a model used for pre-training;

a first training module 702 is configured to train a language generation model having an initialized network structure using training data.

In one embodiment, the apparatus 500 further comprises:

the first display module is used for displaying the title of the text to be processed and prompting the user to input the information of the title;

and the second display module is used for displaying the title input by the user in a preset title display area under the condition of receiving the title input by the user.

In one embodiment, the language generation model is a sequence-to-sequence model;

the obtaining module 501 includes:

the encoding unit is used for inputting a text to be processed into an encoder of the sequence model to obtain an intermediate semantic vector;

the decoding unit is used for inputting the intermediate semantic vector into a decoder of a sequence model to obtain a plurality of word sequences of an initial time step and the probability of the word sequences;

an ending judgment unit, configured to determine whether an ending symbol is concatenated after the word sequence of the current time step according to the word sequence of the current time step and the decoder;

the grouping unit is used for grouping the word sequences of the current time step under the condition that the word sequences of the current time step do not have the connection end characters;

the selecting unit is used for determining a connecting word of the word sequence according to the scores of the words in the word bank by adopting a cluster searching algorithm; wherein if the term appears in other groups, the score of the term is decreased;

the output unit is used for updating a plurality of word sequences of the current time step and the probability thereof according to the connecting words; and returning to the step of determining whether the ending character is connected after the word sequence of the current time step, and outputting each word sequence as a plurality of candidate titles corresponding to the text to be processed until the ending character is connected after each word sequence.

In one embodiment, the title availability discrimination model comprises a two-classification model; as shown in fig. 8, the apparatus 500 further comprises:

a second recognition module 801, configured to recognize a syntax element of a title of the text to be processed;

a dividing module 802, configured to divide a title of a text to be processed into multiple words according to syntax components;

the removing module 803 is configured to remove a part of words in the title of the text to be processed to obtain an incomplete title;

and the second training module 804 is used for training the two classification models by taking the titles of the texts to be processed as positive examples and taking the incomplete titles as negative examples.

The title generation device provided by the embodiment of the application can realize the title generation method provided by any embodiment of the application, and has corresponding beneficial effects.

As shown in fig. 9, it is a block diagram of an electronic device according to the title generation method of the embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.

As shown in fig. 9, the electronic apparatus includes: one or more processors 901, memory 902, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display Graphical information for a Graphical User Interface (GUI) on an external input/output device, such as a display device coupled to the Interface. In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 9 illustrates an example of a processor 901.

Memory 902 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the title generation method provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to execute the title generation method provided by the present application.

The memory 902, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules (e.g., the obtaining module 501, the calculating module 502, and the generating module 503 shown in fig. 5) corresponding to the title generating method in the embodiments of the present application. The processor 901 executes various functional applications of the server and data processing, i.e., realizes the title generation method in the above-described method embodiments, by running non-transitory software programs, instructions, and modules stored in the memory 902.

The memory 902 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the electronic device of the title generation method, and the like. Further, the memory 902 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 902 may optionally include a memory remotely disposed from the processor 901, and these remote memories may be connected to an electronic device of the title generation method through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device of the title generation method may further include: an input device 903 and an output device 904. The processor 901, the memory 902, the input device 903 and the output device 904 may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example.

The input device 903 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus of the title generation method, such as an input device of a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or the like. The output devices 904 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The Display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) Display, and a plasma Display. In some implementations, the display device can be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, Integrated circuitry, Application Specific Integrated Circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

According to the technical scheme of the embodiment of the application, in the title generation process, the title of the text to be processed is determined by combining the probability and the confusion degree of the candidate title. Compared with the method that the title of the text to be processed is generated only according to the probability, the technical scheme can generate the title with lower confusion degree and higher reliability, and the generated title sentence is prevented from being unsmooth or inaccurate in semantics.

In one embodiment, the probability and the confusion of the candidate titles are weighted and summed to obtain a score, and the score is used as a basis for determining the title of the text to be processed from the candidate titles. By combining the probability and the confusion degree, the basis for generating the title can be objectively and directly obtained, the adjustability is realized, and the generation effect can be optimized by reasonably setting the threshold value.

In one embodiment, a title usability judgment model is set after a title is generated, whether the title is usable or not can be identified, the title with quality problems of grammar, semantics or logic and the like is identified, and text quality reduction caused by generation of an error title is avoided.

In one embodiment, the pre-training of the language generating model is implemented by using a BERT (Bidirectional Encoder representation of transformer) model, so as to obtain an initialized network structure of the language generating model. BERT masks partial title words in the encoding part and predicts the title words according to the context in the decoding part, so that the language generating model learns the basic grammar knowledge and the expression mode of the title language through pre-training. The initial network structure is obtained by using BERT pre-training, and then the language generation model with the initial network structure is trained, so that the generated title is more in line with the requirements of natural language, sentences are smoother, and semantics are more accurate.

In one embodiment, word sequences are grouped during a Beam Search (Beam Search), and when an adapter of a word sequence is selected at a certain time step, if a candidate adapter appears in other groups, a subtractive penalty is applied to the word. Thereby realizing diversity of candidate titles so that the generated title is optimal in a wider range.

In one embodiment, the binary model is trained with the incomplete title as a negative example and the complete title as a positive example. The binary classification model can accurately identify whether semantic expression is complete. The accuracy of whether the title is available or not is improved by the title availability judging model.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.