阅读说明：本技术 文档分割方法及装置 (Document segmentation method and device ) 是由 毛红保 于 2020-06-04 设计创作，主要内容包括：本发明实施例提供了一种文档分割方法及装置,所述方法首先获取待分割文档；然后以所述待分割文档作为个体,所述待分割文档中的分割点作为所述个体的基因,基于遗传算法,对所述待分割文档进行分割,确定所述待分割文档中的分割点位置。通过分割点的长度均匀性参数和语义关联性参数确定每个个体的每个基因的质量,同时考虑了分割点的长度均匀性以及语义关联性,使得分割点两侧的片段长度均匀且分割点两侧的片段之间的语义关联性小。解决了现有技术中的分割方法无法保证分割得到的片段的长度均匀的技术问题,为任务的平均分配提供了极有利的理论基础。(The embodiment of the invention provides a document segmentation method and a document segmentation device, wherein the method comprises the steps of firstly obtaining a document to be segmented; and then taking the document to be segmented as an individual, taking the segmentation point in the document to be segmented as the gene of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the position of the segmentation point in the document to be segmented. The quality of each gene of each individual is determined through the length uniformity parameter and the semantic relevance parameter of the segmentation point, and the length uniformity and the semantic relevance of the segmentation point are considered, so that the lengths of the fragments on two sides of the segmentation point are uniform, and the semantic relevance between the fragments on two sides of the segmentation point is small. The technical problem that the segmentation method in the prior art cannot guarantee the uniform length of the segments obtained by segmentation is solved, and a very favorable theoretical basis is provided for the average distribution of tasks.)

1. A method of document segmentation, comprising:

s1, obtaining a document to be segmented;

s2, taking the document to be segmented containing a specified number of segmentation points as an individual, taking the segmentation points in the document to be segmented as genes of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the positions of the segmentation points in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

2. The document segmentation method according to claim 1, wherein the segmenting the document to be segmented based on a genetic algorithm specifically comprises:

s21, determining an initial individual;

s22, calculating the mass of each gene;

s23, based on the quality of each gene, executing genetic operation to obtain a plurality of new individuals of the current generation;

s24, calculating the individual fitness of each new individual based on the quality of each gene of each new individual, determining the optimal individual of the current generation based on the individual fitness, and returning the optimal individual of the current generation to S22 to continue execution until the difference between the individual fitness of the optimal individual of the current generation and the individual fitness of the optimal individual of the previous generation is smaller than a preset threshold;

and S25, determining the position of the segmentation point in the document to be segmented based on the position of each gene in the current generation of optimal individuals.

3. The document segmentation method according to claim 2, wherein the S21 specifically includes:

s211, determining the length of a segment containing a plurality of continuous paragraphs by taking the initial paragraph of the document to be segmented as a starting point;

s212, if the length of the segment is judged to be less than or equal to the preset length, and a next subsequent segment is added in the segment and lags behind, and the length of the obtained new segment is greater than the preset length, a segmentation point exists at the end of the last segment in the segment or at the end of the subsequent segment;

s213, determining the length of the segment containing a plurality of continuous paragraphs by taking the segmentation points as starting points, and returning to S212 to continue execution until the specified number of segmentation points are determined, so as to obtain the initial individual in the genetic algorithm.

4. The document segmentation method according to claim 2, wherein the genetic manipulation includes genetic adjustment for adjusting the position of a gene whose genetic quality satisfies a first preset condition in the initial individual and genetic variation for adjusting the position of a gene whose genetic quality satisfies a second preset condition in the initial individual.

5. The document segmentation method according to claim 2, wherein the calculating of the individual fitness of each new individual based on the quality of each gene of each new individual in S23 specifically includes:

for each new individual, calculating the sum of the qualities of all genes in the new individual based on the quality of each gene of the new individual, and taking the sum of all the qualities as the individual fitness of the new individual.

6. The document segmentation method according to any one of claims 1 to 5, wherein the length uniformity parameter is determined by:

for each segmentation point of the individual, calculating the length difference between the segments on the left side and the right side of the segmentation point;

and carrying out normalization processing on the length difference value to obtain the length uniformity parameter of the segmentation point.

7. The document segmentation method according to claim 6, wherein the quality of each gene of the individual is determined by:

calculating the difference value between the length uniformity parameter of 1 and each segmentation point;

and calculating the product of the difference value and the semantic relevance parameter, and taking the product as the quality score of the gene corresponding to each segmentation point.

8. A document segmentation apparatus, comprising:

the acquisition module is used for acquiring a document to be segmented;

the segmentation module is used for taking the document to be segmented containing a specified number of segmentation points as an individual, taking the segmentation points in the document to be segmented as genes of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the positions of the segmentation points in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the document segmentation method according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer readable storage medium, having stored thereon a computer program, characterized in that the computer program, when being executed by a processor, is adapted to carry out the steps of the document segmentation method according to any one of claims 1 to 7.

Technical Field

The present invention relates to the field of document processing technologies, and in particular, to a document segmentation method and apparatus.

Background

At present, there is a clear need in some business scenarios to segment a long document into consecutive document segments. For example, in the language service industry, a long document needs to be segmented and then translated by multiple translators simultaneously, so as to complete the translation task quickly.

Segmenting a document typically implies the following constraints: (1) the number of segments to segment, N, is specified and the cut point is at the end of the paragraph to ensure that the integrity of the paragraph is not violated; (2) the lengths of the divided segments are relatively uniform, so that the workload of each translator is balanced for the translation task; (3) the semantic contents of all the segments are as coherent and complete as possible, namely, the segmentation segments are high in semantic cohesion and low in semantic coupling; (4) the automatic segmentation of the program is realized, and on one hand, the efficiency can be improved; on the other hand, for translation tasks, task distributors may lack expertise resulting in a reasonable manual segmentation.

Based on this, a document segmentation method based on word vectors is provided in the prior art, and the word vectors are used as a standard for document segmentation, but the method mainly considers one dimension of semantics, and cannot ensure the uniformity of the lengths of each segment obtained by segmentation.

Disclosure of Invention

To overcome the above problems or at least partially solve the above problems, embodiments of the present invention provide a document segmentation method and apparatus.

In a first aspect, an embodiment of the present invention provides a document segmentation method, including:

s1, obtaining a document to be segmented;

s2, taking the document to be segmented containing a specified number of segmentation points as an individual, taking the segmentation points in the document to be segmented as genes of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the positions of the segmentation points in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

Preferably, the segmenting the document to be segmented based on the genetic algorithm specifically includes:

s21, determining an initial individual;

s22, calculating the mass of each gene;

s23, based on the quality of each gene, executing genetic operation to obtain a plurality of new individuals of the current generation;

s24, calculating the individual fitness of each new individual based on the quality of each gene of each new individual, determining the optimal individual of the current generation based on the individual fitness, and returning the optimal individual of the current generation to S22 to continue execution until the difference between the individual fitness of the optimal individual of the current generation and the individual fitness of the optimal individual of the previous generation is smaller than a preset threshold;

and S25, determining the position of the segmentation point in the document to be segmented based on the position of each gene in the current generation of optimal individuals.

Preferably, the S21 specifically includes:

s211, determining the length of a segment containing a plurality of continuous paragraphs by taking the initial paragraph of the document to be segmented as a starting point;

s212, if the length of the segment is judged to be less than or equal to the preset length, and a next subsequent segment is added in the segment and lags behind, and the length of the obtained new segment is greater than the preset length, a segmentation point exists at the end of the last segment in the segment or at the end of the subsequent segment;

s213, determining the length of the segment containing a plurality of continuous paragraphs by taking the segmentation points as starting points, and returning to S212 to continue execution until the specified number of segmentation points are determined, so as to obtain the initial individual in the genetic algorithm.

Preferably, the genetic manipulation includes genetic adjustment for adjusting the position of a gene whose gene quality satisfies a first preset condition in the initial individual and genetic variation for adjusting the position of a gene whose gene quality satisfies a second preset condition in the initial individual.

Preferably, in S23, calculating the individual fitness of each new individual based on the quality of each gene of each new individual specifically includes:

for each new individual, calculating the sum of the qualities of all genes in the new individual based on the quality of each gene of the new individual, and taking the sum of all the qualities as the individual fitness of the new individual.

Preferably, the length uniformity parameter is determined by the following method:

for each segmentation point of the individual, calculating the length difference between the segments on the left side and the right side of the segmentation point;

and carrying out normalization processing on the length difference value to obtain the length uniformity parameter of the segmentation point.

Preferably, the mass of each gene of said individual is determined in particular by the following method:

calculating the difference value between the length uniformity parameter of 1 and each segmentation point;

and calculating the product of the difference value and the semantic relevance parameter, and taking the product as the quality score of the gene corresponding to each segmentation point.

In a second aspect, an embodiment of the present invention provides a document segmenting device, including: the device comprises an acquisition module and a segmentation module. Wherein the content of the first and second substances,

the acquisition module is used for acquiring a document to be segmented;

the segmentation module is used for taking the document to be segmented containing a specified number of segmentation points as an individual, taking the segmentation points in the document to be segmented as genes of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the positions of the segmentation points in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the document segmentation method according to the first aspect when executing the program.

In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the document segmentation method according to the first aspect.

The embodiment of the invention provides a document segmentation method and a document segmentation device, wherein the method comprises the steps of firstly obtaining a document to be segmented; and then taking the document to be segmented as an individual, taking the segmentation point in the document to be segmented as the gene of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the position of the segmentation point in the document to be segmented. The quality of each gene of each individual is determined through the length uniformity parameter and the semantic relevance parameter of the segmentation point, and the length uniformity and the semantic relevance of the segmentation point are considered, so that the lengths of the fragments on two sides of the segmentation point are uniform, and the semantic relevance between the fragments on two sides of the segmentation point is small. The technical problem that the segmentation method in the prior art cannot guarantee the uniform length of the segments obtained by segmentation is solved, and a very favorable theoretical basis is provided for the average distribution of tasks.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a flowchart illustrating a document segmentation method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a document segmentation process according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating a genetic algorithm in a document segmentation method according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a document segmentation apparatus according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, an embodiment of the present invention provides a document segmentation method, including:

s1, obtaining a document to be segmented;

s2, taking the document to be segmented containing a specified number of segmentation points as an individual, taking the segmentation points in the document to be segmented as genes of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the positions of the segmentation points in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

Specifically, in the document segmentation method provided in the embodiment of the present invention, the execution main body is the processor, and is configured to implement a segmentation action on the document to be segmented, and implement segmentation of the document to be segmented, that is, determine a position of a segmentation point in the document to be segmented. The processor may be a local processor, such as a computer, a tablet computer, a smart phone, or a cloud processor, which is not specifically limited in the embodiment of the present invention.

First, step S1 is executed. The document to be segmented refers to a document to be segmented, and is usually a long document.

Then, step S2 is executed. In the embodiment of the invention, the genetic algorithm is applied to document segmentation, namely, the position of a segmentation point in a document to be segmented is determined through the genetic algorithm. The individual in the genetic algorithm refers to a document to be segmented which comprises a specified number of segmentation points, namely a segmentation scheme of the document to be segmented. The designated number may be set as required, and is not particularly limited in the embodiment of the present invention. The individual genes correspond to the segmentation points in the document to be segmented one to one. The quality of each gene of an individual is used for representing the segmentation performance of the corresponding segmentation point and guiding the specific action of genetic operation, and can be specifically represented by a quality score, wherein the higher the quality score is, the higher the quality is, the better the segmentation performance of the corresponding segmentation point is.

According to the embodiment of the invention, the quality of each individual gene is determined according to the length uniformity parameter and the semantic relevance parameter of the segmentation point corresponding to each gene by considering the length uniformity and the semantic relevance of the segmentation point. The length uniformity parameter of each segmentation point is used for representing the length difference between the segments on the left side and the right side of the segmentation point, and the larger the length uniformity parameter is, the larger the length difference is. The semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point, and the larger the semantic relevance parameter is, the smaller the semantic relevance is, and the smaller the semantic relevance degree is.

The document segmentation method provided by the embodiment of the invention comprises the steps of firstly obtaining a document to be segmented; and then taking the document to be segmented as an individual, taking the segmentation point in the document to be segmented as the gene of the individual, segmenting the document to be segmented based on a genetic algorithm, and determining the position of the segmentation point in the document to be segmented. The quality of each gene of each individual is determined through the length uniformity parameter and the semantic relevance parameter of the segmentation point, and the length uniformity and the semantic relevance of the segmentation point are considered, so that the lengths of the fragments on two sides of the segmentation point are uniform, and the semantic relevance between the fragments on two sides of the segmentation point is small. The technical problem that the segmentation method in the prior art cannot guarantee the uniform length of the segments obtained by segmentation is solved, and a very favorable theoretical basis is provided for the average distribution of tasks.

Fig. 2 is a schematic diagram of a segmentation process of a document segmentation method provided in an embodiment of the present invention, and as shown in fig. 2, on the basis of the foregoing embodiment, the document segmentation method provided in the embodiment of the present invention segments the document to be segmented based on a genetic algorithm, which specifically includes:

s21, determining an initial individual;

s22, calculating the mass of each gene;

s23, based on the quality of each gene, executing genetic operation to obtain a plurality of new individuals of the current generation;

s24, calculating the individual fitness of each new individual based on the quality of each gene of each new individual, determining the optimal individual of the current generation based on the individual fitness, and returning the optimal individual of the current generation to S22 to continue execution until the difference between the individual fitness of the optimal individual of the current generation and the individual fitness of the optimal individual of the previous generation is smaller than a preset threshold;

and S25, determining the position of the segmentation point in the document to be segmented based on the position of each gene in the current generation of optimal individuals.

Specifically, in the embodiment of the present invention, when a document to be segmented is segmented according to a genetic algorithm, the individuals include an initial individual and other individuals except the initial individual, and the difference is that the positions of the segmentation points included in the initial individual and the determination method thereof are different, the positions of the segmentation points in the initial individual are predetermined, and the positions of the segmentation points in the other individuals are determined based on the initial individual after genetic operation.

Step S21 is performed first. The initial individuals are initial segmentation schemes of the documents to be segmented, and the process of determining the initial individuals is a process of determining the initial positions of all segmentation points in the initial segmentation schemes.

Then, step S22 is executed to calculate the quality of each gene in the initial individual, which may be calculated according to the length uniformity parameter and the semantic relevance parameter of the segmentation point.

Then, step S23 is executed. The genetic manipulation refers to an action performed on a gene in an initial individual as a target, and specifically may include gene regulation and gene mutation, both of which are to change the position of the gene, that is, to change the position of a segmentation point in an initial segmentation scheme, and are different only in the targeted gene. The gene targeted by gene regulation is of lower quality than the gene targeted by genetic variation. After the genetic operation is performed, a plurality of new individuals of the current generation are obtained. This is because, in the process of performing genetic manipulation, each gene generates a new individual of the current generation after performing a genetic adjustment or genetic mutation.

Then, step S24 is executed. Wherein, based on the quality of each gene of each new individual, the individual fitness of each new individual is calculated, and the individual fitness of each new individual is used for representing the total gene quality of the new individual, and specifically can be the sum of the qualities of all genes. And determining the optimal individual of the current generation according to the individual fitness of all the new individuals, wherein the new individual with the maximum individual fitness can be selected as the optimal individual of the current generation. Returning the optimal individuals of the current generation to step S22 to continue execution for generation of new individuals of the next generation. At this time, step S22 calculates the quality of each gene in the optimal individual of the current generation.

And repeatedly executing the steps S22 to S24 until the difference between the individual fitness of the current generation of optimal individuals and the individual fitness of the previous generation of optimal individuals is smaller than a preset threshold, ending the repeated execution action, and executing the step S25. The preset threshold may be specifically set according to needs, which is not specifically limited in the embodiment of the present invention.

Finally, step S25 is performed. And determining the position of the segmentation point in the document to be segmented according to the position of each gene in the current generation of optimal individuals. Specifically, the optimal individual of the current generation may be directly used as the final segmentation scheme of the document to be segmented, that is, the positions of the genes in the optimal individual of the current generation are respectively used as the positions of the segmentation points in the document to be segmented.

On the basis of the foregoing embodiment, in the document segmenting method provided in the embodiment of the present invention, step S21 specifically includes:

s211, determining the length of a segment containing a plurality of continuous paragraphs by taking the initial paragraph of the document to be segmented as a starting point;

s212, if the length of the segment is judged to be less than or equal to the preset length, and a next subsequent segment is added in the segment and lags behind, and the length of the obtained new segment is greater than the preset length, a segmentation point exists at the end of the last segment in the segment or at the end of the subsequent segment;

s213, determining the length of the segment containing a plurality of continuous paragraphs by taking the segmentation points as starting points, and returning to S212 to continue execution until the specified number of segmentation points are determined, so as to obtain the initial individual in the genetic algorithm.

Specifically, in the embodiment of the present invention, when obtaining the initial segmentation scheme, the dimension of the length uniformity of the segmentation points is considered. Starting from a starting paragraph of a document to be segmented, calculating the length Sigma L of a segment from the starting paragraph to the ith (i is more than or equal to 1 and less than or equal to N-1, N is the total number of paragraphs in the document to be segmented, and N-1 is a segmentation point contained in the document to be segmented, namely a designated number) paragraph in the document to be segmented one by one_iAnd the length sigma L of the new segment obtained by adding an adjacent subsequent segment, i.e. the (i + 1) th segment, in the constructed segment_i+1And determines sigma L_iAnd a predetermined length L_NIf Σ L, is_i≤L_NAnd sigma L_i+1＞L_NIt is stated that there may be a segmentation point at the end of the ith paragraph or the (i + 1) th paragraph. The selection method is to judge sigma L_iOr Σ L_i+1Which value in (1) is associated with L_NNearest, if Σ L_iAnd L_NClosest, it is determined that there is a segmentation point at the end of the ith paragraph, if ∑ L_i+1And L_NClosest, it is determined that there is a segmentation point at the end of the ith paragraph.

And after the first segmentation point is determined, taking the first segmentation point as a document starting point, and determining the next segmentation point by the same method until all N-1 segmentation points are determined, thereby obtaining the initial individual in the genetic algorithm.

On the basis of the above embodiments, the preset length in the embodiments of the present inventionDegree L_NSpecifically, it can be determined by the following formula:

L_N＝L/N

wherein, L is the total length of the document to be segmented.

By setting the preset length, the resulting initial individual will generally satisfy the maximum length and minimum length constraints of each segment segmented, i.e. the segmentation points have length uniformity.

On the basis of the foregoing embodiments, in the document segmentation method provided in an embodiment of the present invention, the genetic manipulation includes genetic adjustment and genetic variation, the genetic adjustment is used to adjust the position of a gene whose genetic quality satisfies a first preset condition in the initial individual, and the genetic variation is used to adjust the position of a gene whose genetic quality satisfies a second preset condition in the initial individual.

Specifically, the specific actions of genetic manipulation in the embodiments of the present invention may include gene regulation and gene mutation. The genes targeted by the gene regulation are genes with the gene quality meeting a first preset condition, for example, all the genes in the initial individual are sequenced from low to high according to the gene quality, and then the first 20% of the genes in the sequence are selected and belong to the genes with the gene quality meeting the first preset condition. The gene targeted by the gene variation is a gene whose gene quality satisfies a second preset condition, for example, the last 80% of the genes in the sequence are selected, and then 5% of the genes are randomly selected from the last 80% of the genes, and the 5% of the genes are genes whose gene quality satisfies the second preset condition.

When the gene i is subjected to gene adjustment, the adjustable range of the position of the gene i is the ending positions of all paragraphs between the i-1 th segmentation point and the i +1 th segmentation point, the quality score of the position of the gene i at each position in the adjustable range is sequentially calculated, whether the fragments on two sides of the gene i meet the length constraint or not is judged, and the position with the largest quality score is taken as the position of the gene i after adjustment under the condition that the length constraint is met. After the position of the gene i is adjusted, the gene i-1 and the quality score based on i +1 need to be updated, so as to obtain a new individual of the current generation.

Fig. 3 is a schematic diagram illustrating a specific flow of a genetic algorithm in the document segmentation method according to an embodiment of the present invention. 1) Initial individuals were generated, including 6 genes in the initial individuals shown in fig. 3. 2) And calculating the mass of each gene in the initial individual, wherein the higher the mass of the gene is, the smaller the corresponding circle is. 3) And (4) performing genetic operation to generate a new population, wherein the new population is composed of a plurality of new individuals of the current generation. The specific actions of genetic manipulation include gene regulation and gene mutation, and in fig. 3, the gene regulation action is performed for a gene with a large circle, and the gene mutation action is performed for a gene with a small circle. 4) And calculating the individual fitness of each new individual of the current generation, and selecting the optimal individual of the current generation from a plurality of new individuals of the current generation. And taking the optimal individual of the current generation as an initial individual, and performing iteration 2). And ending the iteration until the difference between the individual fitness of the current generation of optimal individuals and the individual fitness of the previous generation of optimal individuals is smaller than a preset threshold value.

On the basis of the above embodiment, the length uniformity parameter is specifically determined by the following method:

for each segmentation point of the individual, calculating the length difference between the segments on the left side and the right side of the segmentation point;

and carrying out normalization processing on the length difference value to obtain the length uniformity parameter of the segmentation point.

Specifically, in the embodiment of the present invention, when determining the length uniformity parameter of a certain dividing point, first, the length difference between the segments on the left and right sides of the dividing point is calculated, for example, the segments on the left and right sides of the dividing point i are respectively S_iAnd S_i+1Then, the length difference between the segments on the left and right sides of the segmentation point i is calculated by the following formula:

wherein the content of the first and second substances,is the difference in length between segments on the left and right sides of the dividing point i, i.e. the difference in length between segments on the left and right sides of the dividing point iAbsolute value of (a), len (S)_i+1) Is the length of the segment to the right of the segmentation point i, len (S)_i) Is the length of the segment to the left of the segmentation point i.

Then, normalizing the length difference, specifically determining the sum of the length differences corresponding to all the segmentation points, and calculating the ratio of the length difference corresponding to the segmentation point i to the sum of the length differences corresponding to all the segmentation points to obtain the length uniformity parameter p of the segmentation point i_i。

On the basis of the above embodiment, the semantic relevance parameter is related to the word vectors of the segments on the left and right sides of the segmentation point, and can be specifically determined by a method existing in the prior art. For example, it can be determined by a Content Vector Segmentation (CVS) algorithm that the larger the semantic relevance parameter is, the better the Segmentation performance of the Segmentation point in the semantic dimension is.

On the basis of the above examples, the mass of each gene of the individual is specifically determined by the following method:

calculating the difference value between the length uniformity parameter of 1 and each segmentation point;

and calculating the product of the difference value and the semantic relevance parameter, and taking the product as the quality score of the gene corresponding to each segmentation point.

Specifically, let the length uniformity parameter of the division point i be p_iCalculating the difference value of the length uniformity parameter between 1 and the division point i to obtain 1-p_i。

And if the semantic relevance parameter of the segmentation point i is Semantic (i), the following steps are performed:

Score(i)＝Semantic(i)*(1-p_i)；

wherein score (i) is the quality score of the gene corresponding to the segmentation point i.

On the basis of the above embodiment as shown in fig. 4, an embodiment of the present invention provides a document segmenting device, including: an acquisition module 41 and a segmentation module 42. Wherein the content of the first and second substances,

the obtaining module 41 is configured to obtain a document to be segmented;

the segmenting module 42 is configured to segment the document to be segmented based on a genetic algorithm with the document to be segmented as an individual and the segmenting point in the document to be segmented as the gene of the individual, and determine the position of the segmenting point in the document to be segmented;

the quality of each gene of the individual is determined based on a length uniformity parameter and a semantic relevance parameter of the segmentation point, wherein the length uniformity parameter is used for representing the length difference between the fragments on the left side and the right side of the segmentation point, and the semantic relevance parameter is used for representing the semantic relevance degree between the fragments on the left side and the right side of the segmentation point.

Specifically, the functions of the modules in the document segmentation apparatus provided in the embodiment of the present invention correspond to the operation flows of the steps in the method embodiments one to one, and the implementation effects are also consistent.

As shown in fig. 5, on the basis of the above embodiment, an embodiment of the present invention provides an electronic device, including: a processor (processor)501, a memory (memory)502, a communication Interface (Communications Interface)503, and a communication bus 504; wherein the content of the first and second substances,

the processor 501, the memory 502 and the communication interface 503 are communicated with each other through a communication bus 504. The memory 502 stores program instructions executable by the processor 501, and the processor 501 is configured to call the program instructions in the memory 502 to perform the document segmentation method provided by the above-mentioned embodiments of the method.

It should be noted that, when being implemented specifically, the electronic device in this embodiment may be a server, a PC, or another device, as long as the structure includes the processor 501, the communication interface 503, the memory 502, and the communication bus 504 shown in fig. 5, where the processor 501, the communication interface 503, and the memory 502 complete mutual communication through the communication bus 504, and the processor 501 may call a logic instruction in the memory 502 to execute the above method. The embodiment does not limit the specific implementation form of the electronic device.

The logic instructions in memory 502 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone article of manufacture. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Further, the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer can execute the document segmentation method provided by the above method embodiments.

On the basis of the above embodiments, the embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, which is implemented by a processor to execute the document segmentation method provided by the above embodiments.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.