阅读说明：本技术 文本纠错分词方法、系统及电子设备 (Text error correction word segmentation method and system and electronic equipment ) 是由 元方 张凡超 于 2020-11-26 设计创作，主要内容包括：本发明提供一种文本纠错分词方法、系统及电子设备,通过建立拼音字符组与任务相关中文词的对应关系,并结合AC自动机可同时完成分词与纠错,准确率较高,计算复杂度较低。(The invention provides a text error correction word segmentation method, a text error correction word segmentation system and electronic equipment.)

1. A text error correction word segmentation method is characterized by comprising the following steps:

establishing a corresponding relation between a pinyin character group and a task-related Chinese word, wherein the pinyin character group corresponding to each Chinese word comprises a mandarin pinyin character group, a dialect pinyin character group and a similar character pinyin character group;

constructing an AC automaton Tree Tree _ A according to the pinyin character groups in the corresponding relationship;

receiving a text of a word to be corrected;

converting the text into pinyin, and removing tones to form a pinyin character string;

performing multi-mode matching on the pinyin character strings by using an AC automaton of Tree _ A to obtain a plurality of successfully matched pinyin character groups;

and restoring the successfully matched pinyin character group into the corresponding Chinese word according to the corresponding relation.

2. The method as claimed in claim 1, wherein the establishing of the correspondence between the pinyin character set and the task-related chinese words further comprises:

a. segmenting words of the pre-collected task related texts through a word segmentation device, and generating a Chinese word list after checking, correcting and deleting completely wrong words;

b. performing phonetization on the Chinese word list to obtain a pinyin mapping word list, wherein the pinyin mapping word list is composed of keys and values, the keys represent pinyin character groups, and the values represent Chinese words corresponding to the keys;

c. expanding the pinyin mapping word list through dialect pinyin:

constructing a dialect pinyin character table, wherein the dialect pinyin character table is composed of keys and values, the keys represent mandarin pinyin character groups, and the values represent corresponding dialect pinyin character groups;

matching the dialect pinyin character table with a pinyin mapping word table, and if one key _ m in the pinyin mapping word table contains the key _ n of the dialect pinyin character table, replacing the corresponding part of the key _ m with the value corresponding to the key _ n to form a new key of the pinyin mapping word table, wherein the value corresponding to the new key is the same as the value corresponding to the key _ m;

d. expanding the pinyin mapping word list through shape-near pinyin:

constructing a shape near character pinyin character table, wherein the shape near character pinyin character table is composed of keys and values, the keys represent pinyin character groups of correct Chinese words, and the values represent pinyin character groups of wrong Chinese words;

and matching the shape near character pinyin character table with a pinyin mapping word table, and if the key _ o of the shape near character pinyin character table is matched with the key _ q of the pinyin mapping word table, taking the value corresponding to the key _ o as a new key of the pinyin mapping word table, wherein the value corresponding to the new key is the same as the value corresponding to the key _ q.

3. The method as claimed in claim 2, wherein the step b further comprises:

if a plurality of keys exist after one Chinese word is spelled, the plurality of keys correspond to the same value;

and if the keys after the spellings of the Chinese words are the same, the keys correspond to a plurality of values.

4. The method as claimed in claim 3, wherein the step c further comprises:

and if the new key exists in the pinyin mapping word list, performing array addition on the value of the new key.

5. The method as claimed in claim 4, wherein the step d further comprises:

and if the new key exists in the pinyin mapping word list, performing array addition on the value of the new key.

6. The method for dividing words into words according to claim 2 or 5, wherein the word-dividing device is a jieba word-dividing device.

7. A text error correction segmentation system comprising a memory module including instructions loaded and executed by a processor, the instructions when executed causing the processor to perform a text error correction segmentation method according to any one of claims 1 to 6.

8. An electronic device, characterized in that it has a text error correction and word segmentation system according to claim 7.

Technical Field

The invention belongs to the technical field of text and natural language processing, and particularly relates to a text error correction word segmentation method, a text error correction word segmentation system and electronic equipment.

Background

In a Chinese text scene, the Pinyin input method is the input method with the largest use amount, and of course, the Chinese text scene also comprises handwriting or five strokes, but the Chinese text scene is used by the users of the small public. After the text is input, there are various reasons for the input error of the text, such as the difference between dialect pinyin and standard mandarin pinyin, e.g. n is l, c is ch, h is f, and en is eng; in addition, polyphones exist in the Chinese text, which also introduces errors, and it is natural that the preprocessing needs to correct the errors, which is also called a text error correction task in the industry, and besides, a crucial task is to perform word segmentation on the text.

At present, for the two pre-processing tasks, word segmentation and error correction are generally completed independently in the industry, and different algorithms are generally adopted to respectively and independently process the two problems, so that the method has the advantages of high accuracy and increased computational complexity by multiple orders of magnitude.

Disclosure of Invention

Based on the above, a text error correction word segmentation method, a text error correction word segmentation system and an electronic device are provided for solving the technical problems.

In order to solve the technical problems, the invention adopts the following technical scheme:

in one aspect, the present invention provides a text error correction word segmentation method, including:

establishing a corresponding relation between a pinyin character group and a task-related Chinese word, wherein the pinyin character group corresponding to each Chinese word comprises a mandarin pinyin character group, a dialect pinyin character group and a similar character pinyin character group;

constructing an AC automaton Tree Tree _ A according to the pinyin character groups in the corresponding relationship;

receiving a text of a word to be corrected;

converting the text into pinyin, and removing tones to form a pinyin character string;

performing multi-mode matching on the pinyin character strings by using an AC automaton of Tree _ A to obtain a plurality of successfully matched pinyin character groups;

and restoring the successfully matched pinyin character group into the corresponding Chinese word according to the corresponding relation.

In another aspect, the present invention provides a text error correction word segmentation system, which includes a storage module including instructions loaded and executed by a processor, wherein the instructions, when executed, cause the processor to execute a text error correction word segmentation method as described above.

In still another aspect, the present invention provides an electronic device having the above-mentioned text error correction and word segmentation system.

The invention can complete word segmentation and error correction simultaneously by establishing the corresponding relation between the pinyin character group and the task-related Chinese words and combining the AC automaton, and has higher accuracy and lower calculation complexity.

Drawings

The invention is described in detail below with reference to the following figures and detailed description:

FIG. 1 is a flow chart of the present invention.

Detailed Description

As shown in fig. 1, an embodiment of the present specification provides a text error correction word segmentation method, including:

s101, establishing a corresponding relation between pinyin character groups and task-related Chinese words, wherein the pinyin character group corresponding to each Chinese word comprises a Mandarin pinyin character group, a dialect pinyin character group and a similar character pinyin character group. The specific process is as follows:

a. and performing word segmentation on the pre-collected task related text through a word segmentation device, and generating a Chinese word list after checking, correcting and deleting completely wrong words.

In this embodiment, the full-mode word segmentation of the jieba word segmenter is used, so that all possible words can be segmented, thereby reducing errors caused by word segmentation errors, but no errors can be ensured, so that verification is required, the wrong words are corrected, and the completely wrong words are directly deleted, thereby ensuring the accuracy of the Chinese vocabulary.

b. And performing phonetization on the Chinese word list to obtain a pinyin mapping word list, wherein the pinyin mapping word list is the expression form of the corresponding relation, the pinyin mapping word list is composed of keys and values, the keys represent pinyin character groups, and the values represent Chinese words corresponding to the keys.

If there are multiple keys after one Chinese word is spelled, then multiple keys correspond to the same value, such as { feij: airplane, feiji: airplane, fj: airplane }.

If the keys after the spellings of the plurality of Chinese words are the same (if the keys after the spellings are the same although the actual tones of the plurality of Chinese words are different), the keys correspond to the plurality of values.

c. Expanding the pinyin mapping word list through dialect pinyin:

a dialect pinyin character table is constructed, the dialect pinyin character table is composed of keys and values, the keys represent mandarin pinyin character sets, and the values represent corresponding dialect pinyin character sets (reading confusion of flat-tongue sound and warped-tongue sound, unclear front and back nasal sound resolution, and the like), for example: { L: N, F: H, CH: C, ENG: EN.

Matching the dialect pinyin character table with the pinyin mapping word list, if one key _ m in the pinyin mapping word list contains the key _ n of the dialect pinyin character table, replacing the corresponding part of the key _ m with the value corresponding to the key _ n to form a new key of the pinyin mapping word list, wherein the value corresponding to the new key is the same as the value corresponding to the key _ m, and if the pinyin mapping word list has { feij: airplane, dialect pinyin character table having { f: h, we can see feij contains f, and finally expand to { feiji: airplane, heiji: airplane }.

If the new key exists in the pinyin mapping word list, the value of the new key is added in an array form, and the result is expressed as { new key: [ value1, value 2. ] }.

d. The pinyin mapping word list is expanded through pinyin of shape-close characters (similar five strokes input strokes/similar writing input forms, and the like):

and constructing a shape near character pinyin character table, wherein the shape near character pinyin character table is composed of keys and values, the keys represent the pinyin character groups of correct Chinese words, and the values represent the pinyin character groups of wrong Chinese words. If the characters of the sun and the moon are similar to the characters of the moon, mistakes are easy to make, and the characters are expressed as { riyue: yueyue }, where key is riyue and value is yueyue.

And matching the shape near character pinyin character table with the pinyin mapping word table, and if the key _ o of the shape near character pinyin character table matches the key _ q of the pinyin mapping word table, taking the value corresponding to the key _ o as a new key of the pinyin mapping word table, wherein the value corresponding to the new key is the same as the value corresponding to the key _ q. For example, in the shape-similar character pinyin character table, there are { riyue: yueyue }, and a pinyin mapping word list contains { riyue: day and month, then finally expand to { riyue: day and month, yueyue: sun and moon }.

If the new key already exists in the pinyin mapping word list, adding an array to the value of the new key.

S102, constructing an AC automaton Tree Tree _ A according to the pinyin character groups in the corresponding relation.

S103, receiving a text of the word segmentation to be corrected.

And S104, converting the text into pinyin, and removing tones to form a pinyin character string.

And S105, performing multi-mode matching on the pinyin character strings by using an AC automaton of Tree _ A to obtain a plurality of successfully matched pinyin character groups.

And S106, restoring the successfully matched pinyin character group into a corresponding Chinese word according to the corresponding relation.

Based on the same inventive concept, the present specification also provides a text error correction word segmentation system, which includes a storage module including instructions (program code) loaded and executed by a processor, where the instructions, when executed, cause the processor to execute the steps according to the various exemplary embodiments of the present invention described in the above text error correction word segmentation method section of the present specification.

The memory module may include a readable medium in the form of a volatile memory unit, such as a random access memory unit (RAM) and/or a cache memory unit, and may further include a read only memory unit (ROM).

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

Based on the same inventive concept, an embodiment of the present specification further provides an electronic device, where the electronic device has the text error correction word segmentation system described above in the present specification, and details are not described here.

However, those skilled in the art should realize that the above embodiments are illustrative only and not limiting to the present invention, and that changes and modifications to the above described embodiments are intended to fall within the scope of the appended claims, provided they fall within the true spirit of the present invention.