阅读说明：本技术 一种站内搜索系统及方法 (in-station search system and method ) 是由 刘宝柱 吴小铭 周宏印 于 2019-08-12 设计创作，主要内容包括：本发明公开了一种站内搜索系统及方法,系统包括：网页爬虫模块对获取页面内容进行数据清洗及页面去重；页面内容分词模块对页面内容进行分词操作；网页数据库包含原始数据库和页面分词数据库；全文检索模块对原始数据库和页面分词数据库建立索引,以完成查询和输出结果；传销行为规则库和样本库包含网络上已公布的传销相关新闻数据和利用该新闻数据分析生成的传销行为的特征规则数据；网页内容相关性分析模块对全文检索得出的结果与用户查询关键字进行权重计算页面内容相关性。本发明解决了网页内容的分词方式不具备完整性,实际分词结果往往差强人意,导致搜索引擎搜索出的结果并不是用户想要的数据,或者搜索结果中包含太多的无用信息的问题。(the invention discloses an in-station search system and a method, wherein the system comprises: the webpage crawler module performs data cleaning and page duplication removal on the acquired page content; the page content word segmentation module carries out word segmentation operation on page content; the web page database comprises an original database and a page participle database; the full-text retrieval module establishes indexes for the original database and the page participle database to complete query and output results; the distribution behavior rule base and the sample base comprise news data related to distribution published on the network and characteristic rule data of distribution behaviors generated by analyzing the news data; and the webpage content correlation analysis module carries out weight calculation on the results obtained by the full text retrieval and the user query keywords to calculate the page content correlation. The invention solves the problems that the word segmentation mode of the webpage content is not complete, the actual word segmentation result is often poor, the result searched by a search engine is not the data wanted by the user, or the search result contains too much useless information.)

1. An in-station search system, comprising:

The webpage crawler module is used for acquiring page contents, and performing data cleaning and page duplication removal on the page contents;

The page content word segmentation module is used for carrying out word segmentation operation on the acquired processed page content;

a web page database comprising an original database and a page participle database; the original database is maintained by continuously acquiring webpage data through a webpage crawler module so as to ensure that the original database keeps the latest state; the page word segmentation database is maintained by performing word segmentation operation on webpage content in the original database by a page content word segmentation module;

the full-text retrieval module is used for establishing indexes for the original database and the page participle database so as to complete query and output results;

the system comprises a distribution behavior rule base and a sample base, wherein the distribution behavior rule base and the sample base comprise news data related to distribution published on a network and characteristic rule data of distribution behaviors generated by analyzing the news data and are used for detecting the distribution behaviors of a user retrieval result page;

And the webpage content correlation analysis module is used for carrying out weight calculation on the results obtained by full-text retrieval and the user query keywords to calculate the page content correlation and reordering the output results.

2. the in-site search system of claim 1, wherein the web crawler module employs a distributed crawler system, and combines a breadth-first traversal algorithm to obtain the content of the page.

3. the in-site search system of claim 1, wherein the full-text retrieval module comprises two parts of indexing and searching, specifically as follows:

establishing an index:

(1) Acquiring a series of indexed files through a webpage crawler module;

(2) The indexed file is processed by syntax analysis and language to form a series of words;

(3) Creating a dictionary and a reverse index table through indexes;

(4) writing the index into a disk through index storage;

searching:

(a) transmitting a query statement;

(b) carrying out syntactic analysis and language analysis on the query statement to obtain a series of words;

(c) Obtaining a query tree through syntactic analysis;

(d) Reading the index into a memory through index storage;

(e) searching indexes by using the query tree so as to obtain a document linked list of each word, and performing intersection, difference and operation on the document linked lists to obtain a result document;

(f) Ranking the query relevance of the searched result documents;

(g) and returning the query result to the user.

4. the in-site search system of claim 1, wherein the web crawler module comprises a crawler controller, a task scheduler, a filter, and a crawler monitor module;

a crawler controller: the crawler is used for realizing the strategy control of the depth and the breadth of network resource grabbing and the grabbing priority according to the seed link;

And (3) a task scheduler: the method is used for improving the efficiency and the speed of crawling network resources by a crawler, and a webpage crawler module adopts distributed deployment and needs to perform load balancing and parallel crawling operations on crawler tasks by a task scheduler;

a filter: the system is used for filtering useless information and repeated contents in the network, and improving the utilization rate of resource space and the running speed of a search engine;

crawler monitoring module: the crawler module monitoring system is used for ensuring the efficient operation of the crawler and monitoring the abnormal operation of the whole crawler module in real time.

5. the in-site search system of claim 1, wherein the page content segmentation module performs a segmentation operation on the web content using a conditional random field algorithm.

6. the in-station search system of claim 5, wherein the conditional random field algorithm is specifically:

k1 transfer characteristics are provided, K2 state characteristics are provided, and K is K1+ K2;

wherein the transition feature tk is a feature function defined on the edge, dependent on the current position i and the previous position i-1; the state characteristic sl is a characteristic function defined on the node, dependent on the current position i; l is 1 to K2 status features; x is an observation sequence, and y is a state sequence;

summing all features at position i:

Denote the weight of the feature fk (y, x) by wk, i.e.:

λ k is the weight of the transfer characteristic tk, μ l is the weight of the state characteristic sl;

the conditional random field is represented as:

where P (y | x) is the conditional random field and Z (x) is the normalization factor, which is the sum of all possible values of y.

7. the in-site search system of claim 1, wherein the web content relevance analysis module uses BM25F algorithm to weight the results obtained from full-text search and the user query keywords to calculate the relevance of the web content and reorder the output results.

8. the in-station search system of claim 7, wherein the BM25F algorithm is specifically:

wherein, Bu represents the length condition of each domain, fiu represents the weighted sum of the ith word in each domain, fui represents the word frequency; bu is an adjusting factor of the length of each domain, ulu is the actual length of the domain, and uvulu is the average length of the domain; the calculation formula of the BM25F model is as follows:

In the formula, R represents the number of relevant documents, N represents the total number of documents, di ═ 1 represents words appearing in the documents, qi represents words queried by the user, ri represents the number of words appearing in the relevant documents, ni represents the number of words appearing in all the documents, and k1 is an empirical parameter.

9. an intra-site search method is characterized by comprising the following steps:

1) Inputting a query keyword;

2) performing word segmentation operation on the keywords;

3) capturing corresponding network resources according to the word segmentation result of the keyword;

4) Performing word segmentation operation on the network resources captured by the crawler, and warehousing;

5) establishing an index for the newly-put data, and searching according to the keyword segmentation result;

6) analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) and the output results are reordered, so that the matching accuracy is improved.

10. a computer-readable storage medium storing a computer program, the program, when executed by a processor, implementing a method of:

1) inputting a query keyword;

2) performing word segmentation operation on the keywords;

3) capturing corresponding network resources according to the word segmentation result of the keyword;

4) Performing word segmentation operation on the network resources captured by the crawler, and warehousing;

5) establishing an index for the newly-put data, and searching according to the keyword segmentation result;

6) analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) and the output results are reordered, so that the matching accuracy is improved.

Technical Field

The invention belongs to the technical field of full-text retrieval, and particularly relates to an in-site search system and method with website marketing behavior analysis.

background

with the development and rapid popularization of internet technology, more and more websites are provided, the number of webpages on the internet per day is increased in the order of tens of millions, and the required materials are required to be searched for in massive data resources, which is not different from a large sea fishing needle. In order to meet the search requirements of people, search engines are produced. The search engine is a system that collects information from the internet by using a specific computer program according to a certain policy, provides a retrieval service for a user after organizing and processing the information, and displays information related to user retrieval to the user. At present, most search engines are all network search, information is noisy, and even the information is flooded by a large number of advertisements. The in-site search engine can perform in-site global search for websites concerned by the user, and the relevance of information is greatly improved.

For the establishment of the in-site search engine, the current workload and technical points are mainly in the aspects of natural language processing technology and output result relevance sequencing, and the word segmentation mode of the webpage content mainly comprises the following steps: 1. in the word segmentation algorithm based on the dictionary, word segmentation based on the dictionary is usually dependent on the dictionary and a rule base, however, with the development of the internet, new words frequently appear, and the method cannot effectively perform word segmentation. 2. The statistical-based machine learning algorithm is not limited by the field of texts to be processed, does not need a special dictionary, but needs a large amount of training texts, and has poor recognition accuracy of common words.

Because the modes are independent and not complete, and have differences and defects, the actual word segmentation result is often poor in popularity, the result searched by the search engine is not data wanted by the user, or the search result contains too much useless information. Because the information on the internet is too noisy, marketing frauds are more and more coming, and in order to improve the identification degree of a user on reliable information, an in-site search engine with website marketing behavior analysis is provided, and a more reliable search engine is described.

disclosure of Invention

in view of the above deficiencies of the prior art, an object of the present invention is to provide a system and a method for in-site search, so as to solve the problems that the existing word segmentation method of web page content does not have integrity, and the actual word segmentation result is often poor in popularity, so that the result searched by the search engine is not the data desired by the user, or the search result contains too much useless information.

in order to achieve the purpose, the technical scheme adopted by the invention is as follows:

the invention relates to an in-station search system, which comprises:

The webpage crawler module is used for acquiring page contents, and performing data cleaning and page duplication removal on the page contents;

The page content word segmentation module is used for carrying out word segmentation operation on the acquired processed page content;

a web page database comprising an original database and a page participle database; the original database is maintained by continuously acquiring webpage data through a webpage crawler module so as to ensure that the original database keeps the latest state; the page word segmentation database is maintained by performing word segmentation operation on webpage content in the original database by a page content word segmentation module;

the full-text retrieval module is used for establishing indexes for the original database and the page participle database so as to complete query and output results;

the system comprises a distribution behavior rule base and a sample base, wherein the distribution behavior rule base and the sample base comprise news data related to distribution published on a network and characteristic rule data of distribution behaviors generated by analyzing the news data and are used for detecting the distribution behaviors of a user retrieval result page;

And the webpage content correlation analysis module is used for carrying out weight calculation on the results obtained by full-text retrieval and the user query keywords to calculate the page content correlation and reordering the output results.

Furthermore, the webpage crawler module adopts a distributed crawler system and combines a breadth-first traversal algorithm to obtain the page content.

further, the full-text retrieval module comprises two parts of index establishment and search, and the following steps are specifically performed:

Establishing an index:

(1) acquiring a series of indexed files through a webpage crawler module;

(2) the indexed file is processed by syntax analysis and language to form a series of words;

(3) Creating a dictionary and a reverse index table through indexes;

(4) writing the index into a disk through index storage;

Searching:

(a) transmitting a query statement;

(b) Carrying out syntactic analysis and language analysis on the query statement to obtain a series of words;

(c) obtaining a query tree through syntactic analysis;

(d) reading the index into a memory through index storage;

(e) searching indexes by using the query tree so as to obtain a document linked list of each word, and performing intersection, difference and operation on the document linked lists to obtain a result document;

(f) ranking the query relevance of the searched result documents;

(g) and returning the query result to the user.

furthermore, the webpage crawler module comprises a crawler controller, a task scheduler, a filter and a crawler monitoring module;

a crawler controller: the crawler is used for realizing the strategy control of the depth and the breadth of network resource grabbing and the grabbing priority according to the seed link;

and (3) a task scheduler: the method is used for improving the efficiency and the speed of crawling network resources by a crawler, and a webpage crawler module adopts distributed deployment and needs to perform load balancing and parallel crawling operations on crawler tasks by a task scheduler;

a filter: the system is used for filtering useless information and repeated contents in the network, and improving the utilization rate of resource space and the running speed of a search engine;

Crawler monitoring module: the system is used for ensuring the efficient operation of the crawler, monitoring the abnormal operation of the whole crawler module in real time and improving the operation reliability of the whole system.

further, the page content word segmentation module performs word segmentation on the page content by using a Conditional Random Field (CRF).

Further, the conditional random field algorithm specifically comprises:

k1 transfer characteristics are provided, K2 state characteristics are provided, and K is K1+ K2;

Wherein the transition feature tk is a feature function defined on the edge, dependent on the current position i and the previous position i-1; the state characteristic sl is a characteristic function defined on the node, dependent on the current position i; l is 1 to K2 status features; x is an observation sequence, and y is a state sequence;

summing all features at position i:

Denote the weight of the feature fk (y, x) by wk, i.e.:

λ k is the weight of the transfer characteristic tk, μ l is the weight of the state characteristic sl;

The conditional random field is represented as:

where P (y | x) is the conditional random field and Z (x) is the normalization factor, which is the sum of all possible values of y.

further, the web content relevance analysis module adopts BM25F algorithm to calculate the relevance of the web content by weighting the results obtained from the full text search and the user query keywords, and re-orders the output results.

Further, the BM25F algorithm is specifically:

wherein, Bu represents the length condition of each domain, fiu represents the weighted sum of the ith word in each domain, fui represents the word frequency; bu is an adjusting factor of the length of each domain, ulu is the actual length of the domain, and uvulu is the average length of the domain; the calculation formula of the BM25F model is as follows:

In the formula, R represents the number of relevant documents, N represents the total number of documents, di ═ 1 represents words appearing in the documents, qi represents words queried by the user, ri represents the number of words appearing in the relevant documents, ni represents the number of words appearing in all the documents, and k1 is an empirical parameter.

the invention can effectively search information data on the Internet, more importantly, the invention can automatically adjust the analysis result, mark the reliability of the data source and finally output the query result which is most in line with the expectation of the user by the processing of the natural language technology and the research and judgment intervention behavior of the biography and marketing behavior rules. The method is suitable for industries making decisions based on information or intelligence by various services such as industry and commerce, public security, market supervision and the like.

such as: the search engine can be used for carrying out full search on the websites and analyzing the uploading and selling behaviors to obtain results quickly; according to the alarm description, quickly retrieving and tracking information of a certain website or forum; and quickly carrying out online public opinion investigation and the like on a certain market main body.

the invention discloses an in-station searching method, which comprises the following steps:

1) inputting a query keyword;

2) performing word segmentation operation on the keywords;

3) capturing corresponding network resources according to the word segmentation result of the keyword;

4) performing word segmentation operation on the network resources captured by the crawler, and warehousing;

5) establishing an index for the newly-put data, and searching according to the keyword segmentation result;

6) analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) and the output results are reordered, so that the matching accuracy is improved.

a computer-readable storage medium storing a computer program which, when executed by a processor, implements a method of:

1) Inputting a query keyword;

2) Performing word segmentation operation on the keywords;

3) Capturing corresponding network resources according to the word segmentation result of the keyword;

4) performing word segmentation operation on the network resources captured by the crawler, and warehousing;

5) establishing an index for the newly-put data, and searching according to the keyword segmentation result;

6) analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) and the output results are reordered, so that the matching accuracy is improved.

the invention has the beneficial effects that:

1. and (3) searching contents with higher precision: the CRF algorithm labels Chinese characters, namely, words (word groups) are formed by the characters, not only the frequency information of the appearance of the words of the characters is considered, but also the context is considered, and the CRF algorithm has better learning capacity, so that the CRF algorithm has good effect on identifying ambiguous words and unknown words.

2. Sequencing results with high reliability: different from the traditional ranking algorithm, the BM25F algorithm has a strong advantage in ranking web pages, and when calculating the relevance, the BM25F algorithm divides documents into different domains for weighted statistics, which is very suitable for web page search, because there are title information, meta information, page content information, etc. in a web page, and the title information is definitely the most important, and the meta information is the last web page content, and when calculating the relevance, the BM25F divides the web page into different areas, and respectively counts its own word frequency in each area.

3. recognition of the biography behavior: the method and the device can analyze the marketing behavior of the website or the webpage content, and improve the user experience.

Drawings

FIG. 1 is a technical framework diagram of the present invention.

FIG. 2 is a schematic diagram of full-text indexing.

FIG. 3 is a diagram of a CRF algorithm map model.

fig. 4 illustrates a method of operating a search engine.

Detailed Description

In order to facilitate understanding of those skilled in the art, the present invention will be further described with reference to the following examples and drawings, which are not intended to limit the present invention.

Breadth-First traversal (Breadth First Search), which is a traversal of the nodes of the tree along the width of the tree starting from the root node, and if all nodes are visited, the algorithm terminates. Is an important link of the search engine in the station.

CRF (conditional random field) represents a new generation of machine learning technology word segmentation, and the basic idea is to label Chinese characters, namely to form words (group words) by characters, so that the method not only considers the frequency information of the appearance of the words, but also considers the context, and has better learning ability, and therefore, the method has good effect on identifying ambiguous words and unknown words. Given X, if each random variable Yv obeys markov property, that is: p (Yv | X, Yw, w ≠ v) ═ P (Yv | X, Yw, w to v);

BM25F, BM25F is a modified algorithm of the typical BM 25; the BM25 considers documents as a whole when calculating relevance, but as search techniques develop. Documents are slowly replaced by structured data. Each document is divided into multiple independent domains, especially for verticalized searches. For example, a web page may be divided into fields such as title, content, and subject word, and these fields do not contribute equally to the subject of the article, so the weight is weighted. The BM25 does not take this into account. BM25F makes some improvement over this in that words are not considered individually any more, and documents are also divided into individual considerations according to field, so BM25F is a weighted sum of the scores of each word in the respective fields.

referring to fig. 1, an intra-site search system according to the present invention includes:

The webpage crawler module is used for acquiring page contents, and performing data cleaning and page duplication removal on the page contents;

the page content word segmentation module is used for carrying out word segmentation operation on the acquired processed page content;

A web page database comprising an original database and a page participle database; the original database is maintained by continuously acquiring webpage data through a webpage crawler module so as to ensure that the original database keeps the latest state; the page word segmentation database is maintained by performing word segmentation operation on webpage content in the original database by a page content word segmentation module;

The full-text retrieval module is used for establishing indexes for the original database and the page participle database so as to complete query and output results;

the system comprises a distribution behavior rule base and a sample base, wherein the distribution behavior rule base and the sample base comprise news data related to distribution published on a network and characteristic rule data of distribution behaviors generated by analyzing the news data and are used for detecting the distribution behaviors of a user retrieval result page;

and the webpage content correlation analysis module is used for carrying out weight calculation on the results obtained by full-text retrieval and the user query keywords to calculate the page content correlation and reordering the output results.

the webpage crawler module adopts a distributed crawler system and combines an breadth-first traversal algorithm to obtain the page content.

the full-text retrieval module comprises two parts of establishing indexes (1) - (4) and searching (a) - (g), and the following steps are specifically carried out:

establishing an index:

(1) acquiring a series of indexed files through a webpage crawler module;

(2) The indexed file is processed by syntax analysis and language to form a series of words;

(3) creating a dictionary and a reverse index table through indexes;

(4) writing the index into a disk through index storage;

Searching:

(a) transmitting a query statement;

(b) carrying out syntactic analysis and language analysis on the query statement to obtain a series of words;

(c) Obtaining a query tree through syntactic analysis;

(d) reading the index into a memory through index storage;

(e) searching indexes by using the query tree so as to obtain a document linked list of each word, and performing intersection, difference and operation on the document linked lists to obtain a result document;

(f) ranking the query relevance of the searched result documents;

(g) And returning the query result to the user.

the webpage crawler module comprises a crawler controller, a task scheduler, a filter and a crawler monitoring module;

a crawler controller: the crawler is used for realizing the strategy control of the depth and the breadth of network resource grabbing and the grabbing priority according to the seed link;

and (3) a task scheduler: the method is used for improving the efficiency and the speed of crawling network resources by a crawler, and a webpage crawler module adopts distributed deployment and needs to perform load balancing and parallel crawling operations on crawler tasks by a task scheduler;

a filter: the system is used for enriching a large amount of useless information and repeated contents of network contents, filtering, and improving the utilization rate of resource space and the running speed of a search engine;

crawler monitoring module: the system is used for ensuring the efficient operation of the crawler, monitoring the abnormal operation of the whole crawler module in real time and improving the operation reliability of the whole system.

Referring to fig. 3, the page content word segmentation module performs word segmentation on the web content by using a Conditional Random Field (CRF). When the CRF is a discriminant model modeled by conditional probabilities, and a logarithmic HMM model can be regarded as the CRF by using arbitrary weights, the weights of the feature functions are all less than or equal to 0 because of formal probabilities, and the probabilities also satisfy corresponding restrictions. The basic idea of CRF is to label Chinese characters, i.e. to form words (word groups) from characters, not only considering the frequency information of the occurrence of words, but also considering the context, and having better learning ability, so it has good effect on the recognition of ambiguous words and unknown words.

the conditional random field algorithm specifically comprises the following steps:

under the condition of giving a random variable X, a Markov random field of the random variable Y;

k1 transfer characteristics are provided, K2 state characteristics are provided, and K is K1+ K2;

wherein the transition feature tk is a feature function defined on the edge, dependent on the current position i and the previous position i-1; the state characteristic sl is a characteristic function defined on the node, dependent on the current position i; l is 1 to K2 status features; x is the observation sequence and y is the state sequence.

Summing all features at position i:

denote the weight of the feature fk (y, x) by wk, i.e.:

λ k is the weight of the transfer characteristic tk, μ l is the weight of the state characteristic sl;

the conditional random field is represented as:

Where P (y | x) is the conditional random field and Z (x) is the normalization factor, which is the sum of all possible values of y.

the webpage content relevance analysis module adopts BM25F algorithm to perform weight calculation on the results obtained by full text retrieval and the user query keywords to calculate the webpage content relevance, and re-orders the output results.

referring to fig. 2, the BM25F algorithm specifically includes:

Wherein, Bu represents the length condition of each domain, fiu represents the weighted sum of the ith word in each domain, fui represents the word frequency; bu is an adjusting factor of the length of each domain, ulu is the actual length of the domain, and uvulu is the average length of the domain; the calculation formula of the BM25F model is as follows:

in the formula, R represents the number of relevant documents, N represents the total number of documents, di ═ 1 represents words appearing in the documents, qi represents words queried by the user, ri represents the number of words appearing in the relevant documents, ni represents the number of words appearing in all the documents, and k1 is an empirical parameter.

the search engine relies on a quick index mechanism established by full-text retrieval, improves the accuracy of the index by combining a CRF algorithm, calculates the relevance of the webpage content according to the BM25F algorithm, and reorders the sequence of the search results, thereby improving the user experience.

the invention can effectively search information data on the Internet, more importantly, the invention can automatically adjust the analysis result, mark the reliability of the data source and finally output the query result which is most in line with the expectation of the user by the processing of the natural language technology and the research and judgment intervention behavior of the biography and marketing behavior rules. The method is suitable for industries making decisions based on information or intelligence by various services such as industry and commerce, public security, market supervision and the like.

such as: the search engine can be used for carrying out full search on the websites and analyzing the uploading and selling behaviors to obtain results quickly; according to the alarm description, quickly retrieving and tracking information of a certain website or forum; and quickly carrying out online public opinion investigation and the like on a certain market main body.

Referring to fig. 4, the intra-site search method of the present invention includes the following steps:

1) inputting a query keyword;

2) performing word segmentation operation on the keywords;

3) capturing corresponding network resources by using a webpage crawler module according to the word segmentation result of the keyword;

4) Performing word segmentation operation on the network resources captured by the crawler by using a page content word segmentation module, and storing the network resources in a storage;

5) establishing an index for the newly-put data by using a full-text retrieval module, and retrieving according to the keyword segmentation result;

6) analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) and the output results are reordered by utilizing the webpage content correlation analysis module, so that the matching accuracy is improved.

a computer-readable storage medium storing a computer program which, when executed by a processor, implements a method of:

1) inputting a query keyword;

2) performing word segmentation operation on the keywords;

3) capturing corresponding network resources according to the word segmentation result of the keyword;

4) performing word segmentation operation on the network resources captured by the crawler, and warehousing;

5) Establishing an index for the newly-put data, and searching according to the keyword segmentation result;

6) Analyzing the reimbursement behaviors in the returned results according to the query results returned by full-text retrieval by combining the reimbursement behavior rule base and the sample base;

7) And the output results are reordered, so that the matching accuracy is improved.

while the invention has been described in terms of its preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.