阅读说明：本技术 一种用于收集工业链数据的互联网系统 (Internet system for collecting industrial chain data ) 是由 张瑞磊 于 2021-09-03 设计创作，主要内容包括：本发明公开了一种用于收集工业链数据的互联网系统,该互联网系统旨在解决现有技术下不能从网页上收集需要的数据,且不能对数据收集到的数据进行归纳,无法形成工业链大数据库,不能为工业互联网和工业智能化提供有价值的数据支持的技术问题。该互联网系统包括数据收集系统和数据归纳分析系统,所述数据收集系统用于收集产品的工业链数据,所述新建任务模块根据用户的指令创建新的任务进程,从而去收集商品、服务及关联知识的详细信息。该互联网系统利用数据收集系统从网页上选择性的收集商品、服务及关联知识的详细信息,将收集到的详细信息归纳分析,从而形成工业链大数据库,为工业互联网、工业智能化提供有价值的数据支持。(The invention discloses an internet system for collecting industrial chain data, which aims to solve the technical problems that the required data cannot be collected from a webpage, the collected data cannot be summarized, an industrial chain large database cannot be formed, and valuable data support cannot be provided for industrial internet and industrial intelligence in the prior art. The internet system comprises a data collection system and a data induction analysis system, wherein the data collection system is used for collecting industrial chain data of products, and the newly-built task module is used for creating a new task process according to instructions of a user so as to collect detailed information of commodities, services and associated knowledge. The internet system selectively collects detailed information of commodities, services and associated knowledge from webpages by using a data collection system, and induces and analyzes the collected detailed information, so that a large industrial chain database is formed, and valuable data support is provided for industrial internet and industrial intelligence.)

1. An internet system for collecting industrial chain data, the internet system comprising a data collection system and a data summarization analysis system, wherein:

the data collection system is used for collecting industrial chain data of products, the data collection system comprises a controller, an analyzer and a resource library, the controller comprises a newly-built task module and a task allocation module, the newly-built task module creates a new task process according to instructions of a user so as to collect detailed information of commodities, services and associated knowledge, the commodities comprise industrial equipment, scientific instruments and industrial raw materials, the services comprise simple technical support, complete solution scheme providing and the like, the associated knowledge comprises case analysis, application scheme, national industry standard and the like related to the products/services, the task allocation module is responsible for allocating work tasks to all threads in multiple threads, the analyzer comprises a similarity judgment module, a webpage downloading module and a webpage processing module, the similarity judgment module calculates the correlation between a webpage theme and a user search theme, calculating a webpage fitness value, storing webpage links meeting the composite requirement of the matching degree of the webpages in the seed URL set into the URL set, generating a task to be crawled according to a URL result, downloading the matched webpages to the local according to the task by the webpage downloading module, processing Js script tags, css code contents, space characters and HTML tag contents in the downloaded webpages by the webpage processing module, wherein the resource library comprises a construction index module and a cache module, the construction index module allows a user to create indexes in a table for fast query of data, and the cache module stores the data;

the data induction analysis system is used for inducing the detailed information of the collected commodities, services and associated knowledge and then analyzing the detailed information to form a large industrial chain database, and comprises a data information input module, a data information induction analysis module and a data information storage module, wherein the data information input module is used for receiving various detailed information data collected by the data collection system, the data information induction analysis module is used for analyzing the data and classifying the collected data, and the data information storage module is used for storing the data processed by the data information induction analysis module.

2. The internet system for collecting data of industrial chain as claimed in claim 1, wherein the detailed information of the commodity in the data collection system includes the name specification of the commodity, the application industry, the product category under the industry, the role in the industry, the key indexes under the industry, the upstream and downstream commodities, the configuration scheme, the role in the industrial chain of the product and the related cases, the application scheme, the national industry standard, etc.

3. The internet system for collecting data of industrial chain as claimed in claim 1, wherein the service details in the data collection system include service title, service industry field, service details and related case, matching scheme, national industry standard, etc.

4. The internet system for collecting industrial chain data of claim 1, wherein the build indexing module is provided with a build indexing algorithm: the method comprises the steps of firstly, specifying the number of reduce of data, enabling a map process to detect an input format of an input file, calculating a key, then outputting, specifying a partition function, partitioning records, namely enabling the records to be distributed to each reduce process according to the requirements of respective projects, creating a writer of a search engine by each reduce process, continuously executing writer and add document operation, triggering commit operation in the process, correspondingly generating a plurality of index fragments, calling forced combination of the search engine to synthesize an index into a whole block, and obtaining index data.

5. The internet system for collecting industrial chain data of claim 1, wherein the cache module has disposed therein an LRU-K algorithm:

(a) data is accessed for the first time and added into an access history queue;

(b) if the data does not reach K times of access after being accessed in the history queue, the data is eliminated according to the rules (FIFO, LRU);

(c) when the access times of data in the access history queue reach K times, deleting the data index from the history queue, moving the data to a cache queue, caching the data, and reordering the cache queue according to time;

(d) re-ordering after the cache data queue is accessed again;

(e) when the data needs to be eliminated, eliminating the data arranged at the tail end in the cache queue, namely: the data "last Kth visit oldest now" is eliminated.

6. The internet system for collecting data of an industrial chain according to claim 1, wherein the similarity determination module is provided with a determination rule: calculating the correlation between the webpage theme and the theme searched by the user, calculating a webpage adaptability value S, comparing the webpage adaptability value S with a preset webpage adaptability threshold value Y, when S is larger than or equal to Y, the webpage adaptability value S of the webpage meets the requirement, storing the webpage link into a URL set, generating a task to be crawled according to the URL result, when S is smaller than or equal to Y, the webpage adaptability value S of the webpage does not meet the requirement, and the webpage link set does not need to be stored into the URL set.

7. The internet system for collecting data of industrial chain as claimed in claim 6, wherein the web page download module comprises a selection unit, a cross unit and a mutation unit, the selection unit selects the web page links meeting the requirement from the seed URL set according to the determination rule, the cross unit is used for selecting the web page links not stored in the RUL set so as to expand the search range, and the mutation unit retrieves the web page links according to the web page links obtained by the cross unit.

8. The internet system for collecting industrial chain data as claimed in claim 1, wherein the data information induction analysis module is preset with classification rules: the rule is represented by a disjunctive paradigm R ═ R1 v2 v … v rk, R is the rule set, ri is the classification rule, each classification rule ri is (condition i) → yi, the left side of the rule becomes the rule antecedent or premise, which is the conjunctive of the attribute tests: the condition i is (a1 op v1) Λ (a2 op v2) Λ … a (A3 op v3), wherein (Aj, vj) are attribute value pairs, op is a comparison operator, and is selected from the set { } not, <, > ≧ and ≧ respectively, each attribute test (Aj op vj) is called an conjunction item, the right side of the rule is called a rule back piece, and contains a prediction class yi, if the attributes of the front piece of the rule r and the record x match, it is called r to cover x, when r covers a given record, it is activated or triggered, the quality of the classification rule can be measured by a coverage rate (coverage) and an accuracy rate (accuracy), given data set D and classification rule r: a → y, the coverage rate of the rule is defined as the proportion of the records of the trigger rule r in D, the accuracy or confidence factor is defined as the proportion of the records with class labels equal to y in the records of the trigger rule r, coverage (r) | a |/| D |, accuracy (r) | a |, y |/| a |, where | a | is the number of records satisfying the rule front piece, | a |, y | is the number of records satisfying both the rule front piece and the rule back piece, and D is the total number of records.

Technical Field

The invention belongs to the technical field of computer networks and data processing, and particularly relates to an internet system for collecting industrial chain data.

Background

The internet has been deeply buried in the aspects of our lives, the concept of big data is not strange due to the penetration of the internet, the value of the big data is more and more obvious, along with the development of internet technology, a plurality of world-level internet companies appear, the big data create brilliant values in respective fields and collect valuable data at the same time, the development of the internet is found to be mainly the development in the aspects of clothes, eating and drinking of our country and abroad, while the development in the aspect of an industrial chain is slow, the development of the industrial internet starts to rise, but the development of the industrial internet is limited, a unified system is not provided, various industrial devices can be linked as much as possible, and the industrial chain data is mainly lacked.

Currently, the invention patent with patent number CN201010102378.4 discloses a method for actively collecting data, which includes: defining an index structure of data to be collected in a database of a server; the index structure includes: index name, and corresponding index ID and acquisition mode; searching a corresponding index ID and an acquisition mode in a database of a server according to the name of a monitoring index customized by a user for a monitoring terminal; and the server accesses the monitoring terminal according to the acquisition mode and collects data information corresponding to the index ID. Preferably, the acquiring means includes: SNMNP mode, WMI mode, Web services mode or access oracle database mode. Preferably, the server collects data to a plurality of monitoring terminals; the server records the terminal ID of each monitoring terminal and accesses the appointed monitoring terminal according to the terminal ID. Specifically, when the acquisition mode is the WMI mode, the server collects corresponding data information by calling a dll dynamic library file. Specifically, when the obtaining mode is a Web services mode, the server calls a corresponding interface according to an interface form issued by the monitoring terminal, and collects corresponding data information. Specifically, when the obtaining mode is a mode of accessing an oracle database, the server obtains the names of the corresponding performance table and view from the mib of the monitoring terminal, accesses the performance table and view, and collects corresponding data information. In addition, the invention also discloses a server for actively collecting data, which comprises: the index definition unit is used for defining an index structure of the data to be collected in a database of the server; the index structure includes: index name, and corresponding index ID and acquisition mode; the index searching unit is used for searching a corresponding index ID and an acquisition mode in a database of the server according to a monitoring index customized by a user for the monitoring terminal; and the data collection unit is used for accessing the monitoring terminal according to the acquisition mode and collecting data information corresponding to the index ID. The server can collect data to a plurality of monitoring terminals through the terminal ID, but the server cannot collect required data from a webpage and can not summarize the data collected by the data, so that a large industrial chain database cannot be formed, and valuable data support cannot be provided for industrial internet and industrial intelligence.

Therefore, in order to solve the problem that the data cannot be analyzed and summarized to form a large database of an industrial chain, a solution is needed to improve the use scenario of the internet system.

Disclosure of Invention

(1) Technical problem to be solved

Aiming at the defects of the prior art, the invention aims to provide an internet system for collecting industrial chain data, which aims to solve the technical problems that the required data cannot be collected from a webpage, the collected data cannot be summarized, a large industrial chain database cannot be formed, and valuable data support cannot be provided for industrial internet and industrial intelligence in the prior art.

(2) Technical scheme

In order to solve the above technical problems, the present invention provides an internet system for collecting data of an industrial chain, the internet system comprising a data collection system and a data induction analysis system,

the data collection system is used for collecting industrial chain data of products, the data collection system comprises a controller, an analyzer and a resource library, the controller comprises a newly-built task module and a task allocation module, the newly-built task module creates a new task process according to instructions of a user so as to collect detailed information of commodities, services and associated knowledge, the commodities comprise industrial equipment, scientific instruments, industrial raw materials and the like, the services comprise simple technical support, complete solution provision and the like, the associated knowledge comprises case analysis, application schemes, national industry standards and the like related to the products/services, the task allocation module is responsible for allocating work tasks to all threads in multiple threads, the analyzer comprises a similarity judgment module, a webpage downloading module and a webpage processing module, the similarity judgment module calculates the correlation between a webpage theme and a user search theme, calculating a webpage fitness value, storing webpage links meeting the composite requirement of the matching degree of the webpages in the seed URL set into the URL set, generating a task to be crawled according to a URL result, downloading the matched webpages to the local according to the task by the webpage downloading module, processing Js script tags, css code contents, space characters and HTML tag contents in the downloaded webpages by the webpage processing module, wherein the resource library comprises a construction index module and a cache module, the construction index module allows a user to create indexes in a table for fast query of data, and the cache module stores the data;

the data induction analysis system is used for inducing the detailed information of the collected commodities, services and associated knowledge and then analyzing the detailed information to form a large industrial chain database, and comprises a data information input module, a data information induction analysis module and a data information storage module, wherein the data information input module is used for receiving various detailed information data collected by the data collection system, the data information induction analysis module is used for analyzing the data and classifying the collected data, and the data information storage module is used for storing the data processed by the data information induction analysis module.

Preferably, the detailed information of the commodities in the data collection system includes the name specification of the commodity, the application industry, the product category under the industry, the role in the industry, the key indexes under the industry, the upstream and downstream commodities, the configuration scheme, the role in the product industry chain and the related cases, the application scheme, the national industry standard and the like.

Preferably, the service details in the data collection system include service titles, service industry fields, service details and related cases, supporting schemes, national industry standards, and the like.

Preferably, an index building algorithm is arranged in the index building module: the method comprises the steps of firstly, specifying the number of reduce of data, enabling a map process to detect an input format of an input file, calculating a key, then outputting, specifying a partition function, partitioning records, namely enabling the records to be distributed to each reduce process according to the requirements of respective projects, creating a writer of a search engine by each reduce process, continuously executing writer and add document operation, triggering commit operation in the process, correspondingly generating a plurality of index fragments, calling forced combination of the search engine to synthesize an index into a whole block, and obtaining index data.

Preferably, an LRU-K algorithm is provided in the cache module:

(a) data is accessed for the first time and added into an access history queue;

(b) if the data does not reach K times of access after being accessed in the history queue, the data is eliminated according to the rules (FIFO, LRU);

(c) when the access times of data in the access history queue reach K times, deleting the data index from the history queue, moving the data to a cache queue, caching the data, and reordering the cache queue according to time;

(d) re-ordering after the cache data queue is accessed again;

(e) when the data needs to be eliminated, eliminating the data arranged at the tail end in the cache queue, namely: the data "last Kth visit oldest now" is eliminated.

Preferably, a judgment rule is set in the similarity judgment module: calculating the correlation between the webpage theme and the theme searched by the user, calculating a webpage adaptability value S, comparing the webpage adaptability value S with a preset webpage adaptability threshold value Y, when S is larger than or equal to Y, the webpage adaptability value S of the webpage meets the requirement, storing the webpage link into a URL set, generating a task to be crawled according to the URL result, when S is smaller than or equal to Y, the webpage adaptability value S of the webpage does not meet the requirement, and the webpage link set does not need to be stored into the URL set.

Preferably, the web page downloading module includes a selecting unit, a crossing unit and a mutation unit, the selecting unit selects a web page link meeting requirements from the seed URL set according to a determination rule, the crossing unit is configured to select a web page link that is not stored in the RUL set, so as to expand a search range, and the mutation unit performs retrieval again according to the web page link obtained by the crossing unit.

Preferably, the data information induction analysis module is internally preset with classification rules: the rule is represented by a disjunctive paradigm R ═ R1 v2 v … v rk, R is the rule set, ri is the classification rule, each classification rule ri is (condition i) → yi, the left side of the rule becomes the rule antecedent or premise, which is the conjunctive of the attribute tests: the condition i is (a1 op v1) Λ (a2 op v2) Λ … a (A3 op v3), wherein (Aj, vj) are attribute value pairs, op is a comparison operator, and is selected from the set { } not, <, > ≧ and ≧ respectively, each attribute test (Aj op vj) is called an conjunction item, the right side of the rule is called a rule back piece, and contains a prediction class yi, if the attributes of the front piece of the rule r and the record x match, it is called r to cover x, when r covers a given record, it is activated or triggered, the quality of the classification rule can be measured by a coverage rate (coverage) and an accuracy rate (accuracy), given data set D and classification rule r: a → y, the coverage rate of the rule is defined as the proportion of the records of the trigger rule r in D, the accuracy or confidence factor is defined as the proportion of the records with class labels equal to y in the records of the trigger rule r, coverage (r) | a |/| D |, accuracy (r) | a |, y |/| a |, where | a | is the number of records satisfying the rule front piece, | a |, y | is the number of records satisfying both the rule front piece and the rule back piece, and D is the total number of records.

(3) Advantageous effects

Compared with the prior art, the invention has the beneficial effects that: the Internet system selectively collects detailed information of commodities, services and associated knowledge from webpages by using a data collection system, screens webpage links by using a similarity judgment module, stores the webpage links meeting the composite requirement of the matching degree of the webpages in a seed URL set into a URL set, generates a task to be crawled according to a URL result, downloads the webpages, and enables the efficiency of data query to be higher by constructing an index module component data index.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the technical solutions in the prior art will be briefly described below, it is obvious that the drawings in the following description are only one embodiment of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic diagram of an overall framework architecture of an embodiment of the Internet system of the invention;

FIG. 2 is a flowchart illustrating operation of an embodiment of the present invention.

Detailed Description

In order to make the technical means, the original characteristics, the achieved purposes and the effects of the invention easily understood and obvious, the technical solutions in the embodiments of the present invention are clearly and completely described below to further illustrate the invention, and obviously, the described embodiments are only a part of the embodiments of the present invention, but not all the embodiments.

Example 1

The embodiment is an internet system for collecting industrial chain data, the overall framework structure of which is schematically shown in fig. 1, the internet system comprises a data collection system and a data induction analysis system,

the data collection system is used for collecting industrial chain data of a product, the data collection system comprises a controller, an analyzer and a resource library, the controller comprises a newly-built task module and a task allocation module, the newly-built task module creates a new task process according to an instruction of a user so as to collect detailed information of commodities, services and associated knowledge, the commodities comprise industrial equipment, scientific instruments, industrial raw materials and the like, the services comprise simple technical support, complete solution scheme provision and the like, the associated knowledge comprises case analysis, application scheme, national industry standard and the like related to the product/service, the task allocation module is responsible for allocating work tasks to each thread in multiple threads, the analyzer comprises a similarity judgment module, a webpage downloading module and a webpage processing module, the similarity judgment module calculates the correlation between a webpage theme and a user search theme, calculating a webpage fitness value, storing webpage links meeting the composite requirement of the matching degree of the webpages in the seed URL set into the URL set, generating a task to be crawled according to a URL result, downloading the matched webpages to the local according to the task by a webpage downloading module, processing Js script tags, css code contents, space characters and HTML tag contents in the downloaded webpages by a webpage processing module, and storing data by a cache module, wherein the resource library comprises an index building module and a cache module, the index building module allows a user to build indexes in a table for fast query of the data, and the cache module stores the data;

the data induction analysis system is used for inducing the detailed information of the collected commodities, services and associated knowledge and then analyzing the detailed information to form a large industrial chain database, and comprises a data information input module, a data information induction analysis module and a data information storage module, wherein the data information input module is used for receiving various detailed information data collected by the data collection system, the data information induction analysis module is used for analyzing the data and classifying the collected data, and the data information storage module is used for storing the data processed by the data information induction analysis module.

The detailed commodity information in the data collection system comprises the name specification of the commodity, the application industry, the product category under the industry, the action in the industry, key indexes applicable to the industry, upstream and downstream commodities, configuration schemes, roles in a product industrial chain, related cases, application schemes, national industry standards and the like, and the detailed service content in the data collection system comprises service titles, service industry fields, service details, related cases, matching schemes, national industry standards and the like.

Meanwhile, an index building algorithm is arranged in the index building module: the method comprises the steps of firstly, specifying the number of reduce of data, enabling a map process to detect an input format of an input file, calculating a key, outputting, specifying a partition function, partitioning records, namely enabling the records to be distributed to each reduce process according to the requirements of respective projects, establishing a writer of a search engine by each reduce process, continuously executing writer and add document operation, triggering commit operation in the process, correspondingly generating a plurality of index fragments, calling forced combination of the search engine to synthesize an index into a whole block to obtain index data, and arranging an LRU-K algorithm in a cache module:

(a) data is accessed for the first time and added into an access history queue;

(b) if the data does not reach K times of access after being accessed in the history queue, the data is eliminated according to the rules (FIFO, LRU);

(c) when the access times of data in the access history queue reach K times, deleting the data index from the history queue, moving the data to a cache queue, caching the data, and reordering the cache queue according to time;

(d) re-ordering after the cache data queue is accessed again;

(e) when the data needs to be eliminated, eliminating the data arranged at the tail end in the cache queue, namely: the data "last Kth visit oldest now" is eliminated.

In addition, a judgment rule is arranged in the similarity judgment module: the method comprises the steps of calculating the correlation between a webpage theme and a user search theme, calculating a webpage fitness value S, comparing the webpage fitness value S with a preset webpage fitness threshold value Y, when S is larger than or equal to Y, the webpage fitness value S of the webpage meets requirements, storing webpage links into a URL set, generating a task to be crawled according to a URL result, when S is smaller than or equal to Y, the webpage fitness value S of the webpage does not meet the requirements, the webpage links do not need to be stored into the URL set, a webpage downloading module comprises a selecting unit, a crossing unit and a variation unit, the selecting unit selects the webpage links meeting the requirements from a seed URL set according to a judgment rule, the crossing unit is used for selecting the webpage links which are not stored into the RUL set, and therefore the search range is expanded, and the variation unit retrieves again according to the webpage links obtained by the crossing unit.

In addition, classification rules are preset in the data information induction analysis module: the rule is represented by a disjunctive paradigm R ═ R1 v2 v … v rk, R is the rule set, ri is the classification rule, each classification rule ri is (condition i) → yi, the left side of the rule becomes the rule antecedent or premise, which is the conjunctive of the attribute tests: the condition i is (a1 op v1) Λ (a2 op v2) Λ … a (A3 op v3), wherein (Aj, vj) are attribute value pairs, op is a comparison operator, and is selected from the set { } not, <, > ≧ and ≧ respectively, each attribute test (Aj op vj) is called an conjunction item, the right side of the rule is called a rule back piece, and contains a prediction class yi, if the attributes of the front piece of the rule r and the record x match, it is called r to cover x, when r covers a given record, it is activated or triggered, the quality of the classification rule can be measured by a coverage rate (coverage) and an accuracy rate (accuracy), given data set D and classification rule r: a → y, the coverage rate of the rule is defined as the proportion of the records of the trigger rule r in D, the accuracy or confidence factor is defined as the proportion of the records with class labels equal to y in the records of the trigger rule r, coverage (r) | a |/| D |, accuracy (r) | a |, y |/| a |, where | a | is the number of records satisfying the rule front piece, | a |, y | is the number of records satisfying both the rule front piece and the rule back piece, and D is the total number of records.

The work flow diagram of the internet system is shown in fig. 2.

TABLE 1

Having thus described the principal technical features and basic principles of the invention, and the advantages associated therewith, it will be apparent to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, but is capable of other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.

Furthermore, it should be understood that although the present description is described in terms of various embodiments, not every embodiment includes only a single embodiment, and such descriptions are provided for clarity only, and those skilled in the art will recognize that the embodiments described herein can be combined as a whole to form other embodiments as would be understood by those skilled in the art.