阅读说明：本技术 一种快速通用的埋点数据采集方法 (Rapid and universal buried point data acquisition method ) 是由 俞志坚 于 2020-04-27 设计创作，主要内容包括：本发明公开了一种快速通用的埋点数据采集方法,属于数据采集技术领域,包括如下步骤：S1：建立数据采集模块,通过数据采集模块提供数据接口,数据接口接收渠道上报的埋点原始数据；S2：建立渠道管理模块,将渠道管理模块与数据采集模块相连接,通过渠道管理模块为对应的渠道提供参数配置管理,以及数据处理分析的参数依据；S3：建立数据处理模块；S4：建立数据存储模块。采用固定数据上报接口上传参数并从渠道管理模块动态读取的方法,如果采集数据的需求有变,直接在对应的渠道管理模块里添加或修改需要上报的字段,无需再耗费大量的人力去开发新的接口,快速高效接收数据上报。(The invention discloses a quick and universal buried point data acquisition method, which belongs to the technical field of data acquisition and comprises the following steps: s1: establishing a data acquisition module, providing a data interface through the data acquisition module, and receiving the buried point original data reported by a channel through the data interface; s2: establishing a channel management module, connecting the channel management module with a data acquisition module, and providing parameter configuration management and parameter basis for data processing and analysis for a corresponding channel through the channel management module; s3: establishing a data processing module; s4: and establishing a data storage module. By adopting the method of uploading parameters through a fixed data reporting interface and dynamically reading the parameters from the channel management module, if the requirement of data acquisition is changed, the fields to be reported are directly added or modified in the corresponding channel management module, a large amount of manpower is not required to be consumed to develop a new interface, and the data report is quickly and efficiently received.)

1. A quick and universal buried point data acquisition method is characterized by comprising the following steps:

s1: establishing a data acquisition module, providing a data interface through the data acquisition module, and receiving the buried point original data reported by a channel through the data interface;

s2: establishing a channel management module, connecting the channel management module with a data acquisition module, and providing parameter configuration management and parameter basis for data processing and analysis for a corresponding channel through the channel management module;

s3: establishing a data processing module, connecting the channel management module with the data processing module, and filtering the data received by the channel management module through the data processing module;

s4: and establishing a data storage module, wherein the data storage module is connected with the data processing module, classifying and sequencing the data through the data storage module, and regularizing and persisting the data to lay a foundation for subsequent analysis and processing.

2. The fast and universal embedded data collection method as claimed in claim 1, wherein S1 sets embedded points of web pages, databases, terminals, clients in advance, and establishes channels between the embedded points of web pages, databases, terminals, clients and data collection modules.

3. The fast and universal buried point data acquisition method as claimed in claim 1, wherein the data acquisition module performs data transmission through 485 interface, the data processing module is connected with the network server through 7S5 serial port, and the data storage module is connected with the database through UART serial interface.

4. A rapid and universal buried point data acquisition method as claimed in claim 3, wherein the working voltage of the data acquisition module is dc3.5v or 5V, the working voltage of the data processing module is DC4-4.2V, and the working voltage of the data storage module is DC 3.3-5V.

5. The fast and universal buried point data acquisition method according to claim 1, wherein S1 comprises the steps of:

s101: the data acquisition module consists of a data acquisition interface, an interface verification unit, a data reporting unit and a deletion unit;

s102: the data acquisition interface is respectively connected with the channel and the interface verification unit in a butt joint mode, data information of each embedded point sequentially passes through the channel and the data acquisition interface transmission interface verification unit, and the interface verification unit screens out invalid data;

s103: the interface verification unit is respectively connected with the data reporting unit and the deletion unit in a butt joint mode, invalid data are directly deleted through the deletion unit, and screened valid data are transmitted to the channel management module through the data reporting unit.

6. The fast and universal buried point data acquisition method according to claim 1, wherein S1 comprises the steps of:

s101: the data acquisition module consists of a data acquisition interface, an interface verification unit, a data reporting unit, an invalid database and a periodic cleaning unit;

s102: the data acquisition interface is respectively connected with the channel and the interface verification unit in a butt joint mode, data information of each embedded point sequentially passes through the channel and the data acquisition interface transmission interface verification unit, and the interface verification unit screens out invalid data;

s103: the interface verification unit is respectively connected with the data reporting unit and the invalid database in a butt joint mode, invalid data are transmitted to the invalid database and stored in the invalid database in a sequencing mode according to dates, and screened valid data are transmitted to the channel management module by the data reporting unit;

s104: the invalid database is connected with the regular cleaning unit, the regular cleaning unit actively cleans all data beyond a certain period of time, and only part of invalid data is reserved in the invalid database.

7. The fast and universal buried point data acquisition method according to claim 1, wherein S2 comprises the steps of:

s201: the channel management module is composed of a data acquisition unit, a judgment unit, an execution unit and a deletion unit;

s202: the data acquisition unit is respectively connected with the judgment unit and the data acquisition module, the data acquisition unit dynamically acquires the buried point original data provided by the data acquisition module and transmits the data to the judgment unit, and the judgment unit judges the validity of the data;

s203: the judging unit is respectively connected with the executing unit and the deleting unit, the screened effective data are transmitted to the executing unit, the effective unit transmits the data to the data processing module, and the ineffective data are transmitted to the deleting unit to be directly deleted.

8. The fast and universal buried point data acquisition method according to claim 1, wherein S2 comprises the steps of:

s201: the channel management module is composed of a data acquisition unit, a judgment unit, an execution unit, an invalid database and a regular cleaning unit;

s202: the data acquisition unit is respectively connected with the judgment unit and the data acquisition module, the data acquisition unit dynamically acquires the buried point original data provided by the data acquisition module and transmits the data to the judgment unit, and the judgment unit judges the validity of the data;

s203: the judging unit is respectively connected with the execution unit and the invalid database, the screened valid data are transmitted to the execution unit, the valid unit transmits the data to the data processing module, and the invalid data are transmitted to the invalid database;

s204: the invalid database is connected with the regular cleaning unit, the regular cleaning unit actively cleans all data beyond a certain period of time, and only part of invalid data is reserved in the invalid database.

9. A rapid and versatile method of buried data collection as claimed in claim 1, wherein the data in S3 is processed by classification screening, clustering, transformation, similarity matching, frequent itemsets, statistical description, link prediction and causal analysis.

10. The fast and universal buried point data acquisition method according to claim 1, wherein S4 includes the following steps:

s401: the data storage module is composed of a data sorting unit, a data compression unit and a storage unit;

s402: the data processing module is connected with the data sorting unit, data entering the data storage module are sorted, compressed by the data compression unit after being sorted, the volume occupied by the data is reduced, and finally the data enter the storage unit to be stored persistently.

Technical Field

The invention relates to the technical field of data acquisition, in particular to a quick and universal buried point data acquisition method.

Background

With the rapid development of internet technology, big data has played a great advantage in human social practice, and the utilization value thereof exceeds our imagination, especially in recent years, the big data is increasingly applied to marketing analysis, and the formulation of marketing strategies, and the optimization of business processes, such as supply chain or distribution path optimization, and other scenes;

data is a foundation for realizing big data research, so how to quickly and efficiently acquire multi-channel and multi-dimensional data plays an extremely important role in big data application and research. The new retail background has fierce downlink industry competition, and the market information is changed constantly, so that the market information enters the era with the efficiency of king. How to promote data acquisition docking becomes crucial; in the new retail background, the digitization of users, the digitization of stores, the digitization of channels, the digitization of supply chains and the digitization of marketing are urgently needed to be realized; each link of new retail sales generates data which is the internal core driving force of new retail sales; in the face of various scene data, how to collect and uniformly collect the data becomes a problem which needs to be solved urgently.

Patent number CN201210044164.5 provides a data acquisition method and device, and a mobile terminal. The data acquisition method may be used in an apparatus comprising a plurality of data acquisition units, the method comprising: the method comprises the following steps that at least two data acquisition units simultaneously acquire preliminary data at respective acquisition frequencies according to an acquisition starting moment; acquiring the preliminary data acquired by the at least two data acquisition units according to a synchronous acquisition frequency to obtain synchronous data; wherein the synchronous acquisition frequency is a common divisor frequency of the respective acquisition frequencies of the at least two data acquisition units. According to the data acquisition method and device and the mobile terminal, the acquisition starting time of at least two data acquisition units is adjusted, and meanwhile, a mode of capturing synchronous data from preliminary data according to synchronous acquisition frequency is adopted, so that multiple groups of accurate data acquired by each data acquisition unit at the same time are obtained, and therefore errors can be avoided by an application program utilizing the multiple groups of accurate data.

Patent No. CN201711143888.4 provides a data acquisition method, which includes: configuring a plurality of preset query modules according to user query requirements, and generating a task configuration table according to the preset query modules and corresponding data source address information; acquiring a user instruction, analyzing a plurality of user query requirements related in the user instruction, and writing the user query requirements into a preset task record table; the task record table calls a corresponding preset query module in the task configuration table according to the user query requirement, and the query conditions are supplemented into the preset query module to obtain a customized query statement; and querying a database according to the customized query statement and the data source address information corresponding to the customized query statement to obtain target data.

However, conventional collection methods are used in both of the above patents: designing an acquisition mode aiming at each scene, and redefining and developing an acquisition interface when the requirement changes or new application scenes need to be added; docking and reporting are time-consuming and labor-consuming, and errors easily occur, so that the defects of the traditional acquisition mode are that the requirement is changeable, the retail scene is changed continuously, the data acquisition can be changed quickly according to the requirement, and the problem that the data acquisition is needed to be solved quickly is an important problem.

Disclosure of Invention

The invention aims to provide a quick and universal buried point data acquisition method, which adopts a method of uploading parameters through a fixed data reporting interface and dynamically reading the parameters from a channel management module, directly adds or modifies fields to be reported in the corresponding channel management module if the requirement of acquiring data changes, does not need to consume a large amount of manpower to develop a new interface, and quickly and efficiently receives data report so as to solve the problems in the background technology.

In order to achieve the purpose, the invention provides the following technical scheme:

a quick and universal buried point data acquisition method comprises the following steps:

s1: establishing a data acquisition module, providing a data interface through the data acquisition module, and receiving the buried point original data reported by a channel through the data interface;

s2: establishing a channel management module, connecting the channel management module with a data acquisition module, and providing parameter configuration management and parameter basis for data processing and analysis for a corresponding channel through the channel management module;

s3: establishing a data processing module, connecting the channel management module with the data processing module, and filtering the data received by the channel management module through the data processing module;

s4: and establishing a data storage module, wherein the data storage module is connected with the data processing module, classifying and sequencing the data through the data storage module, and regularizing and persisting the data to lay a foundation for subsequent analysis and processing.

Further, S1 sets a webpage embedded point, a database embedded point, a terminal device embedded point, and a client embedded point in advance, and establishes a channel between the webpage embedded point, the database embedded point, the terminal device embedded point, the client embedded point, and the data acquisition module.

Furthermore, the data acquisition module carries out data transmission through a 485 interface, the data processing module is connected with the network server through a 7S5 serial port, and the data storage module is connected with the database through a UART serial interface.

Furthermore, the working voltage of the data acquisition module is DC3.5V or 5V, the working voltage of the data processing module is DC4-4.2V, and the working voltage of the data storage module is DC 3.3-5V.

Further, S1 includes the following steps:

s101: the data acquisition module consists of a data acquisition interface, an interface verification unit, a data reporting unit and a deletion unit;

s102: the data acquisition interface is respectively connected with the channel and the interface verification unit in a butt joint mode, data information of each embedded point sequentially passes through the channel and the data acquisition interface transmission interface verification unit, and the interface verification unit screens out invalid data;

s103: the interface verification unit is respectively connected with the data reporting unit and the deletion unit in a butt joint mode, invalid data are directly deleted through the deletion unit, and screened valid data are transmitted to the channel management module through the data reporting unit.

Further, S1 includes the following steps:

s101: the data acquisition module consists of a data acquisition interface, an interface verification unit, a data reporting unit, an invalid database and a periodic cleaning unit;

s102: the data acquisition interface is respectively connected with the channel and the interface verification unit in a butt joint mode, data information of each embedded point sequentially passes through the channel and the data acquisition interface transmission interface verification unit, and the interface verification unit screens out invalid data;

s103: the interface verification unit is respectively connected with the data reporting unit and the invalid database in a butt joint mode, invalid data are transmitted to the invalid database and stored in the invalid database in a sequencing mode according to dates, and screened valid data are transmitted to the channel management module by the data reporting unit;

s104: the invalid database is connected with the regular cleaning unit, the regular cleaning unit actively cleans all data beyond a certain period of time, and only part of invalid data is reserved in the invalid database.

Further, S2 includes the following steps:

s201: the channel management module is composed of a data acquisition unit, a judgment unit, an execution unit and a deletion unit;

s202: the data acquisition unit is respectively connected with the judgment unit and the data acquisition module, the data acquisition unit dynamically acquires the buried point original data provided by the data acquisition module and transmits the data to the judgment unit, and the judgment unit judges the validity of the data;

s203: the judging unit is respectively connected with the executing unit and the deleting unit, the screened effective data are transmitted to the executing unit, the effective unit transmits the data to the data processing module, and the ineffective data are transmitted to the deleting unit to be directly deleted.

Further, S2 includes the following steps:

s201: the channel management module is composed of a data acquisition unit, a judgment unit, an execution unit, an invalid database and a regular cleaning unit;

s202: the data acquisition unit is respectively connected with the judgment unit and the data acquisition module, the data acquisition unit dynamically acquires the buried point original data provided by the data acquisition module and transmits the data to the judgment unit, and the judgment unit judges the validity of the data;

s203: the judging unit is respectively connected with the execution unit and the invalid database, the screened valid data are transmitted to the execution unit, the valid unit transmits the data to the data processing module, and the invalid data are transmitted to the invalid database;

s204: the invalid database is connected with the regular cleaning unit, the regular cleaning unit actively cleans all data beyond a certain period of time, and only part of invalid data is reserved in the invalid database.

Further, the data in S3 is processed by classification screening, clustering, transformation, similarity matching, frequent itemsets, statistical description, link prediction, and causal analysis.

Further, S4 includes the following steps:

s401: the data storage module is composed of a data sorting unit, a data compression unit and a storage unit;

s402: the data processing module is connected with the data sorting unit, data entering the data storage module are sorted, compressed by the data compression unit after being sorted, the volume occupied by the data is reduced, and finally the data enter the storage unit to be stored persistently.

Compared with the prior art, the invention has the beneficial effects that: aiming at the problem that the traditional data acquisition technical scheme can not quickly meet the variable and uncertain requirements of new retail, the invention provides a quick and universal buried point data acquisition method, which adopts a method of dynamically reading a fixed data reporting interface, an interface uploading parameter and a channel management module; if the requirement of the collected data is changed, directly adding or modifying fields needing to be reported in the corresponding channel management module; because the reporting parameters are uniquely determined according to the channel numbers, the requirement changes of different channels cannot influence each other; if a new channel scene exists, only a channel number and a field which is required to be reported by a corresponding channel need to be added in the channel management module, and a large amount of manpower is not required to be consumed to develop a new interface, so that data can be rapidly and efficiently received and reported.

Drawings

FIG. 1 is an overall structure diagram of the fast and general buried point data acquisition method of the present invention;

FIG. 2 is a topological diagram of a fast and general buried point data collection method of the present invention;

FIG. 3 is a block diagram of a data acquisition module of the fast and general buried data acquisition method of the present invention;

FIG. 4 is a block diagram of a channel management module of the fast and general buried point data collection method of the present invention;

FIG. 5 is a flow chart of a fast and general buried point data collection method of the present invention;

FIG. 6 is a flow chart of a data collection module of the fast and general buried point data collection method according to the first embodiment of the present invention;

FIG. 7 is a flow chart of a channel management module of the fast and general buried point data collection method according to the first embodiment of the present invention;

FIG. 8 is a flow chart of a data storage module of the fast and general buried point data collection method of the present invention;

FIG. 9 is a flow chart of a data collection module of the fast and general buried point data collection method according to the second embodiment of the present invention;

fig. 10 is a flow chart of a channel management module of the fast and general buried data collection method according to the second embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.