阅读说明：本技术 一种日志采集系统及其数据传输方法、装置 (A kind of Log Collect System and its data transmission method, device ) 是由 李明 李晓宇 张月鹏 张伟东 裴广超 刘立超 于 2019-07-25 设计创作，主要内容包括：本申请实施例公开了一种日志采集系统及其数据传输方法、装置。日志采集系统的数据传输方法包括：在日志采集系统Flume的Sink组件中配置与业务需求对应的Sink扩展组件；利用各所述Sink扩展组件从所述Sink组件中获取原始数据,并按照预设业务处理规则对所述原始数据进行处理,得到目标数据；将所述目标数据传输到相应的数据接收方。通过增加Sink扩展组件,利用Sink扩展组件按照业务要求对原始数据进行处理,提取需要的、可用的数据向后传输,避免了后端服务器IO流瓶颈以及数据积压问题。(The embodiment of the present application discloses a kind of Log Collect System and its data transmission method, device.The data transmission method of Log Collect System includes: that Sink extension element corresponding with business demand is configured in the Sink component of Log Collect System Flume；Initial data is obtained from the Sink component using each Sink extension element, and handles rule according to pre-set business and the initial data is handled, and obtains target data；The target data is transferred to corresponding data receiver.By increasing Sink extension element, initial data is handled according to business need using Sink extension element, extract needs, available data transmit backward, avoid back-end server IO stream bottleneck and data overstock problem.)

1. a kind of data transmission method of Log Collect System characterized by comprising

Sink extension element corresponding with business demand is configured in the Sink component of Log Collect System Flume；

Initial data is obtained from the Sink component using each Sink extension element, and handles rule according to pre-set business The initial data is handled, target data is obtained；

The target data is transferred to corresponding data receiver.

2. the method as described in claim 1, which is characterized in that described to match in the Sink component of Log Collect System Flume Setting Sink extension element corresponding with business demand includes:

Create a frame clsss, the AbstractSink abstract class of Log Collect System Flume described in the frame Similar integral, and Realize the Configurable interface of the Log Collect System Flume；

Common interface is defined, includes multiple business processing method in the common interface；

Sink extension element is created, the Sink extension element inherits the frame clsss, and realizes the common interface.

3. method according to claim 2, which is characterized in that described to match in the Sink component of Log Collect System Flume Setting Sink extension element corresponding with business demand includes:

According to data filtering business demand, configuration and the data filtering industry in the Sink component of Log Collect System Flume The corresponding Sink filter assemblies of business demand；

It is described to obtain initial data from the Sink component using each Sink extension element, and handled according to pre-set business Rule handles the initial data, obtains target data and includes:

Initial data is obtained from the Sink component by the frame clsss using the Sink filter assemblies, and passes through the public affairs The filter traffic processing method in interface handles the initial data altogether, obtains target data；

Wherein, the filter traffic processing method is for filtering invalid data, filtering spam data, in filtering black list data It is one or more.

4. the method as described in claim 1, which is characterized in that described to match in the Sink component of Log Collect System Flume Setting Sink extension element corresponding with business demand includes:

According to first business demand and the second business demand with stream compression relationship, Log Collect System Flume's The first Sink extension element corresponding with first business demand is configured in Sink component；And according to the second business demand, In The 2nd Sink extension element corresponding with second business demand, institute are configured in the Sink component of Log Collect System Flume State the upper level Sink extension element that the first Sink extension element is the 2nd Sink extension element；

It is described to obtain initial data from the Sink component using each Sink extension element, and handled according to pre-set business Rule handles the initial data, obtains target data and includes:

Initial data is obtained from the Sink component using the first Sink extension element, and handles rule according to pre-set business The initial data is handled；

Using the 2nd Sink extension element from the first Sink extension element acquisition process data, and according to pre-set business Processing rule handles the process data, obtains target data；Wherein, the first Sink extension element is filtering group Part, the 2nd Sink extension element are modular unit.

5. a kind of data transmission device of Log Collect System, which is characterized in that the device includes:

Component expansion module, for configuring Sink corresponding with business demand in the Sink component of Log Collect System Flume Extension element；

Data processing module for obtaining initial data from the Sink component using each Sink extension element, and is pressed The initial data is handled according to pre-set business processing rule, obtains target data；

Transmission module, for the target data to be transferred to corresponding data receiver.

6. device as claimed in claim 5, which is characterized in that the component expansion module is specifically used for one frame of creation Class, the AbstractSink abstract class of Log Collect System Flume described in the frame Similar integral, and realize the log collection The Configurable interface of system Flume；Common interface is defined, includes multiple business processing method in the common interface； Sink extension element is created, the Sink extension element inherits the frame clsss, and realizes the common interface.

7. device as claimed in claim 6, which is characterized in that the component expansion module is specifically used for according to data filtering Business demand configures Sink corresponding with the data filtering business demand in the Sink component of Log Collect System Flume Filter assemblies；

The data processing module is specifically used for passing through the frame clsss from the Sink component using the Sink filter assemblies Initial data is obtained, and the initial data is handled by the filter traffic processing method in the common interface, is obtained To target data；Wherein, the filter traffic processing method is for filtering invalid data, filtering spam data, filtering black list It is one or more in data.

8. device as claimed in claim 5, which is characterized in that the component expansion module is specifically used for according to data The first business demand and the second business demand of circulation relationship, configuration and institute in the Sink component of Log Collect System Flume State the corresponding first Sink extension element of the first business demand；And according to the second business demand, Log Collect System Flume's The 2nd Sink extension element corresponding with second business demand is configured in Sink component, the first Sink extension element is The upper level Sink extension element of the 2nd Sink extension element；

The data processing module is specifically used for obtaining original number from the Sink component using the first Sink extension element According to, and handle rule according to pre-set business and the initial data is handled；Using the 2nd Sink extension element from institute The first Sink extension element acquisition process data are stated, and handles rule according to pre-set business and the process data is handled, Obtain target data；Wherein, the first Sink extension element is filter assemblies, and the 2nd Sink extension element is standardization Component.

9. a kind of Log Collect System, wherein the system includes: client and log acquisition server；

It include the data transmission of the Log Collect System as described in any one of claim 5-8 in the log acquisition server Device.

10. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the calculating The step of any one of claim 1-4 the method is realized when machine program is executed by processor.

Technical field

This application involves field of computer technology, and in particular to a kind of Log Collect System and its data transmission method, dress It sets.

Background technique

Flume is a High Availabitity, highly reliable, distributed massive logs acquisition, polymerization and the system transmitted. Flume, which is provided, carries out simply dealt ability to data, and the core of Flume operation is Agent, and Flume is the smallest with Agent Independent operating unit.One Agent is exactly a JVM (Java Virtual Machine).It is a complete data gathering tool, is contained Three core components, are Source, Channel, Sink respectively.By these components, Event (event) can be from a place Another place is flowed to, as shown in Figure 1, Source is the collecting terminal of data, is responsible for data capture post package to event (event) inner, then event is pushed into Channel, Channel (pipeline) be for transmitting data, Sink component from Event is taken out in Channel, and data are then dealt into other places and (for example is distributed to file system, database, can also be distributed to it At the Source of his Agent).The direction of arrow represents the flow direction of data in Fig. 1, for example data are from Web server -- > Source-- > Channel-- > Sink (component) -- the storage of > data, the part that the centre of Fig. 1 is framed by rectangle is one Agent。

Since existing Flume is mainly used for collection of log data and transmission, in order to guarantee log data acquisition efficiency, its is right The processing of daily record data is substantially unrelated with business, when the data volume of transmission is larger, such as per second tens data volume, it is easy It causes back end database server or service server to be unable to the data of real-time consumption processing Flume transmission, causes data product The problems such as pressure, server is seemingly-dead.

Summary of the invention

In view of this, the embodiment of the present application provides a kind of data transmission method of Log Collect System, device, pass through increasing Add Sink extension element so that Log Collect System is handled initial data according to business need, extract needs, it is available Data transmit backward, avoid back-end server IO stream bottleneck and data overstock problem.

According to the one aspect of the application, a kind of data transmission method of Log Collect System is provided, comprising:

Sink extension element corresponding with business demand is configured in the Sink component of Log Collect System Flume；

Initial data is obtained from the Sink component using each Sink extension element, and is handled according to pre-set business Rule handles the initial data, obtains target data；

The target data is transferred to corresponding data receiver.

According to further aspect of the application, a kind of data transmission device of Log Collect System is provided, comprising:

Component expansion module, it is corresponding with business demand for being configured in the Sink component of Log Collect System Flume Sink extension element；

Data processing module, for obtaining initial data from the Sink component using each Sink extension element, And handle rule according to pre-set business and the initial data is handled, obtain target data；

Transmission module, for the target data to be transferred to corresponding data receiver.

According to the another aspect of the application, a kind of Log Collect System is provided, wherein the system includes: client And log acquisition server；It include the log collection system as described in further aspect of the application in the log acquisition server The data transmission device of system.

According to another aspect of the application, a kind of non-transient computer readable storage medium is provided, is stored thereon There is computer program, which realizes one aspect the method for the application when being executed by processor the step of.

The utility model has the advantages that the data transmission scheme of the Log Collect System using the embodiment of the present application, in Log Collect System Corresponding with business demand Sink extension element is configured in the Sink component of Flume, using each Sink extension element from Sink group It obtains initial data in part, and handles rule according to pre-set business and initial data is handled, obtain mesh after target data Mark data are transferred to corresponding data receiver, thus advance in Sink component business process rule and handle, avoid Data are not made with business processing for Sink component but back-end server pressure caused by simple transmission is big, data overstock, server Seemingly-dead problem.

Detailed description of the invention

Fig. 1 is the data transmission schematic diagram of prior art Flume system；

Fig. 2 is the flow chart of the data transmission method of the Log Collect System of the embodiment of the present application one；

Fig. 3 is the flow chart of the data transmission method of the Log Collect System of the embodiment of the present application two；

Fig. 4 is the flow chart of the data transmission method of the Log Collect System of the embodiment of the present application three；

Fig. 5 is the flow chart of the data transmission method of the Log Collect System of the embodiment of the present application four；

Fig. 6 is the flow chart of the data transmission method of the Log Collect System of the embodiment of the present application five；

Fig. 7 is the block diagram of the data transmission device of the Log Collect System of the application one embodiment；

Fig. 8 is the block diagram of the Log Collect System of the application one embodiment；

Fig. 9 is the structural schematic diagram of the non-transient computer readable storage medium of the application one embodiment.

Specific embodiment

To keep the above objects, features, and advantages of the embodiment of the present application more obvious and easy to understand, with reference to the accompanying drawing and Specific embodiment is described in further detail the embodiment of the present application.Obviously, described embodiment is the application one Divide embodiment, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not making Every other embodiment obtained under the premise of creative work belongs to the range of the embodiment of the present application protection.