阅读说明：本技术 一种确保客户端数据采集准确性的方法及客户端 (Method for ensuring data acquisition accuracy of client and client ) 是由 孙夺 施王兴 杜晓祥 于 2021-01-15 设计创作，主要内容包括：本发明实施例公开了一种确保客户端数据采集准确性的方法及客户端,所述方法包括：向服务器端发送请求以获取在服务器端预先配置的需要采集的事件以及事件对应的优先级；将生成的不同优先级的事件数据在内存中进行序列化并存储到磁盘上；由所述磁盘中读取数据并按照优先级从高到低的顺序依次将数据放入待发送队列后发送至服务器端,若发生通信连接中断导致数据发送失败,则将发送失败的数据存储至磁盘中待下次重新发送。采用了数据实时持久化以及多级缓存的方式,在客户端程序崩溃、异常或者网络状况不佳的情况下,能够保证数据不丢失,在程序再次启动时可以将之前缓存的数据发送出来,确保数据采集准确性和完整性。(The embodiment of the invention discloses a method for ensuring the data acquisition accuracy of a client and the client, wherein the method comprises the following steps: sending a request to a server to acquire an event to be acquired and a priority corresponding to the event, wherein the event is pre-configured at the server; serializing the generated event data with different priorities in a memory and storing the event data on a magnetic disk; and reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the queue to a server, and if the data is failed to be sent due to the interruption of the communication connection, storing the data failed to be sent into the disk for the next resending. The data real-time persistence and multi-level caching are adopted, so that the data can be ensured not to be lost under the condition that a client program is crashed, abnormal or the network condition is not good, the cached data can be sent out when the program is restarted, and the data acquisition accuracy and integrity are ensured.)

1. A method of ensuring client data acquisition accuracy, the method comprising:

sending a request to a server to acquire an event to be acquired and a priority corresponding to the event, wherein the event is pre-configured at the server;

serializing the generated event data with different priorities in a memory and storing the event data on a magnetic disk;

and reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the queue to a server, and if the data is failed to be sent due to the interruption of the communication connection, storing the data failed to be sent into the disk for the next resending.

2. The method according to claim 1, wherein the data is read from the disk, and the data is sequentially placed into a queue to be sent according to a sequence of priorities from high to low, and then sent to the server, and specifically, the method further comprises:

and sending data according to a preset time interval, limiting the maximum data volume sent each time according to the preset maximum occupied network bandwidth, and if the maximum occupied network bandwidth exceeds the limited network bandwidth quota, reserving the data for next sending.

3. The method according to claim 1, wherein the data is read from the disk, and the data is sequentially placed into a queue to be sent according to a sequence of priorities from high to low, and then sent to the server, and specifically, the method further comprises:

and for different events with the same priority, sequentially putting the data into a queue to be sent according to the sequence of the event end timestamps and then sending the data to the server.

4. The method according to claim 2, wherein the data is read from the disk, and the data is sequentially placed into a queue to be sent according to a sequence of priorities from high to low, and then sent to the server, and specifically, the method further comprises:

in each data sending process, if data sending failure exists, the data with the sending failure is stored in a sending failure cache file, the failed data is sent preferentially when the data are sent, and then the data are sent according to the priority sequence.

5. The method according to claim 2, wherein if the data transmission fails due to a communication connection interruption, the data that has failed to be transmitted is stored in a disk for a next retransmission, and the method further comprises:

in each data sending process, the data under the current communication connection is preferentially sent, and on the premise that the data volume does not exceed the limited network bandwidth quota each time, the failed data under the old connection generated by the interruption of the previous communication connection is sent again.

6. The method according to claim 1, wherein the generated data with different priorities are serialized in a memory and stored on a disk, and further comprising:

and performing file division processing on the stored data files with different priorities, setting the size of a single file, and if the size of the single file exceeds the limit, establishing a new file to continuously store data.

7. The method of claim 1, wherein the method further comprises:

checking the occupation of the memory and the disk according to a preset time interval, if the occupation of the memory or the disk exceeds the preset maximum occupation memory space or the maximum occupation disk space, preferentially discarding the failure data under the old connection generated by the interruption of the previous communication connection, then discarding the data according to the sequence from low to high in priority, and discarding the data according to the sequence from first to last of different events under the same priority.

8. The method according to claim 1, wherein sending a request to a server to obtain an event to be collected and a priority corresponding to the event, which are pre-configured at the server, further comprises:

and under the condition that the network is available, sending a ping request to the server side according to a preset time interval, and if the server side event is found to be configured with an update, sending an update request to update the local data acquisition strategy.

9. A client, the client comprising:

the request module is used for sending a request to the server to acquire an event which is required to be acquired and is pre-configured at the server and the priority corresponding to the event;

the data storage module is used for serializing the generated event data with different priorities in the memory and storing the event data on a magnetic disk;

and the data sending module is used for reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the data to the server, and if the data sending fails due to the communication connection interruption, storing the data which is failed to be sent into the disk for the next resending.

10. A computer storage medium comprising one or more program instructions for execution by a client of the method of any one of claims 1-8.

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a method for ensuring data acquisition accuracy of a client and the client.

Background

In the conventional client data acquisition technology, data transmission is generally performed by directly using an HTTP request, and meanwhile, a retransmission mechanism is not provided, so that data is lost once transmission fails. Meanwhile, due to the fact that no persistence scheme exists, once the client is in a weak network or disconnected network state, data cannot be uploaded immediately, data loss can also be caused, and the phenomenon is more obvious on the mobile device. These defects can result in a substantial degradation of data acquisition integrity and accuracy.

Disclosure of Invention

Therefore, the embodiment of the invention provides a method for ensuring the data acquisition accuracy of a client and the client, so as to solve the problem that the data acquisition accuracy and the integrity of the data are greatly reduced due to the fact that data are easily lost in the existing client data acquisition.

In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:

according to a first aspect of an embodiment of the present invention, a method for ensuring accuracy of client data acquisition is provided, where the method includes:

sending a request to a server to acquire an event to be acquired and a priority corresponding to the event, wherein the event is pre-configured at the server;

serializing the generated event data with different priorities in a memory and storing the event data on a magnetic disk;

and reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the queue to a server, and if the data is failed to be sent due to the interruption of the communication connection, storing the data failed to be sent into the disk for the next resending.

Further, reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, and then sending the queue to a server, specifically, the method further includes:

and sending data according to a preset time interval, limiting the maximum data volume sent each time according to the preset maximum occupied network bandwidth, and if the maximum occupied network bandwidth exceeds the limited network bandwidth quota, reserving the data for next sending.

Further, reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, and then sending the queue to a server, specifically, the method further includes:

and for different events with the same priority, sequentially putting the data into a queue to be sent according to the sequence of the event end timestamps and then sending the data to the server.

Further, reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, and then sending the queue to a server, specifically, the method further includes:

in each data sending process, if data sending failure exists, the data with the sending failure is stored in a sending failure cache file, the failed data is sent preferentially when the data are sent, and then the data are sent according to the priority sequence.

Further, if the data transmission fails due to the occurrence of the communication connection interruption, the data with the transmission failure is stored in the disk for the next retransmission, and the method specifically includes:

in each data sending process, the data under the current communication connection is preferentially sent, and on the premise that the data volume does not exceed the limited network bandwidth quota each time, the failed data under the old connection generated by the interruption of the previous communication connection is sent again.

Further, serializing the generated data with different priorities in a memory and storing the serialized data on a disk, and specifically including:

and performing file division processing on the stored data files with different priorities, setting the size of a single file, and if the size of the single file exceeds the limit, establishing a new file to continuously store data.

Further, the method further comprises:

checking the occupation of the memory and the disk according to a preset time interval, if the occupation of the memory or the disk exceeds the preset maximum occupation memory space or the maximum occupation disk space, preferentially discarding the failure data under the old connection generated by the interruption of the previous communication connection, then discarding the data according to the sequence from low to high in priority, and discarding the data according to the sequence from first to last of different events under the same priority.

Further, sending a request to the server to obtain an event to be collected and a priority corresponding to the event, which are pre-configured at the server, specifically includes:

and under the condition that the network is available, sending a ping request to the server side according to a preset time interval, and if the server side event is found to be configured with an update, sending an update request to update the local data acquisition strategy.

According to a second aspect of the embodiments of the present invention, there is provided a client, including:

the request module is used for sending a request to the server to acquire an event which is required to be acquired and is pre-configured at the server and the priority corresponding to the event;

the data storage module is used for serializing the generated event data with different priorities in the memory and storing the event data on a magnetic disk;

and the data sending module is used for reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the data to the server, and if the data sending fails due to the communication connection interruption, storing the data which is failed to be sent into the disk for the next resending.

According to a third aspect of embodiments of the present invention, a computer storage medium is provided, which contains one or more program instructions for executing, by a client, a method for ensuring client data collection accuracy as described above.

The embodiment of the invention has the following advantages:

the embodiment of the invention provides a method for ensuring the data acquisition accuracy of a client and the client, wherein the method comprises the following steps: sending a request to a server to acquire an event to be acquired and a priority corresponding to the event, wherein the event is pre-configured at the server; serializing the generated event data with different priorities in a memory and storing the event data on a magnetic disk; and reading data from the disk, sequentially putting the data into a queue to be sent according to the priority from high to low, then sending the queue to a server, and if the data is failed to be sent due to the interruption of the communication connection, storing the data failed to be sent into the disk for the next resending. The data real-time persistence and multi-level caching are adopted, so that the data can be ensured not to be lost under the condition that a client program is crashed, abnormal or the network condition is not good, the cached data can be sent out when the program is restarted, and the data acquisition accuracy and integrity are ensured.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are illustrative only and are not exhaustive by those skilled in the art

Fig. 1 is a schematic flowchart of a method for ensuring accuracy of data acquisition of a client according to embodiment 1 of the present invention;

fig. 2 is a schematic diagram of a data storage structure in a method for ensuring accuracy of data acquisition of a client according to embodiment 1 of the present invention;

fig. 3 is a schematic diagram of a data acquisition flow in a method for ensuring accuracy of data acquisition of a client according to embodiment 1 of the present invention;

fig. 4 is a schematic diagram of a data sending flow in a method for ensuring accuracy of data acquisition of a client according to embodiment 1 of the present invention;

fig. 5 is a schematic structural diagram of a client according to embodiment 2 of the present invention.

Detailed Description

The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Embodiment 1 of the present invention provides a method for ensuring accuracy of data acquisition at a client, and as shown in fig. 1, the method includes the following steps:

s110, sending a request to the server to acquire the events needing to be acquired and the priorities corresponding to the events, wherein the events are configured in advance at the server.

The user can configure the events needing to be collected at the server side and configure the priority of the events, and the priority can be divided into 1-3 levels. The event is a series of operations such as user behavior or a change of the APP itself, and is artificially defined, for example, a login event for logging in the APP, a logout event for logging out of the APP, and the like, and data is transmitted in the form of the event. When the client is started, the client requests the server for the events to be collected and the corresponding priorities.

Further, step S110 specifically includes: during the operation of the client, under the condition that a network is available, sending a ping request to the server according to a preset time interval, and if the server event is found to be configured with update, sending an update request to update a local data acquisition strategy. The time interval may be set to 20S.

The client acquisition strategy can be dynamically updated by configuring the acquired events and the event priority in the server without restarting the client, so that the acquisition strategy changes are ensured to be effective immediately, and data is acquired and transmitted immediately.

And S120, serializing the generated event data with different priorities in a memory and storing the event data on a magnetic disk.

Before sending data, the client serializes the data in the memory, stores the data in the disk, and directly reads the data from the disk when sending the data. The data storage structure is shown in fig. 2, the stored data includes both currently connected data and old connected data, the old connected data may include multiple data, the data is stored according to different priorities, different events under the same priority are stored in different file blocks respectively, and the transmission failure cache file is used for storing the transmission failure data. And performing file division processing on the stored data files with different priorities, setting the size of a single file, and if the size of the single file exceeds the limit, establishing a new file to continuously store data.

The storage mode of dividing files according to the data priority is adopted, and high-priority data are transmitted preferentially, so that the high-priority data can still be guaranteed to be sent in time under the condition of poor network condition, and when resources such as a local memory or a disk are insufficient, low-priority data can also be deleted preferentially.

Further, the method further comprises: checking the occupation of the memory and the disk according to a preset time interval, if the occupation of the memory or the disk exceeds the preset maximum occupation memory space or the maximum occupation disk space, preferentially discarding the failure data under the old connection generated by the interruption of the previous communication connection, then discarding the data according to the sequence from low to high in priority, and discarding the data according to the sequence from first to last of different events under the same priority. And the client side defaults to check the occupation of the acquisition system on the memory and the disk every 200 milliseconds. The time interval can be configured to be a minimum of 200 milliseconds. If the memory and disk size exceed the limits, the data of the old connection is discarded first, the data with low priority is discarded second, and the data with the highest priority and the earliest priority is discarded again.

The data acquisition process is as shown in fig. 3, data serialization is performed in a memory after data generation, the serialized data is added into a temporary cache queue, whether data exists in the temporary cache queue is detected, operation is performed once every 200 milliseconds, if data exists, a corresponding storage queue is found according to the priority of an event, whether the size of a tail node of the storage queue exceeds the configuration size is detected, if the size of the tail node exceeds the configuration size, a new storage node is built and set as a tail node, then the data is written into the tail node, whether a total storage file exceeds a configuration threshold is detected, if the size of the total storage file exceeds the configuration threshold, old connected data is preferentially discarded, low-priority data is discarded, and the highest-priority data is discarded again.

By providing a plurality of configurable function parameters, a user can conveniently configure according to the characteristics of own equipment and programs. The configurable program occupies the maximum disk space, the maximum memory space and the maximum network bandwidth. And (3) configuring the maximum occupied disk space and the maximum occupied memory space, when the size of the occupied resources reaches a configuration value, preferentially discarding the data of the old connection, then discarding the low-priority data of the connection, and discarding the data of the highest priority and the earliest of the connection. According to the scheme, data is sent once every 200 milliseconds, and the size of the data sent each time is determined according to the configured maximum network bandwidth occupation, so that the aim of not influencing the transmission speed of other networks in a program is fulfilled.

S130, reading data from the disk, sequentially putting the data into a queue to be sent according to the sequence of the priority levels from high to low, sending the data to a server, and if the data fails to be sent due to communication connection interruption, storing the data which fails to be sent into the disk for next resending.

Further, step S130 specifically includes: and sending data according to a preset time interval, limiting the maximum data volume sent each time according to the preset maximum occupied network bandwidth, and if the maximum occupied network bandwidth exceeds the limited network bandwidth quota, reserving the data for next sending. The client sends data every 200 milliseconds, and the maximum data amount sent each time is the unit event bandwidth limit/sending frequency. If the data volume to be sent exceeds the maximum data volume which can be sent, the data is reserved for sending again. And the excessive occupation of network bandwidth due to excessive single-time data transmission is avoided.

Further, step S130 specifically includes: and for different events with the same priority, sequentially putting the data into a queue to be sent according to the sequence of the event end timestamps and then sending the data to the server. Different events can be configured to have the same priority, for example, an APP login event and an APP exit event are both equally important for APP operation, data corresponding to different events of the same priority, and a later event end timestamp can be sent preferentially.

Further, step S130 specifically includes: in each data sending process, if data sending failure exists, the data with the sending failure is stored in a sending failure cache file, the failed data is sent preferentially when the data are sent, and then the data are sent according to the priority sequence. When preparing data to be sent, the data connected at present is sent preferentially, whether sending failure data exists is checked, if sending failure data exists, the failure data is sent preferentially, and the data is obtained sequentially according to the priority.

Further, step S130 specifically includes: in each data sending process, the data under the current communication connection is preferentially sent, and on the premise that the data volume does not exceed the limited network bandwidth quota each time, the failed data under the old connection generated by the interruption of the previous communication connection is sent again.

As shown in fig. 4, the data sending process is performed once every 200 milliseconds, the current transmission quota is obtained according to the configured network broadband limit, the data of the current connection is preferentially sent, whether sending failure data exists in the current connection is checked, if sending failure data exists, a sending failure cache file is added to a queue to be sent and sent to a server, the data is put into the queue to be sent and sent to the server from high to low according to the priority sequence, and if the size of the queue to be sent does not exceed the network bandwidth quota at the moment, stored old connection data is sequentially put into the queue to be sent and sent to the server from new to old according to the event sequence until the network bandwidth quota is reached.

Through a multi-level persistent cache mode, the data are stored on the local storage device in a serialized mode immediately after being generated, and the data cannot be lost unless a program crashes in a very short time from the generation to the completion of the storage of the data. And a multi-level cache strategy of connection → data priority → data block is used, so that the data of each part are not interfered with each other during reading and writing, and meanwhile, when the data volume is too large and the data needs to be discarded, a ready discarding strategy is provided.

The mechanism of storing a plurality of connection data is adopted, so when the program crashes, the data before the program crashes can be persisted, and when the program is started again, the data connected before can be transmitted. The method can ensure the integrity of data acquisition before the crash, can also be used as a data base for analyzing the crash of the client program, and can provide data support for services such as APM (application program management) and the like.

The method for ensuring the data acquisition accuracy of the client provided by the embodiment of the invention adopts a real-time data persistence and multi-level cache mode, can ensure that data is not lost under the condition that a client program is crashed, abnormal or the network condition is poor, and can send out the previously cached data when the program is restarted to ensure the data acquisition accuracy and integrity; and flexible configuration of the size of the disk space, the size of the memory space and the network bandwidth is realized, and program abnormity caused by excessive resource occupation can be avoided.

Corresponding to the foregoing embodiment 1, embodiment 2 of the present invention provides a client, as shown in fig. 5, where the client includes:

a request module 210, configured to send a request to a server to obtain an event to be acquired and a priority corresponding to the event, where the event is pre-configured at the server;

the data storage module 220 is configured to serialize the generated event data with different priorities in a memory and store the serialized event data on a disk;

and the data sending module 230 is configured to read data from the disk, sequentially place the data into a queue to be sent according to a sequence from high to low priority, and send the queue to the server, and if data sending fails due to communication connection interruption, store the data that has failed in sending into the disk for next resending.

The functions executed by each component in the client terminal proposed in embodiment 2 of the present invention have been described in detail in embodiment 1, and therefore, redundant description is not repeated here.

In accordance with the foregoing embodiments, embodiment 3 of the present invention provides a computer storage medium, where one or more program instructions are contained, and the one or more program instructions are used by a client to execute the above method for ensuring the accuracy of data acquisition of the client.

Although the invention has been described in detail above with reference to a general description and specific examples, it will be apparent to one skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, such modifications and improvements are intended to be within the scope of the invention as claimed.