阅读说明：本技术 一种数据持久化方法、系统、设备以及介质 (Data persistence method, system, device and medium ) 是由 李兴华 于 2020-10-29 设计创作，主要内容包括：本发明公开了一种数据持久化方法,包括以下步骤：创建数据库,并将多个节点上报的数据写入所述数据库中；判断利用创建的所述数据库存储所述数据是否达到预设条件；响应于达到预设条件,创建新的数据库并将所述数据写入所述新的数据库中,并返回判断利用创建的所述数据库存储所述数据是否达到预设条件；响应于触发数据合并,将当前已创建的所有数据库中的数据合并到磁盘。本发明还公开了一种系统、计算机设备以及可读存储介质。本发明提出的方案通过依次创建数据库来存储数据,不仅能够使得每一个数据库存储的数据少,进而提高数据库的读写速率,相比一个数据库也能减少进行数据合并的次数。(The invention discloses a data persistence method, which comprises the following steps: creating a database, and writing data reported by a plurality of nodes into the database; judging whether the data stored by using the created database reaches a preset condition or not; in response to the preset condition, creating a new database, writing the data into the new database, and returning to judge whether the data stored by using the created database reaches the preset condition; and in response to triggering data merging, merging the data in all the databases which are created currently to the disk. The invention also discloses a system, a computer device and a readable storage medium. According to the scheme provided by the invention, the databases are sequentially created to store data, so that the data stored in each database is less, the reading and writing speed of the databases is further improved, and the times of data combination can be reduced compared with one database.)

1. A method of data persistence, comprising the steps of:

creating a database, and writing data reported by a plurality of nodes into the database;

judging whether the data stored by using the created database reaches a preset condition or not;

in response to the preset condition, creating a new database, writing the data into the new database, and returning to judge whether the data stored by using the created database reaches the preset condition;

and in response to triggering data merging, merging the data in all the databases which are created currently to the disk.

2. The method of claim 1, further comprising:

and in response to the completion of data merging, deleting all the currently created databases and returning to the step of creating the database.

3. The method of claim 1, wherein in response to reaching a preset condition, creating a new database and writing the data into the new database, further comprising:

and in response to the time length for continuously storing the data in the database for storing the data at present reaching a time length threshold value, creating a new database and writing the data into the new database.

4. The method of claim 3, further comprising:

in response to receiving a data reading request, determining a generation time of data to be read;

determining a database corresponding to the data to be read according to the generation time;

and acquiring the data to be read from the corresponding database and feeding back the data.

5. The method of claim 1, wherein merging data in all databases that have been currently created to disk in response to triggering a data merge, further comprises:

and triggering data merging in response to the number of the currently created databases reaching a number threshold and the duration of the last created database for continuously storing data reaching a duration threshold.

6. The method of claim 5, wherein merging data in all databases that have been currently created to disk, further comprises:

and screening, filtering and compressing the data in all the databases which are created at present, and then storing the data in the disk.

7. The method of claim 1, wherein creating a database further comprises:

a database is created in the memory.

8. A data persistence system, comprising:

the system comprises a creating module, a sending module and a receiving module, wherein the creating module is configured to create a database and write data reported by a plurality of nodes into the database;

the judging module is configured to judge whether the data stored by the created database reaches a preset condition;

the circulation module is configured to respond to the fact that a preset condition is met, create a new database, write the data into the new database, and return to judge whether the data stored by the created database meets the preset condition;

a merging module configured to merge data in all databases that have been currently created to a disk in response to triggering a data merge.

9. A computer device, comprising:

at least one processor; and

memory storing a computer program operable on the processor, wherein the processor executes the program to perform the steps of the method according to any of claims 1-7.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1 to 7.

Technical Field

The present invention relates to the field of data storage, and in particular, to a data persistence method, system, device, and storage medium.

Background

As shown in fig. 1, the general performance data collection and storage architecture has two roles: the node comprises a Master node and an Agent node, wherein the Agent node collects performance data of the node and sends the performance data to the Master node, and the Master node receives the data sent by the Agent node and stores the data into a memory database. With the increase of Agent nodes and the increase of data received by a Master node, all data are stored in a memory database, and a large amount of memory resources are occupied. Meanwhile, the larger the data volume of the Master node needing to be persisted in unit time is, the more the data volume is, the processing capacity of the database can be exceeded, and data persistence blocking or data loss can be caused.

How to realize persistence of a large amount of data becomes a problem which needs to be solved urgently.

Disclosure of Invention

In view of the above, in order to overcome at least one aspect of the above problems, an embodiment of the present invention provides a data persistence method, including the following steps:

creating a database, and writing data reported by a plurality of nodes into the database;

judging whether the data stored by using the created database reaches a preset condition or not;

in response to the preset condition, creating a new database, writing the data into the new database, and returning to judge whether the data stored by using the created database reaches the preset condition;

and in response to triggering data merging, merging the data in all the databases which are created currently to the disk.

In some embodiments, further comprising:

and in response to the completion of data merging, deleting all the currently created databases and returning to the step of creating the database.

In some embodiments, in response to reaching a preset condition, creating a new database and writing the data into the new database, further comprising:

and in response to the time length for continuously storing the data in the database for storing the data at present reaching a time length threshold value, creating a new database and writing the data into the new database.

In some embodiments, further comprising:

in response to receiving a data reading request, determining a generation time of data to be read;

determining a database corresponding to the data to be read according to the generation time;

and acquiring the data to be read from the corresponding database and feeding back the data.

In some embodiments, in response to triggering the data merge, merging the data in all databases that have been currently created to disk, further comprising:

and triggering data merging in response to the number of the currently created databases reaching a number threshold and the duration of the last created database for continuously storing data reaching a duration threshold.

In some embodiments, merging the data in all databases that have been currently created to disk, further comprises:

and screening, filtering and compressing the data in all the databases which are created at present, and then storing the data in the disk.

In some embodiments, creating a database further comprises:

a database is created in the memory.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data persistence system, including:

the system comprises a creating module, a sending module and a receiving module, wherein the creating module is configured to create a database and write data reported by a plurality of nodes into the database;

the judging module is configured to judge whether the data stored by the created database reaches a preset condition;

the circulation module is configured to respond to the fact that a preset condition is met, create a new database, write the data into the new database, and return to judge whether the data stored by the created database meets the preset condition;

a merging module configured to merge data in all databases that have been currently created to a disk in response to triggering a data merge.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer apparatus, including:

at least one processor; and

a memory storing a computer program operable on the processor, wherein the processor executes the program to perform any of the steps of the data persistence methods described above.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of any of the data persistence methods described above.

The invention has one of the following beneficial technical effects: according to the scheme provided by the invention, the databases are sequentially created to store data, so that the data stored in each database is less, the reading and writing speed of the databases is further improved, and the times of data combination can be reduced compared with one database.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.

FIG. 1 is a block diagram of a prior art data persistence process;

FIG. 2 is a flow chart of a data persistence method provided by an embodiment of the present invention;

FIG. 3 is a block flow diagram of data persistence provided by an embodiment of the invention;

FIG. 4 is a schematic structural diagram of a data persistence system provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;

fig. 6 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.

According to an aspect of the present invention, an embodiment of the present invention proposes a data persistence method, as shown in fig. 1, which may include the steps of:

s1, creating a database, and writing the data reported by a plurality of nodes into the database;

s2, judging whether the data stored by the created database reaches a preset condition;

s3, in response to the preset condition, creating a new database and writing the data into the new database, and returning to judge whether the data stored by the created database reaches the preset condition;

and S4, in response to triggering data merging, merging the data in all the databases which are created currently to the disk.

According to the scheme provided by the invention, the databases are sequentially created to store data, so that the data stored in each database is less, the reading and writing speed of the databases is further improved, and the times of data combination can be reduced compared with one database.

In some embodiments, the step S1 creates a database, and writes the data reported by the plurality of nodes into the database, further including:

a database is created in the memory.

Specifically, the general performance data collection and storage architecture has two roles: the system comprises a Master node and an Agent node, wherein the Agent node collects performance data of the node and sends the performance data to the Master node, and the Master node receives the data sent by the Agent node and stores the data into a database created in a memory.

In some embodiments, in response to reaching a preset condition, creating a new database and writing the data into the new database, further comprising:

and in response to the time length for continuously storing the data in the database for storing the data at present reaching a time length threshold value, creating a new database and writing the data into the new database.

Specifically, after the database is created, the database storage Agent is used to report the data to the Master, and if the duration of the data continuously stored in the database for storing the data reaches a duration threshold, for example, 1 hour, a new database is created and the received data is stored by using the newly created database, and the previous database stops storing the data. Similarly, if the duration of the newly created database for continuously storing the data reaches the duration threshold, the new database is created again and the received data is stored by using the newly created database again, and the previous database stops storing the data. For example, as shown in fig. 3, when the Master receives data, it stores the data in the in-memory database DB1, and thereafter, until the next hour, the received data is stored in the DB 1. When a new hour begins, the data received by the Master is stored in the in-memory database DB2, and the database DB1 no longer stores data. When a new hour begins, the data received by the Master is stored in the in-memory database DB3, and the database DB2 no longer stores data.

In some embodiments, in response to triggering the data merge, merging the data in all databases that have been currently created to disk, further comprising:

and triggering data merging in response to the number of the currently created databases reaching a number threshold and the duration of the last created database for continuously storing data reaching a duration threshold.

In some embodiments, further comprising:

and in response to the completion of data merging, deleting all the currently created databases and returning to the step of creating the database.

Specifically, if the number of the currently created databases reaches a set number threshold, for example, 3 databases, and the duration for the last created database to continuously store data reaches a duration threshold, that is, the total duration reaches a number threshold × a duration threshold, the data merging is triggered. For example, the mass may merge DB1, DB2, and DB3 into one database DB every 3 hours, and save the database DB on a disk, and delete DB1, DB2, and DB 3.

It should be noted that the created database may be cleared instead of deleted, but since clearing the tables in the created database is more cumbersome than recreating the database, the created database is preferentially deleted and the database is recreated to store the received data.

In some embodiments, further comprising:

in response to receiving a data reading request, determining a generation time of data to be read;

determining a database corresponding to the data to be read according to the generation time;

and acquiring the data to be read from the corresponding database and feeding back the data.

Specifically, when a request for reading data is received, if the data is stored in one of the multiple databases that have been created at the present time, since each period of time corresponds to one database, it is possible to determine in which database the data is located according to the time when the data is generated, and then read the data in the database for feedback.

In some embodiments, merging the data in all databases that have been currently created to disk, further comprises:

and screening, filtering and compressing the data in all the databases which are created at present, and then storing the data in the disk.

Specifically, when data merging is performed, data in each database can be subjected to screening, compression, filtering and other processing, so that the size of finally obtained merged data can be smaller than the size of data addition in each database, and the storage space is further saved.

According to the scheme provided by the invention, only the data with the preset duration is stored in the memory, so that the memory occupation is greatly reduced. Because each database is smaller, the reading and writing speed is higher, and more data are stored in unit time, the problem of data persistence blocking or loss caused by excessive data is solved.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a data persistence system 400, as shown in fig. 4, including:

a creating module 401, wherein the creating module 401 is configured to create a database and write data reported by a plurality of nodes into the database;

a judging module 402, where the judging module 402 is configured to judge whether the data stored by using the created database reaches a preset condition;

a loop module 403, where the loop module 403 is configured to create a new database and write the data into the new database in response to reaching a preset condition, and return to determine whether the data stored in the created database reaches the preset condition;

a merge module 404, the merge module 404 configured to merge data in all databases that have been currently created to disk in response to triggering a data merge.

In some embodiments, further comprising a deletion module configured to:

and in response to the completion of data merging, deleting all the currently created databases and returning to the step of creating the database.

In some embodiments, the loop module 403 is configured to:

and in response to the time length for continuously storing the data in the database for storing the data at present reaching a time length threshold value, creating a new database and writing the data into the new database.

In some embodiments, further comprising a data reading module configured to:

in response to receiving a data reading request, determining a generation time of data to be read;

determining a database corresponding to the data to be read according to the generation time;

and acquiring the data to be read from the corresponding database and feeding back the data.

In some embodiments, the merge module 404 is further configured to:

and triggering data merging in response to the number of the currently created databases reaching a number threshold and the duration of the last created database for continuously storing data reaching a duration threshold.

In some embodiments, the merge module 404 is further configured to: and screening, filtering and compressing the data in all the databases which are created at present, and then storing the data in the disk.

According to the scheme provided by the invention, only the data with the preset duration is stored in the memory, so that the memory occupation is greatly reduced. Because each database is smaller, the reading and writing speed is higher, and more data are stored in unit time, the problem of data persistence blocking or loss caused by excessive data is solved.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 5, an embodiment of the present invention further provides a computer apparatus 501, comprising:

at least one processor 520; and

a memory 510, the memory 510 storing a computer program 511 executable on the processor, the processor 520 executing the program to perform the steps of any of the data persistence methods as described above.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 6, an embodiment of the present invention further provides a computer-readable storage medium 601, where the computer-readable storage medium 601 stores computer program instructions 610, and the computer program instructions 610, when executed by a processor, perform the steps of any of the above data persistence methods.

Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program to instruct related hardware to implement the methods.

Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.