Data backup method, data recovery method and device

文档序号：95812 发布日期：2021-10-12 浏览：5次中文

阅读说明：本技术 数据备份方法、数据恢复方法及装置 (Data backup method, data recovery method and device ) 是由赵成阮伟毅于 2020-03-20 设计创作，主要内容包括：本申请公开了一种数据备份方法、数据恢复方法及装置,属于数据库领域。所述方法包括：数据备份方法,其特征在于,所述方法包括：获取多个分片数据中每个分片数据的第一元数据,所述多个分片数据是对日志数据进行分片处理得到的数据；获取所述日志数据的第二元数据；存储备份数据,所述备份数据包括多个所述第一元数据和所述第二元数据,所述备份数据用于恢复第一时刻至第二时刻之间的增量分片数据,所述第一时刻为开始获取所述第一元数据的时刻,所述第二时刻为结束获取所述第二元数据的时刻。本申请解决了无法确定备份过程中产生的增量分片数据的问题,本申请用于数据库中的数据备份。(The application discloses a data backup method, a data recovery method and a data recovery device, and belongs to the field of databases. The method comprises the following steps: a method for data backup, the method comprising: acquiring first metadata of each piece of fragment data in a plurality of pieces of fragment data, wherein the plurality of pieces of fragment data are data obtained by carrying out fragment processing on log data; acquiring second metadata of the log data; and storing backup data, wherein the backup data comprises a plurality of first metadata and second metadata, and the backup data is used for restoring incremental fragment data between a first time and a second time, the first time is the time when the first metadata starts to be acquired, and the second time is the time when the second metadata finishes to be acquired. The method and the device solve the problem that incremental fragment data generated in the backup process cannot be determined, and are used for data backup in the database.)

1. A method for data backup, the method comprising:

acquiring second metadata of the log data;

and storing backup data, wherein the backup data comprises a plurality of first metadata and second metadata, and the backup data is used for restoring incremental fragment data between a first time and a second time, the first time is the time when the first metadata starts to be acquired, and the second time is the time when the second metadata finishes to be acquired.

2. The method of claim 1, wherein the backup data further comprises log data corresponding to the second metadata, first indication information and second indication information, wherein the first indication information is used for indicating the first time, and the second indication information is used for indicating the second time.

3. The method of claim 1, wherein the backup data further comprises incremental log data generated between the first time and the second time in the log data; the method further comprises the following steps:

determining incremental log data generated between the first time and the second time in the log data based on first indication information and second indication information, wherein the first indication information is used for indicating the first time, and the second indication information is used for indicating the second time.

4. The method according to claim 2 or 3, wherein the first indication information comprises: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment;

the second indication information includes: and the continuous disk-dropping LSN at the second moment is the largest LSN in the LSNs corresponding to the log data of which the disk-dropping has been continuously performed at the corresponding moment.

5. The method according to any one of claims 1 to 4, wherein the backup data further includes a third element of fragmentation relation data, and the fragmentation relation data is used for recording the correspondence between the sub-data and the fragments in the log data.

6. The method of claim 5, wherein the backing up data further comprises: the plurality of shard data respectively corresponding to the plurality of first metadata, and the shard relation data corresponding to the third metadata.

7. The method of any of claims 1 to 6, wherein the metadata of any of the log data and the sharded data comprises: data writing information of each file in at least one file, wherein the at least one file is used for storing any data;

the data writing information of any file is used for recording the position of the any file in the at least one file and the end position of the written data in the any file.

8. The method according to claim 7, wherein the data writing information of any file comprises: the maximum log serial number LSN carried in the written data in any file, and the file position identification of any file.

9. A method for data recovery, the method comprising:

restoring incremental fragmented data between the first time and the second time based on the backup data;

the first time is the time when the first metadata starts to be acquired before the backup data is stored, and the second time is the time when the second metadata finishes to be acquired before the backup data is stored.

10. The method of claim 9, wherein restoring incremental sharded data between a first time and a second time based on the backup data comprises:

recovering incremental log data generated between the first time and the second time in the log data;

recovering the plurality of sliced data based on the plurality of first metadata;

determining at least one piece of incremental fragmentation data between the first time and the second time based on the recovered incremental log data and fragmentation relation data, wherein the fragmentation relation data is used for recording the corresponding relation between subdata of the log data and fragments;

updating the plurality of sharded data based on the sharded relational data and the at least one incremental sharded data, the updated sharded data comprising the at least one incremental sharded data.

11. The method of claim 10, wherein the backup data further comprises incremental log data generated between the first time and the second time in the log data;

or, the backup data further includes log data corresponding to the second metadata, first indication information, and second indication information, where the first indication information is used to indicate the first time, the second indication information is used to indicate the second time, and the restoring incremental log data generated between the first time and the second time in the log data includes: and based on the first indication information and the second indication information, incremental log data is generated between the first time and the second time determined in the log data corresponding to the second metadata.

12. The method of claim 11, wherein the first indication information comprises: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment;

13. The method of claim 10, wherein the backup data further comprises a third metadata of sharding relationship data, the method further comprising:

and recovering the fragmentation relation data based on the third metadata.

14. The method according to any one of claims 9 to 13, wherein the updating the plurality of sharded data based on the sharded relational data and the at least one incremental sharded data comprises:

for each increment fragment data, when the corresponding target fragment stores data which is the same as the increment fragment data, the increment fragment data is prohibited from being written in, or the increment fragment data is adopted to cover the data which is the same as the increment fragment data in the corresponding target fragment, and the target fragment is the fragment corresponding to the increment fragment data determined based on the fragment relation data.

15. A data backup apparatus, characterized in that the apparatus comprises:

the first acquisition module is used for acquiring first metadata of each piece of fragment data in a plurality of pieces of fragment data, wherein the plurality of pieces of fragment data are data obtained by carrying out fragment processing on log data;

the second acquisition module is used for acquiring second metadata of the log data;

the storage module is configured to store backup data, where the backup data includes a plurality of the first metadata and the second metadata, and the backup data is used to restore incremental fragment data between a first time and a second time, where the first time is a time when the first metadata starts to be acquired, and the second time is a time when the second metadata ends to be acquired.

16. The apparatus of claim 15, wherein the backup data further comprises log data corresponding to the second metadata, first indication information and second indication information, the first indication information is used for indicating the first time, and the second indication information is used for indicating the second time.

17. The apparatus of claim 15, wherein the backup data further comprises incremental log data generated between the first time and the second time in the log data; the device further comprises:

the determining module is configured to determine incremental log data generated between the first time and the second time in the log data based on first indication information and second indication information, where the first indication information is used for indicating the first time, and the second indication information is used for indicating the second time.

18. The apparatus according to claim 16 or 17, wherein the first indication information comprises: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment;

19. The apparatus according to any one of claims 15 to 18, wherein the backup data further includes a third element of sharding relationship data, and the sharding relationship data is used for recording a correspondence between sub data in the log data and the shards.

20. The apparatus of claim 19, wherein the backup data further comprises: the plurality of shard data respectively corresponding to the plurality of first metadata, and the shard relation data corresponding to the third metadata.

21. The apparatus according to any of claims 15 to 20, wherein the metadata of any of the log data and the sharded data comprises: data writing information of each file in at least one file, wherein the at least one file is used for storing any data;

the data writing information of any file is used for recording the position of the any file in the at least one file and the end position of the written data in the any file.

22. The apparatus of claim 21, wherein the data writing information of any file comprises: the maximum log serial number LSN carried in the written data in any file, and the file position identification of any file.

23. An apparatus for data recovery, the apparatus comprising:

the acquisition module is used for acquiring stored backup data, wherein the backup data comprises a plurality of first metadata and second metadata, each first metadata is metadata of one fragmented data, and a plurality of fragmented data corresponding to the plurality of first metadata are fragmented data of the log data;

the first recovery module is used for recovering the incremental fragment data between the first time and the second time based on the backup data;

24. The apparatus of claim 23, wherein the recovery module is configured to:

recovering incremental log data generated between the first time and the second time in the log data;

recovering the plurality of sliced data based on the plurality of first metadata;

25. The apparatus of claim 24, wherein the backup data further comprises incremental log data generated between the first time and the second time in the log data;

26. The apparatus of claim 25, wherein the first indication information comprises: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment;

27. The apparatus of claim 24, wherein the backup data further comprises a third metadata of sharding relationship data, the apparatus further comprising:

and the second recovery module is used for recovering the fragment relation data based on the third metadata.

28. The apparatus according to any one of claims 23 to 27, wherein the first recovery module is configured to:

29. A computer device, comprising:

a processor and a memory;

the memory to store computer instructions;

the processor is configured to execute the computer instructions stored in the memory to cause the computer device to perform the data backup method according to any one of claims 1 to 8 or perform the data recovery method according to any one of claims 9 to 14.

30. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises computer instructions that instruct a computer device to perform the data backup method of any one of claims 1 to 8 or to perform the data recovery method of any one of claims 9 to 14.

Technical Field

The present application relates to the field of databases, and in particular, to a data backup method, a data recovery method, and a data recovery device.

Background

A relational database refers to a database that organizes data using a relational model. In a relational database, a transaction (transaction) generates a transaction log (redo log) during execution, which is a set of files that can be directly stored on a disk.

In order to improve disaster tolerance of a Database (DB), a transaction log needs to be backed up (e.g., a transaction log of important data needs to be backed up). However, when the relational Database is a Distributed Database (DDB), the log data of the transaction log may be divided into a plurality of pieces (slice) of log data (referred to as slices for short), and the slices may be stored in the data storage nodes respectively. If a plurality of fragmented data are directly and respectively backed up, incremental fragmented data may be generated in the backup process, the incremental fragmented data cannot be determined during data recovery, and the service consistency of the backed-up data cannot be ensured; if locking is performed in the process of backing up a plurality of fragmented data, the generation of incremental fragmented data is avoided, and the service interruption time is easily overlong. There is therefore a need for a method that enables efficient backup of data in a distributed database.

Disclosure of Invention

The embodiment of the application provides a data backup method, a data recovery method and a data recovery device. The technical scheme is as follows:

in a first aspect, the present application provides a data backup method, which may be performed by a management node, the method including:

acquiring first metadata of each piece of fragment data in a plurality of pieces of fragment data, wherein the plurality of pieces of fragment data are data obtained by carrying out fragment processing on log data; acquiring second metadata of the log data; and storing backup data, wherein the backup data comprises a plurality of first metadata and second metadata, and the backup data is used for recovering incremental fragment data between a first time and a second time, the first time is the time when the first metadata starts to be acquired, and the second time is the time when the second metadata finishes to be acquired.

According to the embodiment of the application, the incremental fragmented data between the first time and the second time can be effectively recovered based on the acquired backup data by carrying the first metadata of the multiple fragmented data and the second metadata of the log data in the backup data. On the premise of not adding a data lock, incremental fragment data is determined, so that the service consistency of backup data is ensured, effective data backup in a distributed database is realized, and database service is not influenced.

The backup data in the embodiment of the application may further include log data, and the log data included in the backup data includes incremental log data of a target backup time period to restore incremental fragmented data. The examples of the present application take the following cases as examples:

In a second case, the backup data further includes log data corresponding to the second metadata, first indication information and second indication information, where the first indication information is used to indicate the first time, and the second indication information is used to indicate the second time.

In a third case, the backup data further includes incremental log data generated between the first time and the second time in the log data; optionally, the data backup method further includes: and determining incremental log data generated between the first time and the second time in the log data based on first indication information and second indication information, wherein the first indication information is used for indicating the first time, and the second indication information is used for indicating the second time.

By way of example, the first indication information includes: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment; the second indication information includes: and the continuous disk-dropping LSN at the second moment is the largest LSN in the LSNs corresponding to the log data of which the disks are continuously dropped at the corresponding moment.

On the basis of the foregoing three cases, the backup data may further include fragmented data acquired at the third time. Alternatively, the backup data may further include a plurality of sharded data corresponding to the plurality of first metadata, respectively.

Since incremental data may be generated from the fragmented data during the storage of the backup data, if all the fragmented data obtained at the third time is stored, the end position (corresponding position at the third time) of the stored data is inconsistent with the end position (corresponding position at the time between the first time and the second time) indicated by the corresponding metadata, and the incremental data (i.e., the incremental data between the second time and the third time) is additionally stored, which increases the load of the storage medium. The fragment data corresponding to the first metadata are stored, so that the end position of the stored data is consistent with the end position indicated by the corresponding metadata, additional storage of incremental data is avoided, and the load of a storage medium is reduced.

Because the management node determines the fragmentation relation data corresponding to the log data based on the fragmentation relation data, the fragmentation relation data also needs to be acquired when recovering the fragmentation relation data. In an alternative, the sharded relational data may be stored directly for use in data recovery; in another alternative, the fragmentation relation rule may be stored, so that when data is restored, fragmentation relation data is obtained based on the fragmentation relation rule; in yet another alternative, the fragmentation relation data has metadata, that is, the data corresponding to the fragmentation relation includes: and the management node may store the third metadata so as to obtain the fragmentation relationship data based on the metadata when the data is restored. Wherein, the third metadata is used for describing the fragmentation relation data. The fragmentation relation data is obtained through the third metadata, and the fragmentation relation data can be rapidly obtained.

Optionally, the backup data further includes: the fragment data corresponding to the first metadata and the fragment relation data corresponding to the third metadata.

Optionally, the metadata of any one of the log data and the shard data includes: and writing the data of each file in at least one file into information, wherein the at least one file is used for storing any data. That is, each file includes two parts: one part is used for storing the arbitrary data and the other part is used for storing data writing information. Wherein, a fixed-size designated location (also referred to as a designated space) may be set in each file for storing data writing information. For example, the specified location may be a header or an end of the file. The data writing information is used for describing the writing condition of any data in the file. For example, the data writing information of any file is used to record the position of the any file in the at least one file, and the end position of the written data (i.e., the any data written) in the any file. The position of each file can be effectively identified through the metadata, so that the files are sequentially and continuously acquired when the data are recovered, the continuity of the files is ensured, and any effective data can be recovered.

Since the LSN characterizes the total number of bytes written by a transaction to the transaction log. It is clear that LSN is an indication of the typical amount of data for a transaction log. And the log data of one file is written continuously. As the number of transaction record writes increases, the LSN also increases. Thus, the largest LSN carried by the written data in the file (i.e., the LSN of the most recently written transaction record) can reflect the end location of the written data. Optionally, the data writing information of any file includes: and the maximum log sequence number LSN carried in the written data in any file is used for identifying the end position of the written data. Optionally, the data writing information of any file corresponding to the log data further includes: the file location identification of any of the files. The file location identifier is used for identifying the location of the any file in at least one file to which the any file belongs.

In a second aspect, the present application provides a data recovery method, which may be performed by a management node, the method comprising:

the method comprises the steps of obtaining stored backup data, wherein the backup data comprise a plurality of first metadata and second metadata, each first metadata is metadata of one piece of fragmented data, and a plurality of pieces of fragmented data corresponding to the plurality of first metadata are fragmented data of the log data; based on the backup data, recovering incremental fragment data between the first time and the second time; the first time is the time when the first metadata starts to be acquired before the backup data is stored, and the second time is the time when the second metadata finishes to be acquired before the backup data is stored.

According to the embodiment of the application, the first metadata of each piece of data in the plurality of piece of data and the second metadata of the piece of relation data are respectively obtained, and the incremental piece data between the first time and the second time can be effectively recovered based on the obtained metadata. On the premise of not adding a data lock, incremental fragment data is determined, so that the service consistency of backup data is ensured, effective data backup in a distributed database is realized, and database service is not influenced.

Optionally, the process of restoring the incremental fragmented data between the first time and the second time based on the backup data includes: recovering incremental log data generated between the first time and the second time in the log data; recovering the plurality of sliced data based on the plurality of first metadata; determining at least one piece of incremental fragmentation data between the first time and the second time based on the recovered incremental log data and fragmentation relation data, wherein the fragmentation relation data is used for recording the corresponding relation between subdata of the log data and fragments; updating the plurality of sharded data based on the sharding relationship data and the at least one incremental sharded data, the updated plurality of sharded data including the at least one incremental sharded data.

The backup data may further include log data including incremental log data of the target backup period. The examples of the present application take the following cases as examples:

in the first case, the backup data further includes log data acquired up to the third time (i.e., log data acquired at the upload time), the first indication information, and the second indication information. The third time is a specified time after the second time, for example, the third time is a time that is a specified time period from the second time, and for example, the third time is a time when all available log data are stored (for example, backup data need to be stored in the cloud storage space, and the third time is a time when all log data locally obtained at the data storage node are uploaded). The first indication information is used for indicating a first time, and the second indication information is used for indicating a second time.

In a first alternative example, it is assumed that the first indication information is global sliced-destaged LSNs, and the second indication information is consecutive destaged LSNs. The management node may determine, based on the obtained data correspondence, a target LSN corresponding to the global fragmented destaging LSN in the log data, and then determine incremental log data based on the target LSN and the continuous destaging LSNs, where the LSN corresponding to the incremental log data is an LSN between the target LSN and the continuous destaging LSNs.

In a second alternative example, the first indication information and the second indication information may be time stamps, such as atomic clock time stamps. The management node may determine a target time period (i.e., a difference between two timestamps) based on the timestamp in the first indication information (characterizing the first time) and the timestamp in the second indication information (characterizing the second time), and then determine incremental log data between the first time and the second time in the log data in the backup data based on the timestamp in the log data in the backup data, the timestamp in the incremental log data belonging to the target time period.

In a third alternative example, the first indication information and the second indication information may be globally unique identifiers. For example, the globally unique identifier may characterize temporal or spatial precedence. The globally unique identification is assigned in an incremental manner. For example, it may be an increasing number, or sequentially increasing letters, etc. The management node may determine a target variable range (i.e., a range with two globally unique identifiers as end points) based on the globally unique identifier in the first indication information (characterizing the first time) and the globally unique identifier in the second indication information (characterizing the second time), and then determine incremental log data between the first time and the second time in the log data in the backup data based on the globally unique identifier in the log data in the backup data, the globally unique identifier in the incremental log data belonging to the target variable range.

In the second case, the backup data further includes log data corresponding to the second metadata, the first indication information, and the second indication information. When data recovery is performed, incremental log data generated between the first time and the second time determined in the log data corresponding to the second metadata can be generated based on the first indication information and the second indication information.

In a third case, the backup data further includes incremental log data generated between the first time and the second time in the log data; when data recovery is performed, the analysis node can directly acquire incremental log data.

Optionally, the first indication information includes: the global fragmentation destaging LSN at the first moment is the largest LSN in LSNs corresponding to the fragmentation data which are subjected to global continuous destaging at the corresponding moment; the second indication information includes: and the continuous disk-dropping LSN at the second moment is the largest LSN in the LSNs corresponding to the log data of which the disks are continuously dropped at the corresponding moment.

The management node may obtain the fragmentation relation data in various ways. In an optional manner, the backup data includes fragmentation relation data, and the management node may directly obtain the fragmentation relation data in the backup data; in another optional mode, the management node may pre-store a fragmentation relation rule, and obtain fragmentation relation data based on the fragmentation relation rule; in yet another alternative, the sharding relationship data has metadata, the metadata of the sharding relationship data is referred to as third metadata, the backup data includes the third metadata, and the management node restores the sharding relationship data based on the third metadata.

Optionally, the process of updating the plurality of shard data based on the shard relationship data and the at least one incremental shard data includes: for each increment fragment data, when the corresponding target fragment stores the data which is the same as the increment fragment data, the increment fragment data is prohibited from being written in, or the increment fragment data is adopted to cover the data which is the same as the increment fragment data in the corresponding target fragment, and the target fragment is the fragment which is determined based on the fragment relation data and is corresponding to the increment fragment data.

When the data stored in the corresponding target fragment is different from the incremental fragment data, the fact that the data stored in the target fragment and the incremental fragment data have repeated data is indicated.

Optionally, the metadata of any one of the log data, the sharding relationship data, and the sharding data includes: data writing information of each file in at least one file, wherein the at least one file is used for storing any data; the data writing information of any file is used for recording the position of the any file in the at least one file and the end position of the written data in the any file.

Optionally, for the log data or the fragment data, the data writing information of any file includes: the maximum log sequence number LSN carried in the written data in any file, and the file position identification of any file.

In a third aspect, the present application provides a data backup apparatus, which may include at least one module, where the at least one module may be configured to implement the data backup method provided in the first aspect or various possible implementations of the first aspect.

In a fourth aspect, the present application provides a data recovery apparatus, which may include at least one module, where the at least one module may be configured to implement the data recovery method provided in the first aspect or various possible implementations of the first aspect.

In a fifth aspect, the present application provides a computer device comprising a processor and a memory. The memory stores computer instructions; the processor executes the computer instructions stored by the memory to cause the computer device to perform the method provided by the first aspect or the various possible implementations of the first aspect, so that the computer device deploys the data backup apparatus provided by the third aspect or the various possible implementations of the third aspect.

In a sixth aspect, the present application provides a computer device comprising a processor and a memory. The memory stores computer instructions; the processor executes the computer instructions stored by the memory to cause the computer device to perform the method provided by the second aspect or the various possible implementations of the second aspect, so that the computer device deploys the data recovery apparatus provided by the fourth aspect or the various possible implementations of the fourth aspect.

In a seventh aspect, the present application provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and the computer instructions instruct the computer device to execute the method provided by the first aspect or the various possible implementations of the first aspect, or instruct the computer device to deploy the data backup apparatus provided by the third aspect or the various possible implementations of the third aspect.

In an eighth aspect, the present application provides a computer-readable storage medium having stored therein computer instructions that instruct a computer device to execute the method provided by the second aspect or the various possible implementations of the second aspect, or instruct the computer device to deploy the data recovery apparatus provided by the fourth aspect or the various possible implementations of the fourth aspect.

In a ninth aspect, the present application provides a computer program product comprising computer instructions stored in a computer readable storage medium. A processor of the computer device may read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the method provided by the first aspect or the various possible implementations of the first aspect, so that the computer device deploys the data backup apparatus provided by the third aspect or the various possible implementations of the third aspect.

In a tenth aspect, the present application provides a computer program product comprising computer instructions stored in a computer readable storage medium. A processor of the computer device may read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the method provided by the second aspect or the various possible implementations of the second aspect, so that the computer device deploys the data recovery apparatus provided by the fourth aspect or the various possible implementations of the fourth aspect.

In an eleventh aspect, a distributed database system is provided, comprising: a management node and a data node, the management node comprising the second aspect or various possible implementations of the second aspect of the data backup apparatus or the computer device of the third aspect. Alternatively, the management node comprises the data recovery apparatus of the fourth aspect or various possible implementations of the fourth aspect or the computer device of the sixth aspect.

In a twelfth aspect, a chip is provided, which may comprise programmable logic circuits and/or program instructions, for implementing the data backup method according to any of the first aspects when the chip is in operation. Or, when the chip is operating, for implementing a data recovery method as claimed in any of the second aspects.

Drawings

Fig. 1 is a schematic diagram of a relationship between log data and sliced log data and slices provided in an embodiment of the present application;

fig. 2 is a schematic diagram of an application environment of a distributed database system related to a data backup method provided in an embodiment of the present application;

fig. 3 is a schematic diagram of another application environment of a distributed database system related to a data backup method provided in an embodiment of the present application;

fig. 4 is a schematic time line diagram of a data backup method according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a data backup method according to an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating a structure of a transaction record provided in an embodiment of the present application;

fig. 7 is a schematic diagram of a storage state of fragmented log data in fragmented data at a first time according to an embodiment of the present application;

fig. 8 is a schematic diagram illustrating a storage state of log data at a second time according to an embodiment of the present application;

fig. 9 is a schematic diagram illustrating an obtaining manner of log data corresponding to a target backup time period according to an embodiment of the present application;

fig. 10 is a schematic diagram illustrating another obtaining manner of log data corresponding to a target backup time period according to an embodiment of the present application;

fig. 11 is a schematic diagram of an obtaining manner of an incremental log according to an embodiment of the present application;

fig. 12 is a schematic diagram of another application environment of a distributed database system related to the data backup method provided in the embodiment of the present application;

fig. 13 is a schematic time line diagram of a data backup method according to an embodiment of the present application;

fig. 14 is a schematic diagram of another application environment of a distributed database system related to the data backup method provided in the embodiment of the present application;

fig. 15 is a schematic flowchart of a data recovery method according to an embodiment of the present application;

FIG. 16 is a schematic diagram illustrating a principle of obtaining at least one incremental fragment data based on incremental log data according to an embodiment of the present application;

FIG. 17 is a schematic flowchart of another data backup method provided in an embodiment of the present application;

fig. 18 is a schematic flowchart of a data recovery method according to an embodiment of the present application;

fig. 19 is a block diagram of a data backup apparatus according to an embodiment of the present application;

fig. 20 is a block diagram of another data backup apparatus provided in an embodiment of the present application;

fig. 21 is a block diagram of a data recovery apparatus according to an embodiment of the present application;

FIG. 22 is a block diagram of another data recovery apparatus provided in an embodiment of the present application;

fig. 23 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

For the convenience of the reader, the terms referred to in the examples of the present application are explained below:

relational database: refers to a database that employs a relational model to organize data, which stores data in rows and columns. Each relational model may be referred to as a relational table. Relational databases can be classified into distributed relational databases and non-distributed relational databases, depending on the storage principle.

Transaction: the method is a unit for maintaining the consistency and the integrity of database data, and has the characteristic of 'full success or failure'. Before a transaction commits, the database is not responsible for the consistency and integrity of its data. However, once the transaction is submitted, the database can ensure the consistency and integrity of the data, and simply, the transaction cannot lose the data once the transaction is submitted. In a relational database, in order to improve performance, a transaction is generally committed after transaction data (i.e., data involved in execution of the transaction, also referred to as transaction content) is written into a transaction log and a Buffer Pool (Buffer Pool).

Page (page): also called a page, data page, physical page, or physical page, a page is the basic unit of database system storage. For example, in a database system where the size of a page is 8 Kilobytes (KB), there are 128 pages per megabyte in the database system.

Transaction logging: in the relational database, the data involved in the execution of the transaction are recorded. The transaction log is a set of files that can be directly deposited on disk, which is typically a sequentially stored physical log. The transaction log is used to record historical modification information (typically physical modification information) for the page. For example, an event log is used to record historical modification information for bytes 10 (byte) to 13 on 1 page. The historical modification information of the page is recorded by using the object log, and the page is more efficiently generated in proportion, so that higher writing efficiency can be ensured. The transaction log is typically stored in one or more log files (e.g., 2 log files) in which transaction records (redo records) are stored, each transaction record also referred to as a transaction entry (redo entry), the transaction records being a set of change vectors (cv), also referred to as change carriers, each of which describes changes to a single data block in the database. For example: a salary value in an employee (employee) table is modified to generate a transaction record that includes a change carrier that describes changes to data blocks in the employee table. In the relational database, each database instance is provided with a corresponding transaction log.

After the data query instruction is processed by the transaction, the corresponding transaction data has changed in both the transaction log and the buffer pool, but the corresponding page has not been recorded in the storage medium (such as a disk), and such a page different from the content recorded in the buffer pool is called a dirty page. The process of recording dirty pages to the storage medium is referred to as scrubbing based on the data of the buffer pool. If a fault, such as a crash (crash), occurs when some dirty pages of the database are not written to the storage medium, the data in the buffer pool is lost, and thus a transaction log stored in the storage medium is needed to restore the data in the buffer pool. Therefore, the main function of the transaction log is to restore the data in the relational database, and the consistency and the integrity of the data in the relational database are ensured.

Metadata (metadata): also known as intermediate data or relay data, is data describing data (data about data), typically information describing properties of the data (property), and metadata may be used to support functions such as indicating storage locations, history data, resource lookups, or file records.

Slicing: is an independent data region (also referred to as storage space) in a data storage node that may be physically or logically isolated from other partitions.

Log Sequence Number (LSN): is an incremental shaping number that represents the total number of bytes that the transaction writes to the transaction log. Because it is always incremented, the LSN may serve as a unique identifier that represents the transaction data written at different times. The LSN is mainly used for recovering data when a database fails. The LSNs may be stored in the transaction log or in a buffer pool.

And (3) falling: refers to writing data to a storage medium. The storage medium is a non-volatile computer-readable storage medium, such as a magnetic disk, a read-only memory, or an optical disk.

Log first (WAL) principle: the method is also called a log advance principle, that is, when a transaction is committed, a transaction log is written in a storage medium, and then page data is modified, where the page data refers to data of a data page and an index page in a buffer pool.

Snapshot (snapshot): with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the time at which the copy began). The snapshot may be a copy of the data it represents or may be a replica of the data. The snapshot Technology in the embodiment of the present application may refer to a snapshot Technology of a New Technology File System (NTFS) or a Fourth generation extended File System (EXT 4). The snapshot technique applied to a database is also called database snapshot.

To improve the disaster tolerance of the database, the transaction log needs to be backed up. In some non-distributed relational databases (e.g., MySQL), a data lock (e.g., a global read lock) is added when the transaction log is backed up, and then unlocked after the transaction log is backed up. However, when the relational database is a distributed database, as shown in fig. 1, the log data of the transaction log may be divided into a plurality of pieces of log data, which are stored in the pieces of the data storage node, respectively, and the data in each piece is referred to as piece data. Typically, each shard data includes at least shard log data and may also include shard page data. The fragmented log data includes partial content of the log data, and the fragmented page data is generated based on the corresponding fragmented data, for example, the fragmented page data is obtained by converting the corresponding fragmented log data by the data storage node. It should be noted that, after a certain piece of fragmented log data in the fragmented data is converted into fragmented page data, the certain piece of fragmented log data may be deleted or retained, which is not limited in this embodiment of the present application. In the distributed database, a plurality of fragment data corresponding to the log data are mainly backed up. Because the fragment data is generated based on the log data, after the plurality of fragment data are acquired, if the log data need to be recovered, the log data can be recovered based on the acquired plurality of fragment data. However, if there are more pieces of data corresponding to one log data, and the time interval between the actual backup completion time and the backup start time is longer, the time for backing up the plurality of pieces of data is longer. If a plurality of fragmented data are directly and respectively backed up, incremental fragmented data may be generated in the backup process, the incremental fragmented data cannot be determined during data recovery, and the service consistency of the backed-up data cannot be ensured; if locking is performed in the process of backing up a plurality of fragmented data, the generation of incremental fragmented data is avoided, and the service interruption time is easily overlong. There is therefore a need for a method that enables efficient backup of data in a distributed database.

The embodiment of the application provides a data backup method which can be applied to a distributed relational database and solves the problems. By way of example, the distributed relational database may be constructed based on the file system of NTFS or EXT4, or may be constructed based on other distributed file systems or distributed storage systems. Such as the greenplus database or taurus db. Wherein greenplus DB is abbreviated gpdb. Referring to fig. 2, fig. 2 is a schematic diagram of an application environment of a Distributed Database System (DDBS) related to the data backup method according to the embodiment of the present application. The DDBS may be a server or a server cluster composed of a plurality of servers, and includes a Distributed Database Management System (DDBMS) and a DDB. In a distributed database system, an application program can transparently operate DDBS through DDBS, and data in DDB are stored in different local databases, managed by one or more DDBMS, run on different machines, supported by different operating systems, and connected together by different communication networks. Wherein, DDBS10 includes: a management node (also known as a database engine, coordinating data node, coordinator)101 and a plurality of data nodes 102. The DDBMS may be deployed on a management node 101 and the DDBs may be deployed on a plurality of data nodes (data nodes) 102.

The DDBS10 may include one or more management nodes 101, where the management nodes 101 are configured to manage the corresponding data nodes 102 and implement operations of the application programs 20 on the data nodes 102, such as data adding operations, data deleting operations, data modifying operations, or data querying operations.

In this embodiment, the management node 101 may be a single node, or a designated data node or an elected data node in the plurality of data nodes 102, which may be a server or a server cluster composed of a plurality of servers. In one implementation, each data node may be a server or a server cluster composed of a plurality of servers; in another implementation, each data node characterizes a set minimum processing unit of the DDBS. Illustratively, each data node may be a virtual machine or a container that manages and/or stores an application instance or a database executing processes of data. A distributed database may have one or more tablespaces (tablespaces), which are logical partitions of a relational database, with one tablespace typically belonging to one database. A tablespace typically includes one or more relational tables.

Further, the management node 101 may also perform backup of the log data of the task log and the corresponding fragment data. As shown in fig. 3, fig. 3 is a schematic diagram of another application environment of a distributed database system related to the data backup method provided in the embodiment of the present application. Fig. 3 illustrates a distributed database system by taking a management node 101 in a DDBS10 as an example, and it is assumed that the management node 101 includes: processing sub-node 1011, backup sub-node 1012, and Virtual Router (Virtual Router)1013, the plurality of data nodes 102 comprising a plurality of data storage nodes, the plurality of data storage nodes comprising data storage node a through data storage node G. The plurality of data storage nodes typically occupy memory space of the distributed database system.

The processing sub-node 1011 is configured to receive a data processing request (such as a read, write, add, delete, or modify request of data), and perform data processing on a target data storage node through the virtual route 1013 based on the received data processing request. The processing sub-node 1011 supports a Structured Query Language (SQL) function. The function of the processing sub-node 1011 may be implemented by a database instance, which is the subject of data writing. The backup sub-node 1012 is a dispatcher of the backup process, and is configured to send respective requests (or instructions) in the backup process to respective data nodes in the DDBS through the virtual route 1013 to backup the log data and the fragmented data, where the requests may include a log backup request or a snapshot request related to a subsequent embodiment. The virtual route 1013 is used for routing the request to implement distribution of the request, so that the data node receiving the request performs an action corresponding to the request, such as reading and writing data. The functions of the processing sub-node 1011, the backup sub-node 1012 and the routing sub-node may be implemented by an independent process, or by the same process, where the same process is a process in the database instance.

The data storage nodes can be divided into log storage nodes and fragment storage nodes due to different types of data stored in the data storage nodes. The log storage node is used for storing log data, each log data has 1 or more copies, the copies are called log data copies, and the content of one log data is stored through the copy. Different copies of the same log data are distributed on different log storage nodes. The fragment storage nodes are used for storing fragment data, each fragment data has 1 or more copies, and the content of one fragment data is stored through the copy. Different copies of the same piece of fragmented data are distributed in fragments of different fragment storage nodes. FIG. 3 assumes that data storage nodes A-C are log storage nodes; the data storage nodes D to G are fragment storage nodes. The log data comprises 3 copies, namely log data copies 1 to 3, wherein the log data copies 1 to 3 are stored in data storage nodes A to C respectively; the 3 pieces of fragmented data corresponding to the log copies are respectively fragmented data 1 to 3, each piece of fragmented data in the 3 pieces of fragmented data has 3 copies, respectively copies 1 to 3, and 3 copies of each piece of fragmented data are respectively stored in 3 data storage nodes in the data storage nodes D to F. For example, copies 1 through 3 of sharded data 1 are stored at data storage nodes D, E and G, respectively; copies 1 through 3 of sharded data 2 are stored at data storage nodes D, E and G, respectively; copies 1 through 3 of sharded data 3 are stored at data storage nodes D, F and G, respectively.

It should be noted that, in the data backup method provided in the embodiment of the present application, the write-in mode of the transaction log or the page adopts an Append only (Append only) mode, and the mode supports only adding write-in data, and does not modify the written-in data in an overlay mode.

In an alternative way, the file system of the distributed relational database provided by the embodiment of the application supports the append-only mode. In the File system in the addition-Only mode, Only addition of a File is permitted but the File cannot be rewritten, and the generated File is called an addition-Only File (AOF).

In another alternative, the file system of the distributed relational database provided in the embodiment of the present application does not set the application only mode itself. However, no matter whether the distributed relational database is constructed based on the file system of NTFS or EXT4, or based on other distributed file systems or distributed storage systems, as long as the write mode of the transaction log or the page adopts the append-only mode, the backup can be performed by the data backup method provided by the embodiment of the present application.

For the convenience of the reader, the following description is provided to introduce the principles of the data backup method provided in the embodiments of the present application. In a distributed database, based on a log priority principle, after a transaction log is generated each time, writing log data of the transaction log into a corresponding data storage node, and realizing the diskless of the log data (namely storing a specified number of copies of the log data); after the log data of the transaction log falls, the log data is divided into a plurality of fragment data, and then the fragment data is written into the corresponding data storage nodes, so that the falling of the fragment data is realized (namely, the specified number of copies of each fragment data are stored). The fragment data comprises at least one of fragment log data and fragment page data. The fragmented page data is generated based on the corresponding fragmented log data, for example, the fragmented page data is obtained by converting the corresponding fragmented log data. In a traditional distributed relational database, log data and fragment data are not provided with metadata correspondingly, and the embodiment of the application configures the metadata for the log data and the fragment data so as to position incremental fragment data.

As shown in fig. 4, fig. 4 is a schematic time line diagram of a data backup method according to an embodiment of the present application. In the embodiment of the application, after receiving the log backup request, the management node starts to backup the log data and the fragment data. In order to obtain incremental fragmented data generated in the fragmented data backup process, it is necessary to determine a target backup time period, and for convenience of understanding of readers, in the following embodiments, a start time of the target backup time period is referred to as a first time, and an end time of the target backup time period is referred to as a second time. The target backup time period comprises a metadata backup time period of the fragment data and a metadata backup time period of the log data, after the target backup time period is determined, the fragment data can be recovered through the metadata of the fragment data acquired in the time period, and the recovered fragment data at least comprises the fragment data before the first time. The incremental fragment data in the time period is recovered through the metadata of the log data in the time period, so that the final fragment data can be determined through the recovered fragment data and the incremental fragment data, and the consistency of the backup data is ensured. For the convenience of the reader, the following embodiments refer to the metadata of the log data as the first metadata and the metadata of the sharded data as the second metadata.

The embodiment of the present application provides a data backup method, which is applied to an application environment shown in fig. 2 or fig. 3, and all or part of the data backup method may be executed by the aforementioned management node, for example, by a backup child node in the management node. In the embodiment of the present application, there are multiple implementation manners of the data backup method, and for different implementation manners, corresponding data recovery methods are also different.

In a first implementation, the indication information indicating the first time and the second time is determined during the data backup phase. As shown in fig. 5, the data backup method includes:

step 201, the management node stores the log data.

When the management node detects a data processing request sent by an application program, the management node controls a corresponding data node to execute the operation requested to be executed by the data processing request through the transaction. For example, the management node may initiate a plurality of distributed transactions based on the data processing request, generate one or more distributed plans upon execution of each distributed transaction, and instruct the corresponding data node to execute the generated distributed plans, thereby implementing the operations requested by the data processing request.

In the process of executing the transaction, corresponding transaction data is generated, and the management node writes the transaction data into the transaction log correspondingly, so as to store the log data, that is, to drop the log data.

Step 202, the management node stores the fragment data corresponding to the log data.

In the distributed database, after the log data is landed, the management node may perform fragmentation processing on the log data to obtain a plurality of fragmented data, and store the plurality of fragmented data on the fragments of the corresponding data storage nodes, that is, perform landing on the plurality of fragmented data. Each sharded data includes at least one of sharded log data and sharded page data.

Optionally, the management node stores fragment relation data, where the fragment relation data is used to record a correspondence between sub-data of the log data and the fragments, and one (or one) sub-data may include one or more transaction records. For example, the sharding relationship data may be maintained by a processing child of the management node in FIG. 3, such as by a process in the processing child. The shard relationship data may be stored in the form of a table and is therefore also referred to as a shard list. The fragment list can update the corresponding relationship between the subdata of the log data and the fragments in a full-updating mode, that is, when the corresponding relationship needs to be updated each time, the original fragment list is replaced by a new fragment list; the fragment list may also update the corresponding relationship between the sub-data of the log data and the fragments in an incremental update manner, that is, after the fragment list is stored for the first time, if the corresponding relationship needs to be updated, only data different from the data recorded in the original fragment list is updated, that is, incremental data is updated.

Optionally, the fragment relation data may include a first corresponding relation and a second corresponding relation, where the first corresponding relation is used to record a corresponding relation between sub-data of the log and the database object, and the second corresponding relation is used to record a corresponding relation between the database object and the fragment, where the database object refers to an object targeted by a data processing process in a relational database. Which may be a tablespace (e.g., a relational table), a page, an index, or other object related to a database classification (also referred to as a home).

For example, the first corresponding relationship may record a corresponding relationship between the identification information of the sub data and the identification information of the page, and the second corresponding relationship may record a corresponding relationship between the identification information of the page and the identification information of the segment. The correspondence between the identification information of the sub-data and the identification information of the page is usually a many-to-one correspondence, and the correspondence between the identification information of the page and the identification information of the segment is usually a one-to-one correspondence, or a many-to-one correspondence, for example, one segment may correspond to 100 pages. The identification information of the page is used for uniquely identifying one page, and the identification information can be composed of one or more characters. For example, the character may be a binary character, such as 0 or 1, or a decimal character. It should be noted that one or more relational tables may be stored in a relational database, each of which includes a plurality of pages. In order to implement unique identification of a page, when only one relationship table is stored in the relationship database, the identification information of the page may only include an identity document (ID, identification for short) of the page; when a plurality of relationship tables are stored in the relationship database, the identification information of the page may include the IDs of the relationship tables and the IDs of the page.

The identification information of a fragment is used to uniquely identify a fragment, and may be composed of one or more characters. For example, the character may be a binary character, such as 0 or 1, or a decimal character. It should be noted that, when the distributed relational database includes a plurality of data storage nodes, each data storage node includes a plurality of shards. In order to realize the unique identification of the fragments, in an optional mode, different fragment IDs can be allocated to all the fragments as the identification information of the corresponding fragments; in another alternative, the shard IDs on different data storage nodes overlap, and the identification information of the shard may include the ID of the data storage node and the shard ID.

Each database object, when operated on, generates log data. The management node may perform fragmentation processing on the log data based on the fragmentation relation data. For example, for each subdata of the obtained log data, the management node may query the first corresponding relationship to obtain a database object corresponding to the subdata; then, the database object corresponding to the sub-data is used to query the second correspondence to obtain corresponding fragments (i.e. determining the fragment corresponding to each sub-data, which may be regarded as mapping each sub-data to the corresponding fragment). Finally, the management node determines the sub-data corresponding to the same fragment as one fragment log data, and then writes each fragment log data into the corresponding fragment, that is, completes the storage of the fragment log data, after one or more fragment log data are stored, the fragment log data can be converted into fragment page data, and finally, the fragment data stored in one fragment can include the fragment log data and/or the fragment page data.

It should be noted that the fragment relationship data may also only include the second correspondence, and each sub-data carries identification information of its corresponding database object (also referred to as the database object) when being generated. For each subdata of the obtained log data, the management node may query the second corresponding relationship by using the identification information of the database object carried by the subdata to obtain a corresponding fragment. There is no need to query two correspondences. For example, assuming that the database object is a page, the second corresponding relationship may be as shown in table 1. The pages with the page IDs of 00-04 are respectively in one-to-one correspondence with the fragments with the fragment IDs of 0-3.

TABLE 1

Page ID	Fragment ID
		00	0
01	1
		02	2
03	3

Assuming that determining currently acquired log data by querying the first corresponding relationship or extracting the page ID carried in the sub data includes: if the subdata A belongs to the page 00 and the subdata B belongs to the page 01, the table 1 is inquired, and the subdata A can be written into the fragment 0 as fragment data; and writing the sub-data B into the fragment 1 as fragment data. Therefore, storage of the fragment data included in the log data is realized.

Step 203, the management node acquires first indication information, where the first indication information is used for indicating a first time.

In the embodiment of the application, after receiving the log backup request, the management node starts to backup the log data and the fragment data. The first time is a starting time of the target backup period. The first time is usually the same time as the subsequent time when the first metadata acquisition is started (i.e. the first metadata acquisition time), or the time difference between the first time and the subsequent time is ignored.

The first indication information for indicating the first time may be a timestamp, such as an atomic clock timestamp, a globally unique identifier, or a global Slice-to-disk lsn (global Slice persistence lsn) of the first time.

When the first indication information is a timestamp, each transaction record of the log data which has fallen from the disk carries a corresponding timestamp. The management node directly acquires the corresponding timestamp at the first moment; when the first indication information is the global unique identifier, the transaction record of each dropped log data carries the corresponding global unique identifier. The management node directly acquires the corresponding global unique identifier at the first moment.

In the current transaction log, the log data of each transaction log comprises one or more transaction records, and each transaction record carries an LSN. As shown in fig. 6, fig. 6 is a schematic diagram of a structure of a transaction record. The transaction record includes a log type bit, a log length bit, an LSN bit, a data bit, and a check bit. Wherein, the log type bit is used to record the modification mode (also called modification type) of the transaction log to the page, for example, the content of the log type bit is used to indicate that 2 bytes are modified, a blank page is created, or the file name is modified, etc., and the log length bit is used to record the length of the transaction record, for example, the length is 100 bytes; the LSN bit is used to record the LSN of the transaction record; the data bit is used for recording the actual data (or actual content) in the transaction record; the check bits are used for recording check values, and the check values are used for checking the accuracy of data carried by the data bits. The length of the log type bit, the log length bit, the LSN bit and the check bit is fixed. For example, the log type bit is 4 bytes long, the log length bit is 8 bytes long, the LSN bit is 8 bytes long, and the check bit is 8 bytes long.

Because the fragmented log data is obtained based on the log data, the data structure of the fragmented log data is the same as that of the log data and carries the LSN; and the fragment page log data is obtained by converting the corresponding fragment log data and also carries the LSN of the corresponding fragment log data. Therefore, the LSN is carried in both the log data and the shard data of the transaction log. The global fragment destaging LSN is determined based on the LSN in the fragment data, the first moment can be identified based on the original data structure in the transaction log by adopting the global fragment destaging LSN, and the accuracy of the determined first moment is ensured on the premise of reducing the influence on the whole file system as much as possible.

The global slice destaging LSN is the largest LSN among LSNs corresponding to slice data that has been destaged globally and continuously at a corresponding time (in this embodiment, the first time). Wherein, the fallen tray is globally continuous and the fallen tray is globally continuous. In the embodiment of the application, the dropped-disk data of each piece of fragmented data may be obtained first, and then the continuous dropped-disk data is obtained from the obtained dropped-disk data, and is used as the globally continuous dropped-disk fragmented data. It should be noted that, because the fragmented page data is obtained by conversion after the fragmented log data is landed, that is, the carried LSN is the same as the LSN carried by the corresponding fragmented log data. Therefore, the global fragment destaging LSN may be the largest LSN among LSNs corresponding to fragment log data that has been globally and continuously destaged at the corresponding time, and the LSNs in the fragment page data do not need to be considered.

A (or called a copy of) sliced data is dropped means that each copy of the sliced data is stored. For the same piece of data, the writing speed is correspondingly different according to the different storage media to be written, so the data dropped from the disk will change continuously, and the corresponding LSN will also change continuously.

Taking fig. 7 as an example, fig. 7 is a schematic diagram of a storage state of fragmented log data in fragmented data at a first time, assuming that copies 1 to 3 of fragmented log data 1 need to be stored in data storage nodes D, E and G, respectively, and LSNs of transaction records in fragmented log data 1 are LSNs 100 to LSN 300; copies 1 to 3 of the fragmented log data 2 need to be stored in the data storage nodes D, E and G, respectively, and LSNs of transaction records in the fragmented log data 2 are LSNs 301 to LSN 200; copies 1 through 3 of sharded log data 3 need to be stored at data storage nodes D, F and G, respectively, with LSNs of transaction records in sharded log data 3 being LSNs 201 through 200. It is assumed that LSNs 100 to 200 represent consecutive LSNs. At the first time, LSNs of data that copy 1 of fragmented log data 1 has written to data storage node D are LSN100 to LSN300, LSNs of data that copy 2 of fragmented log data 1 has written to data storage node E are LSN100 to LSN140, and LSN160 to LSN170, and LSNs of data that copy 3 of fragmented log data 1 has written to data storage node E are LSN100 to LSN140, and LSNs 171 to LSN 180. The LSNs of the transaction records of the sliced log data 1 destage are LSN100 to LSN 140. Similarly, the LSNs 301 to 350 of the transaction records whose fragmented log data 2 has been landed in fig. 7, and the LSNs 401 to 460 of the transaction records whose fragmented log data 3 has been landed in fig. 7.

The continuous disk dropping means that the data are dropped according to the sequence of the LSNs from small to large. Once the LSN is broken in the middle, the following data is not continuous data. The global continuous disk dropping means that all the sliced data are continuously dropped. That is, all the sliced data are destaged in the sequence of LSN from small to large.

With continued reference to fig. 7, the LSNs of the transaction records of which the fragmented log data 1 to 3 have been landed are LSN100 to LSN140, LSN301 to LSN350, and LSN401 to LSN460, respectively. Although the transaction logs with LSNs 301 to 320 may be regarded as continuous destaging for the sliced log data 2, there is a break from the LSN between LSN140 and LSN301 for all the sliced log data, i.e. the sliced log data 1 to 3, and therefore the LSNs corresponding to the data of the global continuous destaging are LSNs 100 to LSN 140. Wherein, the maximum LSN is LSN140, and the global slice destaging LSN is LSN 140.

In this embodiment of the application, the first indication information indicating the first time may also be other information as long as the first time can be represented. It may also be equal to the global tile landing LSN-K, K ≧ 1, K being a number within an acceptable error range, e.g., g ═ 1 or 2, for example.

Step 204, the management node obtains the first metadata of each piece of data in the plurality of pieces of data.

The fragment data is obtained by performing fragment processing on the log data, that is, the fragment data is corresponding to one (or one) log data. In the embodiment of the present application, the data corresponding to the transaction log includes log data and metadata of the log data. The log data is data related to the transaction in the execution process and is also called transaction content; metadata is used to describe the log data. The log data can be located through the metadata, namely, the storage position of the log data is determined.

According to the data backup method provided by the embodiment of the application, the writing mode of the transaction log or the page adopts the only-adding mode, and in the file system adopting the mode, the size of the file (namely the data volume which can be written in the file) is provided with an upper limit. If the data to be written is large (i.e. the size of the data is larger than the upper limit), it needs to be divided into multiple files for writing. Therefore, after the data is landed, the data of the landed data is stored in at least one file. For example, one log data is stored in at least one file (typically at least two files) and one page data is stored in at least one file. When the data of the landed disk needs to be stored in a plurality of files, the plurality of files need to have a certain order. When the data is restored, the files can be linked in sequence, namely the files have a precedence relationship, and the files can be connected through the precedence relationship. Therefore, in this embodiment of the present application, the metadata of the log data may include: and writing data of each file in at least one file, wherein the at least one file is used for storing log data. That is, each file includes two parts: one part is used for storing log data and the other part is used for storing data writing information. Wherein, a fixed-size designated location (also referred to as a designated space) may be set in each file for storing data writing information. For example, the specified location may be a header or an end of the file. The data writing information is used for describing the writing situation of the log data in the file. For example, the data writing information is used to record the position of the corresponding file in at least one file to which the corresponding file belongs, and the end position of the written data (i.e., the written log data) in the corresponding file. The metadata can effectively identify the position of each file, so that the files are sequentially and continuously acquired when the data is recovered, the continuity of the files is ensured, and effective log data are recovered.

In a file system in which the write mode of the transaction log or the page adopts the append only mode, the file can only be written, and the data volume is increased. The end position of the written data in the file can thus be identified by the amount of data written to the file. Alternatively, for any file, after each data writing, the total amount of currently written data (i.e. the sum of the data written at all writing times before the current time) may be used to identify the end position of the written data in the file. For example, suppose that, for the file F1, which records 100 bytes of data written at 3 times, respectively, the management node determines the amount of data written at the corresponding time and records the amount of data each time data is written. It is assumed that the management node writes 3 pieces of indication information indicating that the data amount is 100 bytes, 200 bytes, and 300 bytes at the specified positions at 3 times, respectively. If the indication information is a numerical value of the data quantity, such as 100, 200 and 300. In this way, each time the end position of the write data is queried, the end position of the write data can be quickly determined by directly querying the indication information of the most recently written data amount. For example, if the last written indication information obtained by the query is 300, the end position of the written data is determined to be the position where 300 bytes of data are written.

As previously described, LSN characterizes the total number of bytes written by a transaction to a transaction log. It is clear that LSN is an indication of the typical amount of data for a transaction log. Since log data of a file is written continuously. As the number of transaction record writes increases, the LSN also increases. Thus, the largest LSN carried by the written data in the file (i.e., the LSN of the most recently written transaction record) can reflect the end location of the written data. In this embodiment of the present application, for any file in the at least one file corresponding to the log data, the data writing information of the any file may include: and the maximum LSN carried in the written data in any file is used for identifying the end position of the written data.

For example, for the file F2, it records data written at 3 times. Accordingly, each time data is written, the maximum LSN of the data written this time is determined by reading the LSN bit in the transaction record in the data written at the corresponding time (the structure of the transaction record may refer to fig. 6 mentioned above), and the maximum LSN of the data written this time is recorded in the metadata to identify the end position of the data written this time. Suppose the maximum LSN of the 3 writes of data is 0xFFFFFFFFF000, 0 xFFFFFFFFFFFFF 00F, and 0 xFFFFFFFFFFFFFFF 0FF, respectively. The LSN finally recorded in this file F2 by the management node includes: 0xFFFFFFFFF000, 0xFFFFFFFFF00F, and 0 xFFFFFFFFFFFFF 0 FF. Wherein, the maximum LSN: 0xFFFFFFF 0FF identifies the end location of the most recent write data.

Optionally, the data writing information of any file corresponding to the log data further includes: the file location identification of any of the files. The file location identifier is used for identifying the location of the any file in at least one file to which the any file belongs. Wherein the file location identifier may be composed of one or more characters. For example, the character may be a binary character, such as 0 or 1, or a decimal character.

In an alternative, the file location identifier may be an identifier of a file, since the identifier of some files itself has a function of identifying their location. For example, the log data is stored in 3 files, the identifiers of the three files are allocated according to the arrangement sequence, and are respectively 00, 01 and 10, and then the file position identifier in the file 00 is 00.

In another alternative, the file location identification may include: an identification of a previous file and/or an identification of a subsequent file of the arbitrary file. It is worth to be noted that, when the any file is the first file of the at least one file, the previous file is empty; when the any file is the tail file of the at least one file, the next file is empty; when the log data is stored in only one file, that is, the at least one file is substantially only one file, the data writing information in the file includes the identifier of the previous file and the identifier of the next file of the file, which are both null. Still taking the example that the log data is stored in the aforementioned 3 files, the file location identifier in file 00 is 01 (i.e. only the identifier of the next file is included); the file location identification in file 01 is 00+10 (i.e., includes the identification of the previous file and the identification of the next file); the file location identification in file 10 is 01 (i.e., only the identification of the previous file is included).

In the embodiment of the present application, the data corresponding to each shard includes shard data and metadata of the shard data. Wherein the metadata is used to describe the sliced data. By means of the metadata, the fragment data can be located, that is, the storage position of the fragment data is determined.

Since the fragmented log data is obtained by dividing the log data, each fragmented log data is actually a part of the log data, and has the same attribute as the log data. Referring to the aforementioned log data, after the fragmented log data is landed, one (or one) fragmented log data is stored in at least one file. And the data storage node may convert part of the sharded log data in the shards into sharded page data, which is stored in at least one file. It should be noted that the files stored in the fragment log data and the fragment page data belonging to the same fragment may be the same or different. For example, in the same segment, segment log data is stored in files 1 to 3, and segment page data is stored in files 4 to 6; for another example, the fragmented log data is stored in files 1 to 6, and the fragmented page data is also stored in files 1 to 6, where each file stores partial fragmented log data and partial fragmented page data. The storage mode of the fragmented log data and the fragmented page data in the file is set according to the storage rule of the database system, which is not limited in the embodiment of the present application.

In this embodiment of the present application, the metadata of the sharded data may include: and writing data of each file in at least one file, wherein the at least one file is used for storing log data. That is, each file of the memory slice data includes two parts: one part is used for storing the fragmented data and the other part is used for storing data writing information. Wherein, a fixed-size designated location may be set in each file for storing data writing information. For example, the specified location may be a header or an end of the file. The data writing information is used to describe the writing condition of the fragment data in the file, for example, the data writing information is used to record the position of the corresponding file in the at least one file to which the corresponding file belongs, and the end position of the written data (i.e. the written fragment data) in the corresponding file. The position of each file can be effectively identified through the metadata, so that the files are sequentially and continuously acquired when the data are recovered, the continuity of the files is ensured, and the effective fragment data are recovered.

Similarly to the foregoing file for storing log data, in this embodiment of the present application, for any file in at least one file corresponding to the fragmented data, the data writing information of the any file includes: and the maximum LSN carried in the written data in any file is used for identifying the end position of the written data.

Optionally, the data writing information of any file corresponding to the fragment data further includes: the file location identification of any of the files. The file location identifier is used for identifying the location of the any file in at least one file to which the any file belongs. The structure of the file location identifier may refer to the structure of the file location identifier corresponding to the log data.

The structure of the file storing the fragmented data and the structure of the file storing the log data may be the same or similar, so the structure of the file storing the fragmented data is not described in detail in this embodiment of the application.

In this embodiment of the application, the management node may sequentially (also referred to as serially) obtain the first metadata of each of the plurality of fragmented data, or may obtain the first metadata of each of the plurality of fragmented data in parallel.

Optionally, the management node may obtain the first metadata by using a snapshot technique, so that the speed of obtaining the first metadata can be effectively increased, and the backup efficiency is improved.

And step 205, the management node obtains second metadata of the log data.

Optionally, the management node may obtain the second metadata by using a snapshot technique, so that the speed of obtaining the second metadata can be effectively increased, and the backup efficiency is improved.

Step 206, the management node obtains second indication information, where the second indication information is used to indicate a second time.

The aforementioned second time is the end time of the target backup period. It should be noted that the second time is usually the same as the time of acquiring the second metadata (i.e., the end time of step 205), or the time difference between the two times is ignored.

The second indication information used for indicating the second time is consistent with the type of the first indication information, so that the consistency of indication modes of the first time and the second time can be ensured, the first time and the second time are obtained in the same mode, and the determined timeliness of the target backup time period is improved. Corresponding to the first indication information, the second indication information may be a timestamp, such as an atomic clock timestamp, a globally unique identifier, or a continuous landing lsn (flush to disk lsn) at the second time.

The continuous destaging LSN is determined based on the LSN in the log data, the first moment can be identified based on the original data structure in the transaction log by adopting the continuous destaging LSN, and the accuracy of the determined second moment is ensured on the premise of reducing the influence on the whole file system as much as possible.

The LSNs of consecutive disk drops are the largest LSNs among the LSNs corresponding to the log data that have been continuously dropped at the corresponding time (in this embodiment, the second time). Wherein, the continuous falling refers to the falling and continuous falling. In the embodiment of the application, the dropped-disk data of the log data may be obtained first, and then the data of the continuous dropped-disk in the obtained dropped-disk data may be used as the log data of the continuous dropped-disk.

A log data is landed means that each copy of the log data has been stored. For the same log data, the writing speed is different according to the different storage media written, so the continuous disk-drop LSNs are changed continuously.

Taking fig. 8 as an example, fig. 8 is a schematic diagram of a storage state of log data at a second time, and it is assumed that copies 1 to 3 of the log data need to be stored in the data storage nodes a to C, respectively, and LSNs of transaction records in the log data are LSNs 700 to LSN 1000. Assume that, at the second time, LSNs of data that copy 1 of the log data has been written to data storage node a are LSNs 700 to LSN750, LSNs of data that copy 2 of the log data has been written to data storage node B are LSNs 700 to LSN720, and LSNs of data that copy 3 of the log data has been written to data storage node C are LSNs 700 to LSN 800. The LSNs in the data storage nodes D, E and G corresponding to log data that have been continuously landed are LSNs 700 through LSN 720. Wherein the maximum LSN is LSN 720. The consecutive destage LSN of the log data is LSN 720.

Step 207, the management node stores backup data, where the backup data includes: a plurality of first metadata and second metadata.

The plurality of first metadata and the plurality of second metadata are used for recovering the incremental fragmentation data between the first time and the second time.

As described above, the log backup request is mainly used to indicate a plurality of fragmented data corresponding to the backup log data. The backup data in the embodiment of the application may further include log data, and the log data included in the backup data includes incremental log data of a target backup time period to restore incremental fragmented data. The examples of the present application take the following cases as examples:

in the first case, the backup data further includes log data acquired up to the third time (i.e., log data acquired at the upload time), the first indication information, and the second indication information. The third time is a specified time after the second time, for example, a time that is a specified time from the second time, and for example, a time at which all available log data are stored (for example, backup data need to be stored in the cloud storage space, and the third time is a time at which all log data acquired locally at the data storage node are uploaded). The first indication information is used for indicating a first time, and the second indication information is used for indicating a second time. In performing data recovery, as shown in fig. 9, log data 301 (hatched portion in the figure) corresponding to the target backup period may be determined in log data 30a acquired at the third time based on the first time and the second time indicated by the first indication information and the second indication information.

In the second case, the backup data further includes log data corresponding to the second metadata, the first indication information, and the second indication information. Accordingly, storing the backup data includes storing log data corresponding to the second metadata. That is, for at least one file for storing the log data, the storage of the at least one file is stopped at the end position indicated by the second metadata. For example, assume that the aforementioned file F2 is a file for storing log data, and the second metadata includes data write information corresponding to this file F2: LSN: 0xFFFFFFFFF0FF, the end position of the write data in the stored file F2 is LSN: the transaction record where 0 xffffffffff 0FF is located.

In performing data recovery, as shown in fig. 10, the log data 301 corresponding to the target backup period may be determined in the log data 30b obtained by the second metadata based on the first time and the second time indicated by the first indication information and the second indication information.

In the foregoing first case and the second case, since the log data is carried in the backup data mainly for obtaining the incremental log data of the target backup time period, it is only required to ensure that the log data in the backup data includes the incremental log data, and therefore, in the backup data, the start position of the log data is the position before the corresponding position at the first time. For example, the start position is a position corresponding to a time before the second time and a specified time from the second time, and the specified time is longer than the time of the target backup time period.

In a third case, the backup data further includes incremental log data generated between the first time and the second time in the log data; before the foregoing step 406, the analysis node further needs to determine incremental log data generated between the first time and the second time, that is, the log data corresponding to the target backup time period.

Optionally, the management node may determine, in the log data, incremental log data generated between the first time and the second time based on the first indication information and the second indication information.

Take the first indication information as the global slice destaging LSN and the second indication information as the continuous destaging LSN as an example. Referring to step 202, the management node determines a plurality of fragment data based on the log data through the fragment relation data. In an optional manner, the fragmentation data (e.g., fragmentation log data) is the same as the LSN of the log data to which the fragmentation data belongs, that is, the LSN carried by the transaction record in the fragmentation data is not changed when the fragmentation data is written into the corresponding fragmentation; in another optional mode, the fragment data is different from the LSN of the log data to which the fragment data belongs, that is, when the fragment data is written into the corresponding fragment, the LSN carried by the transaction record in the fragment data is changed (there is a mapping relationship between the LSN carried in the fragment data and the LSN carried in the log data to which the fragment data belongs). In the two alternatives, the correspondence between the LSN of the sliced data and the LSN of the log data to which the sliced data belongs may be recorded by establishing a data correspondence, such as being identical, or there is a mapping.

The management node may determine, based on the data correspondence, a target LSN corresponding to the global fragmented destaging LSN in the log data, and then determine, based on the target LSN and the continuous destaging LSNs, incremental log data, where an LSN corresponding to the incremental log data is an LSN between the target LSN and the continuous destaging LSNs. Referring to fig. 7 and 8, assuming that the slice data is the same as the LSN of the log data to which it belongs, the target LSN is equal to the global slice destage LSN, which is LSN 140. The continuous landing LSN is LSN 720. The LSNs corresponding to the incremental log data are LSNs 140 to LSNs 720, and the transaction records in which LSNs 140 to LSNs 720 are located are determined to be the incremental log data.

Accordingly, storing backup data includes storing incremental log data. That is, for at least one file used to store the log data, transaction records corresponding to LSNs between the target LSN and consecutive destage LSNs in the at least one file are stored. For example, assuming that the aforementioned file F2 is a file for storing log data, and the target LSN and the consecutive landed LSN are 0xffffffff 000 and 0 xffffffffff 0FF, respectively, transaction records in which 0 xffffffffffff 000 to 0 xffffffffffffff 0FF in the stored file F2 are located are stored.

When data recovery is performed, as shown in fig. 11, the analysis node may directly acquire incremental log data 301.

Since the log data may generate incremental data during the storage of the backup data, if the backup data is stored in the manner provided in the first case, that is, all the acquired log data are stored until the third time, the end position (corresponding position to the third time) of the stored data is inconsistent with the end position (corresponding position to the time between the first time and the second time) indicated by the corresponding metadata, and the incremental data (that is, the incremental data between the second time and the third time) is additionally stored, which increases the load of the storage medium. The backup data is stored according to the manners provided in the second and third cases, so that the end position of the stored data is consistent with the end position indicated by the corresponding metadata, thereby avoiding additional storage of incremental data and reducing the load of the storage medium. Further, the backup data is stored in the manner provided in the third case, which can also avoid additional storage of log data before the first time, and further reduce the load of the storage medium.

When the backup data further includes a plurality of pieces of sharded data corresponding to the plurality of first metadata, respectively, the storing of the backup data includes storing the plurality of pieces of sharded data corresponding to the plurality of first metadata, respectively. That is, for at least one file for storing each sliced data, the storing of the at least one file is stopped at the end position indicated by the first metadata. Assuming that the file F3 is a file for storing one piece of sliced data, the second metadata corresponding to the sliced data includes data writing information corresponding to the file F3: LSN: 0xFFFFFFFFF0F1, the end position of the write data in the stored file F3 is LSN: the transaction record where 0xFFFFFFFFF0F1 is located.

In the embodiment of the application, the data backup request is used for indicating to acquire the fragmented data within the target time length before the request issuing time so as to realize the backup of the recent fragmented data. The target duration is greater than the primary backup duration, and the target duration may be determined in a variety of ways. In an alternative, the target time period is a preset time period, such as ten minutes or one hour; in another alternative, the target duration is carried in a data backup request, which may be specified by a user; in yet another alternative, the target time length may be a time length from a time of issuing the previous data backup request to a time of issuing the current data backup request. The embodiment of the present application does not limit this.

Referring to step 202, since the management node determines the fragmentation relation data corresponding to the log data based on the fragmentation relation data, the fragmentation relation data also needs to be acquired when recovering the fragmentation relation data. In an alternative, the sharded relational data may be stored directly for use in data recovery; in another alternative, the fragmentation relation rule may be stored, so that when data is restored, fragmentation relation data is obtained based on the fragmentation relation rule; in yet another alternative, the fragmentation relation data has metadata, that is, the data corresponding to the fragmentation relation includes: the management node may store the metadata so as to obtain the fragmentation relationship data based on the metadata at the time of data recovery. Wherein the metadata is used to describe the sharding relationship data. The position of the corresponding relation recorded in the slicing relation data can be determined through the metadata. E.g. the slice-related data is stored in the form of a table, the metadata is used to record the position of each entry in the slice list. For convenience of identification, the embodiment of the present application refers to the metadata of the sharding relationship data as third metadata.

Similar to the storage mode of the log data and the fragment data, after the fragment relation data is landed, one (or one) fragment relation data is stored in at least one file. In this embodiment of the application, the third metadata of the fragmentation relation data may include: and writing data of each file in at least one file, wherein the at least one file is used for storing the fragmentation relation data. That is, each file storing sharding relationship data includes two parts: one part is used for storing the fragmentation relation data, and the other part is used for storing the data writing information. Wherein, a fixed-size designated location may be set in each file for storing data writing information. For example, the specified location may be a header or an end of the file. The data writing information is used to describe the writing situation of the fragmentation relation data in the file, for example, the data writing information is used to record the position of the corresponding file in at least one file and the end position of the written data (i.e. the written fragmentation relation data) in the corresponding file. The third metadata can effectively identify the position of each file, so that the files can be sequentially and continuously acquired when the data is recovered, the continuity of the files is ensured, and the effective fragment relation data can be recovered.

In this embodiment of the present application, for any file in at least one file corresponding to the fragmentation relation data, the data writing information of the any file includes: the offset (offset) of the written data in any file is used to identify the end position of the written data. The offset, also called effective address or intra-segment offset, refers to the difference between the ending address and the starting address of a memory cell.

Optionally, the data writing information of any file corresponding to the fragmentation relation data further includes: the file location identification of any of the files. The file location identifier is used for identifying the location of the any file in at least one file. The structure of the file location identifier may refer to the structure of the file location identifier corresponding to the log data.

In this embodiment, taking the example of storing the third metadata as an example, after all the second metadata are obtained in step 203, the data backup method further includes: and acquiring third metadata of the fragmentation relation data, wherein the fragmentation relation data is used for recording the corresponding relation between the subdata of the log data and the fragments. The definition of which can refer to the related content in the aforementioned step 202.

Optionally, the management node may obtain the third metadata by using a snapshot technique, so that the speed of obtaining the third metadata can be effectively increased, and the backup efficiency is improved.

It should be noted that, in a first implementation manner of the data backup method, the first indication information and the second indication information may also be obtained in other manners. For example, the first indication information may be determined by the acquired first metadata. Thus, the sequence of step 203 and step 204 may be reversed.

In a first optional manner, each dropped transaction record of the fragmented data carries a timestamp corresponding to a time of the dropped transaction record, and each first metadata of the fragmented data records a timestamp when the metadata is obtained (for example, when a snapshot of the first metadata is obtained by using a snapshot technique, a corresponding timestamp is recorded in the obtained snapshot), then the process of obtaining the first indication information includes: among the timestamps carried by all the acquired first metadata, the timestamp farthest from the current time (i.e., the timestamp of the first acquired first metadata) is determined as first indication information, and the first indication information can effectively indicate the starting time of acquiring the first metadata.

In a second optional manner, each dropped transaction record of the fragmented data carries a global unique identifier corresponding to a dropping time, and for example, the global unique identifier may represent a temporal order or a spatial order. The globally unique identification is assigned in an incremental manner. For example, it may be an increasing number, or sequentially increasing letters, etc. The first metadata of the fragment data is recorded with a global unique identifier when the metadata is obtained (for example, when a snapshot of the first metadata is obtained by using a snapshot technique, a corresponding global unique identifier is recorded in the obtained snapshot), and the process of obtaining the first indication information includes: in the global unique identifiers carried by all the acquired first metadata, the minimum global unique identifier (i.e. the global unique identifier of the first acquired first metadata) is determined as first indication information, and the first indication information can effectively indicate the starting time of acquiring the first metadata.

For example, the second indication information may be determined by the acquired second metadata. Corresponding to the first optional manner, each dropped transaction record of the log data carries a timestamp corresponding to a dropping time, and a timestamp when the metadata is obtained is recorded in the second metadata of the log data (for example, when a snapshot of the second metadata is obtained by using a snapshot technique, a corresponding timestamp is recorded in the obtained snapshot), then the process of obtaining the second indication information includes: and determining the timestamp carried by the second metadata as second indication information, wherein the second indication information can effectively indicate the end time of acquiring the second metadata.

Corresponding to the second optional manner, each transaction record of log data that has fallen disk carries a global unique identifier corresponding to the disk-falling time, and the global unique identifier is allocated in an incremental manner. The process of acquiring the second indication information includes recording a global unique identifier when the metadata is acquired in the second metadata of the log data (for example, when a snapshot of the second metadata is acquired by using a snapshot technique, a corresponding global unique identifier is recorded in the acquired snapshot), and then: and determining the global unique identifier carried by the second metadata as second indication information, wherein the second indication information can effectively indicate the end time of acquiring the second metadata.

The first indication information and the second indication information may also be obtained in other manners as long as the first time and the second time can be effectively identified, which is not described in detail in this embodiment of the application.

It should be noted that, in the foregoing steps 201 and 202, the action of storing the log data and the corresponding fragment data is a storage action when the management node normally works, that is, a normal record of the transaction log; the storage action for the backup data in step 207 is a backup action for the management node, and the corresponding data needs to be stored in the designated backup space. In the embodiment of the present application, it is assumed that the management node supports at least one of the following two backup functions:

the first backup function, the cloud backup function. Accordingly, the backup space is a cloud storage (cloud storage) space. As shown in fig. 12, fig. 12 is a schematic diagram of another application environment of a distributed database system related to the data backup method provided in the embodiment of the present application. The application environment further comprises: the cloud storage node 20 is a cloud storage node on which a cloud storage space is deployed, and the cloud storage node 20 may be one or more servers. In this application environment, the process of the management node storing the backup data of step 207 includes: and the management node uploads the backup data to a cloud storage space of the cloud storage node. After the management node acquires the backup data, the acquired backup data can be directly exported and uploaded without occupying additional storage space. The impact on the data storage nodes is small.

The second backup function, the local backup function. Accordingly, the backup space is a local storage space, and referring to fig. 3, the local storage space is deployed in the data storage node. In this application environment, the process of the management node storing the backup data of step 207 includes: and the management node stores the backup data into a local storage space of the data storage node.

It should be noted that the data storage node typically stores multiple copies of log data and multiple copies of sharded data. All copies of the same data have the same content. Therefore, in order to reduce the data amount of the backup data, especially reduce the network overhead of uploading the backup data during cloud backup, the same data usually only needs to be backed up once, that is, only one copy is backed up, and the copy is determined in a plurality of copies corresponding to the data, for example, randomly selected. As shown in fig. 12, when performing cloud storage, the management node uploads only the log data copy 1, the copy 1 of the shard data 2, and the copy 1 of the shard data 3.

Fig. 13 is a schematic time line diagram of a data backup method according to an embodiment of the present application. In the embodiment of the present application, a data backup process is described with reference to fig. 12 and 13, and it is assumed that a snapshot technique is used to obtain first metadata, second metadata, and third metadata, and for convenience of description, a snapshot of the first metadata is referred to as a first snapshot, a snapshot of the second metadata is referred to as a second snapshot, and a snapshot of the third metadata is referred to as a third snapshot. Referring to fig. 12 and 13, after receiving a log backup request, a management node starts backup, and after acquiring a global fragmented destaging LSN, the management node acquires a plurality of first snapshots, second snapshots, and third snapshots, then acquires a continuous destaging LSN, and then uploads backup data, where the backup data includes the plurality of first snapshots, second snapshots, and third snapshots, and fragmented data, log data, and fragmented relation data corresponding to the plurality of first snapshots, second snapshots, and third snapshots. And finishing the backup after finishing the data uploading. The time period from the acquisition of the global segment disk-dropping LSN (identifying the first time) to the acquisition of the continuous disk-dropping LSN (identifying the second time) is a target backup time period, the time periods of the acquisition of all the first snapshots are metadata backup time periods of the segment data, and the target backup time period comprises the metadata backup time period of the segment data.

Due to the fact that the global segment disk-dropping LSN and the continuous disk-dropping LSN are obtained, the target backup time interval can be determined, and based on the incremental log data of the log data in the target backup time interval, the incremental segment data of the target backup time interval can be determined based on the incremental log data. When the backup data is stored in the manner provided in the third case, only incremental log data of a target backup time period may be stored, and the time duration of the target backup time period is usually only a few minutes, so that the data volume of the incremental log data is small, and the load of the storage medium is effectively reduced. If the data backup is performed in a cloud storage mode, the network overhead of uploading backup data can be reduced.

In the conventional data backup method in the database, after the log backup request is issued, the time when all log data are backed up is taken as the backup completion time, and when the log data volume is large, the backup completion time is far away from the issuing time of the log backup request. If the backup process is visible to the user, the user needs to wait for a long time to see the prompt information of the completion of the data backup, and the user experience is poor.

In the embodiment of the application, the log backup request requires that the backup end time is as close as possible to the issuing time of the log backup request, so that overlong time delay is avoided. In the foregoing process of storing the backup data, the data in the backup data may be saved in parallel. I.e. after each acquisition of a piece of data, the data is saved. Since the data amount of the first metadata to the third metadata is small, the metadata can be quickly acquired and saved. Since the first metadata to the third metadata are acquired and stored before the second time. After the first time, the second time and the first metadata to the third metadata are determined, at least the fragmented data before the second time is recovered after other data (such as log data and fragmented data) in the backup data is subsequently acquired. Therefore, although the other data in the backup data is stored in the backup manner after the second time, the end time that the user can feel that the consistency of the data can be guaranteed is the second time, and therefore the second time can be regarded as the end time of the backup. The backup finishing time is very close to the issuing time of the log backup request, and the current backup requirement is effectively met. And when the snapshot technology is adopted to obtain the first metadata to the third metadata, the backup storage of the first snapshot to the third snapshot can be completed within the time level of minutes or even seconds, so that the time delay between the second time and the issuing time of the log backup request is further shortened, and the timeliness of data backup is improved. Therefore, the management node can send out prompt information of data backup completion after the first metadata to the third metadata are obtained, a user can perceive that the backup is completed quickly, and user experience is improved.

Due to the fact that the data volume of other data in the backup data is large, backup storage of other data can be carried out in some idle periods, and therefore influences on services are reduced.

Corresponding to the first implementation manner of the foregoing data backup method, an embodiment of the present application further provides a data recovery method, which may be applied to an application environment as shown in fig. 2, fig. 3, or fig. 12, and all or part of the data backup method may be executed by the foregoing management node. Optionally, as shown in fig. 14, fig. 14 is a schematic diagram of another application environment of the distributed database system related to the data backup method provided in the embodiment of the present application. The management node 101 may further include: a recovery sub-node 1014, the recovery sub-node 1014 being a dispatcher of the recovery flow, is configured to send respective requests (or instructions) in the recovery flow to respective data nodes in the DDBS through the virtual route 1013 to recover the log data and the fragmented data, where the requests may include log recovery requests related to subsequent embodiments. The data backup method may be performed by the recovery child node.

As shown in fig. 15, the data recovery method includes:

step 401, the management node obtains stored backup data, where the backup data includes: the metadata includes a first metadata, a plurality of first metadata, and a second metadata.

After receiving the log recovery request, the management node acquires the stored backup data to determine incremental fragment data between the first time and the second time. In the backup data, each first metadata is metadata of one piece of fragmented data, and a plurality of pieces of fragmented data corresponding to a plurality of first metadata are pieces of fragmented data of log data. As can be seen from the foregoing data backup method, the first time is a time when the first metadata starts to be acquired before the backup data is stored, and the second time is a time when the second metadata ends to be acquired before the backup data is stored.

The management node may retrieve the backup data in a designated backup space for storing the backup data. Corresponding to the foregoing step 407, since the management node stores the backup data in different backup spaces when supporting different backup functions, in the embodiment of the present application, the following backup functions are supported by the management node, respectively, and a manner of obtaining the backup data is described in this embodiment:

the first backup function, the cloud backup function. Correspondingly, the backup space is a cloud storage space. As shown in fig. 12, in the application environment, the process of the management node acquiring the stored backup data in step 401 includes: and the management node downloads the backup data from the cloud storage space of the cloud storage node.

The second backup function, the local backup function. Accordingly, the backup space is a local storage space, and referring to fig. 3, the local storage space is deployed in the data storage node. In this application environment, the process of the management node obtaining the stored backup data in step 401 includes: and the management node reads the backup data from the local storage space of the data storage node.

The management node may set a recovery space for storing the recovered data. The storage actions in the subsequent steps are all recovery actions of the management node, namely, actions of writing data into the recovery space.

Step 402, the management node restores the incremental fragment data between the first time and the second time based on the backup data.

Optionally, the process of restoring, by the management node, the incremental fragmented data between the first time and the second time based on the backup data may include the following steps:

step A1, the management node recovers incremental log data generated between the first time and the second time in the log data.

Optionally, the process of the management node recovering the incremental log data generated between the first time and the second time in the log data includes:

step A11, the management node obtains incremental log data generated between the first time and the second time in the log data.

Corresponding to step 207 of the aforementioned data backup method, the backup data may further include log data, where the log data included in the backup data includes incremental log data of the target backup time period. The examples of the present application take the following cases as examples:

In a first alternative example, it is assumed that the first indication information is global sliced-destaged LSNs, and the second indication information is consecutive destaged LSNs. For the explanation of the global segment destaging LSN and the continuous destaging LSN, reference may be made to the explanation corresponding to the foregoing step 404, which is not described in detail in this embodiment of the application. Please refer to the process for obtaining incremental log data in the third case of step 207 of the data backup method. The management node may determine, based on the obtained data correspondence, a target LSN corresponding to the global fragmented destaging LSN in the log data, and then determine incremental log data based on the target LSN and the continuous destaging LSNs, where the LSN corresponding to the incremental log data is an LSN between the target LSN and the continuous destaging LSNs. For a specific process, reference may be made to the content in the third case of step 207 of the data backup method, which is not described in detail in this embodiment of the application.

In the second case, the backup data further includes log data corresponding to the second metadata, the first indication information, and the second indication information. In performing data recovery, as shown in fig. 10, the log data 301 corresponding to the target backup period may be determined in the log data 30b obtained by the second metadata based on the first time and the second time indicated by the first indication information and the second indication information.

In the second case, the obtaining manner of the incremental log data may refer to the corresponding obtaining manner in the first case, which is not described in detail in this embodiment of the present application.

In a third case, the backup data further includes incremental log data generated between the first time and the second time in the log data; when data recovery is performed, as shown in fig. 11, the analysis node may directly acquire incremental log data 301.

The implementation processes of the three cases may refer to the implementation process corresponding to step 207, which is not described in detail in this embodiment of the application.

And step A12, the management node writes the incremental log data into a recovery space based on the second metadata, thereby completing the recovery of the incremental log data.

The structure of the second metadata may refer to the explanation corresponding to step 204 in the foregoing data backup method, which is not described in this embodiment of the application again. By the second metadata, a location of a file for storing at least one log data of the incremental log data and an end location of the write data in each file can be determined. And writing the incremental log data into the recovery space according to the file arrangement sequence indicated by the second metadata, wherein the end position of the written data in each file is the end position indicated by the second metadata. Thus, the recovery of the incremental log data can be realized.

Through the second metadata, a plurality of files to which the incremental log data belong can be arranged in sequence, and the data in the files are kept consistent, so that the files are linked in sequence and stored in sequence on the premise of keeping the data consistency.

Step a2, the management node recovers the plurality of sharded data based on the plurality of first metadata.

The structure of the first metadata may refer to the explanation corresponding to step 204 in the foregoing data backup method, which is not described in detail in this embodiment of the application. For each first metadata, a location of a file for storing at least one log data of the corresponding sharded data, and an end location of the write data in each file may be determined by the first metadata. And writing the corresponding fragment data into the recovery space according to the file arrangement sequence indicated by the first metadata, wherein the end position of the written data in each file is the end position indicated by the first metadata. Thus, the recovery of the corresponding fragment data can be realized.

Step A3, the management node determines at least one increment fragmentation data between the first time and the second time based on the recovered increment log data and the fragmentation relation data.

The fragment relation data is used for recording the corresponding relation between the sub-data of the log data and the fragments. The structure of the fragmentation relation data can refer to the corresponding explanation of the foregoing step 202.

Optionally, referring to step 202, it is assumed that the fragmentation relation data includes a first corresponding relation and a second corresponding relation. The management node may perform fragmentation processing on the incremental log data based on the fragmentation relation data. For example, for each subdata of the obtained incremental log data, the management node may query the first corresponding relationship to obtain a database object corresponding to the subdata; then, the database object corresponding to the sub-data is used to query the second correspondence to obtain corresponding fragments (i.e. determining the fragment corresponding to each sub-data, which may be regarded as mapping each sub-data to the corresponding fragment). Finally, the management node determines the subdata corresponding to the same fragment as fragment incremental log data. Thereby yielding at least one incremental sliced datum.

Step A4, the management node updates the plurality of fragment data based on the fragment relation data and the at least one increment fragment data, and the plurality of updated fragment data includes the at least one increment fragment data.

Since the plurality of fragmented data are recovered from the plurality of first metadata, and the first metadata are obtained in a target backup period between the first time and the second time, new data writing exists after some fragmented data are obtained in the target backup period. And at least one piece of incremental fragment data is the complete incremental fragment data in the target backup time interval, and the data volume of the incremental fragment data is greater than or equal to the sum of the data volumes of all the fragment data which are newly increased after the corresponding metadata of the fragment data are acquired in the target backup time interval. Accordingly, the at least one incremental shard data has a likelihood of data duplication with the recovered shard data. As shown in fig. 16, assuming that two pieces of sliced data obtained by recovery based on the second metadata and the sliced relation data are respectively sliced data 1 and sliced data 2, where the sliced data 1 corresponds to the second metadata obtained at the fourth time and the sliced data 2 corresponds to the second metadata obtained at the fifth time, incremental sliced data (both stored in the same slice) corresponding to the sliced data 1 is data 502 in fig. 16 and incremental sliced data (both stored in the same slice) corresponding to the sliced data 2 is data 503 in fig. 16. And at least one incremental fragment data obtained based on the incremental log data is data 501 in fig. 16, which includes data corresponding to fragment data 1 in the target backup period and data corresponding to fragment data 2 in the target backup period. As can be seen from fig. 16, the difference data obtained by subtracting the data 502 from the data 501 and subtracting the data 503 is data that may cause a problem of repeated writing.

Optionally, the update process of the incremental fragmentation data in the embodiment of the present application supports idempotent characteristics. The updating process comprises the following steps:

step A41, for each increment slicing data, detecting whether the data stored in the corresponding target slicing has the same data as the increment slicing data.

The target shard is the shard to which the incremental log data needs to be written as determined based on shard relationship data. For example, the analytics device may compare the LSN of the transaction record in the target sharded stored data to the LSN of the transaction record of the delta sharded data. If the transaction records with the same LSN exist, determining the transaction records with the same LSN as the same data; and if the transaction record with the same LSN does not exist, determining that the data stored in the target fragment does not have the same data as the incremental fragment data.

And step A42, when the data stored in the corresponding target fragment is different from the incremental fragment data, writing the incremental fragment data into the corresponding target fragment.

When the data stored in the corresponding target fragment is different from the incremental fragment data, it is indicated that the data stored in the target fragment and the incremental fragment data do not have repeated data, and the incremental fragment data can be written into the corresponding target fragment.

Step a43, when the corresponding target segment stores the data same as the incremental segment data, prohibiting writing the incremental segment data, or, using the incremental segment data to cover the data same as the incremental segment data in the corresponding target segment, where the target segment is the segment corresponding to the incremental segment data determined based on the segment relation data.

In a second implementation, the indication information for indicating the first time and the second time is determined in the data recovery phase. As shown in fig. 17, the data backup method includes:

step 601, the management node stores the log data.

Step 602, the management node stores the fragment data corresponding to the log data.

Step 603, the management node obtains the first metadata of each piece of sliced data in the plurality of pieces of sliced data.

And step 604, the management node obtains second metadata of the log data.

Step 605, the management node stores backup data, where the backup data includes: a plurality of first metadata and second metadata.

Wherein the backup data does not include the first indication information and the second indication information.

The foregoing steps 601 to 605 may refer to steps 201, 202, 204, 205, and 207 in the foregoing first implementation manner, respectively. This is not described in detail in the embodiments of the present application.

Corresponding to the second implementation manner of the foregoing data backup method, an embodiment of the present application further provides a data recovery method, which may be applied to an application environment as shown in fig. 2, fig. 3, fig. 12, or fig. 14, and all or part of the data backup method may be executed by the foregoing management node, for example, by the recovery child node in fig. 14. As shown in fig. 18, the data recovery method includes:

step 701, the management node obtains stored backup data, where the backup data includes: the metadata includes a first metadata, a plurality of first metadata, and a second metadata.

Step 702, the management node obtains first indication information, where the first indication information is used to indicate a first time.

The management node may determine the first indication information by the acquired first metadata. In the process, reference may be made to a process of determining the first indication information by using the obtained first metadata in the first implementation manner of the data backup method, which is not described in detail in this embodiment of the present application.

Step 703, the management node obtains second indication information, where the second indication information is used to indicate a second time.

The management node may determine the second indication information by the acquired second metadata. In the process, reference may be made to the process of determining the second indication information by using the obtained second metadata in the first implementation manner of the data backup method, which is not described in detail in this embodiment of the application.

Step 704, the management node restores the incremental fragment data between the first time and the second time based on the first indication information, the second indication information and the backup data.

The foregoing steps 701 and 704 may refer to steps 401 and 402 in the foregoing first implementation manner, respectively. This is not described in detail in the embodiments of the present application.

It should be noted that, the sequence of the steps of the data backup method and the data recovery method provided in the embodiments of the present application may be appropriately adjusted, and the steps may also be increased or decreased according to the circumstances, and any method that can be easily changed within the technical scope disclosed in the present application by a person skilled in the art should be included in the protection scope of the present application, and therefore, no further description is given.

An embodiment of the present application provides a data backup apparatus 80, as shown in fig. 19, the apparatus includes:

a first obtaining module 801, configured to obtain first metadata of each piece of fragmented data in multiple pieces of fragmented data, where the multiple pieces of fragmented data are obtained by performing fragment processing on log data; a second obtaining module 802, configured to obtain second metadata of the log data; the storage module 803 is configured to store backup data, where the backup data includes a plurality of the first metadata and the second metadata, and the backup data is used to restore incremental fragment data between a first time and a second time, where the first time is a time when the first metadata starts to be acquired, and the second time is a time when the second metadata ends to be acquired.

To sum up, in the embodiment of the present application, the backup data carries the first metadata of the multiple fragmented data and the second metadata of the log data, so that the incremental fragmented data between the first time and the second time can be effectively recovered based on the acquired backup data. On the premise of not adding a data lock, incremental fragment data is determined, so that the service consistency of backup data is ensured, effective data backup in a distributed database is realized, and database service is not influenced.

Optionally, the backup data further includes log data corresponding to the second metadata, first indication information, and second indication information, where the first indication information is used to indicate the first time, and the second indication information is used to indicate the second time.

Optionally, the backup data further includes incremental log data generated between the first time and the second time in the log data; as shown in fig. 20, the apparatus 80 further includes:

a determining module 804, configured to determine incremental log data generated between the first time and the second time in the log data based on first indication information and second indication information, where the first indication information is used to indicate the first time, and the second indication information is used to indicate the second time.

Optionally, the backup data further includes third metadata of the fragment relation data, and the fragment relation data is used to record a corresponding relation between the sub-data and the fragments in the log data.

Optionally, the backup data further includes: the fragment data corresponding to the first metadata and the fragment relation data corresponding to the third metadata.

Optionally, the metadata of any one of the log data and the shard data includes: data writing information of each file in at least one file, wherein the at least one file is used for storing any data; the data writing information of any file is used for recording the position of the any file in the at least one file and the end position of the written data in the any file.

Optionally, the data writing information of any file includes: the maximum log sequence number LSN carried in the written data in any file, and the file position identification of any file.

An embodiment of the present application provides a data recovery apparatus 90, as shown in fig. 21, the apparatus including:

an obtaining module 901, configured to obtain stored backup data, where the backup data includes multiple first metadata and multiple second metadata, each of the first metadata is metadata of one piece of fragmented data, and multiple pieces of fragmented data corresponding to the multiple first metadata are fragmented data of the log data; a first restoring module 902, configured to restore incremental fragmented data between a first time and a second time based on the backup data; the first time is the time when the first metadata starts to be acquired before the backup data is stored, and the second time is the time when the second metadata finishes to be acquired before the backup data is stored.

Optionally, the recovering module 902 is configured to: recovering incremental log data generated between the first time and the second time in the log data; recovering the plurality of sliced data based on the plurality of first metadata; determining at least one piece of incremental fragmentation data between the first time and the second time based on the recovered incremental log data and fragmentation relation data, wherein the fragmentation relation data is used for recording the corresponding relation between subdata of the log data and fragments; updating the plurality of sharded data based on the sharding relationship data and the at least one incremental sharded data, the updated plurality of sharded data including the at least one incremental sharded data.

Optionally, the backup data further includes incremental log data generated between the first time and the second time in the log data;

or, the backup data further includes log data corresponding to the second metadata, first indication information and second indication information, where the first indication information is used to indicate the first time, the second indication information is used to indicate the second time, and the restoring incremental log data generated between the first time and the second time in the log data includes: and incremental log data generated between the first time and the second time determined in the log data corresponding to the second metadata are generated based on the first indication information and the second indication information.

the second indication information includes: and the continuous disk-dropping LSN at the second moment is the largest LSN in the LSNs corresponding to the log data of which the disks are continuously dropped at the corresponding moment.

Optionally, the backup data further includes a third metadata of the fragmentation relation data, as shown in fig. 22, the apparatus 90 further includes: a second recovering module 903, configured to recover the fragmentation relation data based on the third metadata.

Optionally, the first recovery module is configured to: for each increment fragment data, when the corresponding target fragment stores the data which is the same as the increment fragment data, the increment fragment data is prohibited from being written in, or the increment fragment data is adopted to cover the data which is the same as the increment fragment data in the corresponding target fragment, and the target fragment is the fragment which is determined based on the fragment relation data and is corresponding to the increment fragment data.

Alternatively, FIG. 23 schematically provides one possible basic hardware architecture for a computer device as described herein.

Referring to fig. 23, the computer apparatus 1000 includes a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004.

In the computer device 1000, the number of the processors 1001 may be one or more, and fig. 23 illustrates only one of the processors 1001. Alternatively, the processor 1001 may be a Central Processing Unit (CPU). If the computer device 1000 has multiple processors 1001, the types of the multiple processors 1001 may be different, or may be the same. Alternatively, the plurality of processors 1001 of the computer apparatus 1000 may also be integrated into a multi-core processor.

Memory 1002 stores computer instructions and data; the memory 1002 may store computer instructions and data necessary to implement the data backup methods provided herein, e.g., the memory 1002 stores instructions for implementing the steps of a data backup method or a data recovery method. The memory 1002 may be any one or any combination of the following storage media: nonvolatile memory (e.g., Read Only Memory (ROM), Solid State Disk (SSD), hard disk (HDD), optical disk), volatile memory.

The communication interface 1003 may be any one or any combination of the following: a network interface (e.g., an ethernet interface), a wireless network card, etc. having a network access function.

The communication interface 1003 is used for the computer apparatus 1000 to perform data communication with other computer apparatuses or terminals.

A bus 1004 may connect the processor 1001 with the memory 1002 and the communication interface 1003. Thus, the processor 1001 may access the memory 1002 via the bus 1004 and may also interact with other computer devices or terminals using the communication interface 1003.

In this application, the computer device 1000 executes computer instructions in the memory 1002, so that the computer device 1000 implements the data backup method provided by this application, or so that the computer device 1000 deploys a data backup apparatus.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as a memory, including instructions executable by a processor of a server to perform a data backup method or a data restore method as shown in various embodiments of the present application, is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product comprising one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., solid state disk), among others.

In this application, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The term "plurality" means two or more unless expressly limited otherwise. A refers to B and refers to the simple variation where A is the same as B or A is B.

It should be noted that: in the data backup apparatus provided in the foregoing embodiment, when executing the data backup method, only the division of the functional modules is described as an example, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the data backup device and the data backup method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

46页详细技术资料下载

上一篇：一种医用注射器针头装配设备

下一篇：存储器装置以及安全开机的存储器管理方法

Data backup method, data recovery method and device

相关技术

网友询问留言