阅读说明：本技术 一种基于pm的数据库页缓存方法及系统 (Database page caching method and system based on PM ) 是由 闫宗帅 屠要峰 陈河堆 郭斌 黄震江 韩银俊 解海波 王涵毅 于 2020-06-15 设计创作，主要内容包括：一种基于PM的数据库页缓存方法,包括以下步骤,将表文件映射到内存；对缓存页进行区域划分；为划分区域后的缓存页设置对应的页面描述符；获取修改区域,并修改相应的页面描述符；将修改区域的数据刷写到PM内存。本发明还提供一种基于PM的数据库页缓存系统,减小记录页改动点开销的同时进一步减小写放大,提升性能,以并行的方式将cache line block刷盘,进一步提升刷盘效率和硬件使用寿命。(A database page caching method based on PM comprises the following steps of mapping a table file to a memory; carrying out region division on a cache page; setting a corresponding page descriptor for the cache page after the region division; acquiring a modification area and modifying a corresponding page descriptor; and flashing the data of the modified area to the PM memory. The invention also provides a database page caching system based on the PM, which reduces the cost of change points of record pages, further reduces write amplification, improves the performance, and further improves the disk-brushing efficiency and the service life of hardware by brushing the cache line block in a parallel mode.)

1. A PM-based database page caching method is characterized by comprising the following steps,

mapping the table file to a memory;

carrying out region division on a cache page;

setting a corresponding page descriptor for the cache page after the region division;

acquiring a modification area and modifying a corresponding page descriptor;

and flashing the data of the modification area to a PM memory.

2. The PM-based database page caching method according to claim 1, wherein said step of partitioning cache pages further comprises partitioning each cache page into a plurality of fixed-size sections, each of said sections having a size of 1 cache line.

3. The PM-based database page caching method of claim 2, wherein the cache page after zoning is structured into a header and a footer, wherein,

the header, comprising:

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: pointer arrays recorded in the pages;

the footer, comprising:

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

4. The PM-based database page caching method of claim 1, wherein said page descriptors, comprising,

tablespace id: the table space ID where the table is located;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: page number of data page on PM memory;

cache line bitmap: the modified area in the cache page is marked.

5. The PM-based database page caching method of claim 1, wherein said step of obtaining a modification region and modifying a corresponding page descriptor further comprises,

modifying the cache page;

pointing the upper limit pointer of the free space in the page to the last modification record;

pointing a pointer array of records in a page to the modification record;

acquiring a modification area;

modifying bit corresponding to a modification area in the page descriptor mark cache page to be 1;

and modifying the bit corresponding to the next area of the first area and the modified area into 1.

6. The method according to claim 5, wherein said step of flushing the data of said modified fields to the PM memory further comprises flushing the field with bit 1 to the PM memory by traversing all page descriptor arrays.

7. The PM-based database page caching method according to claim 6, further comprising the step of flushing the area with bit 1 to the PM memory, calling the PMDK interface in a parallel manner, and flushing the PM memory in a user mode to perform disk flushing persistence.

8. The method according to claim 1, further comprising, after the persistent flushing is completed, clearing 0 the bitmap in the page descriptor.

9. A database page cache system based on PM is characterized by comprising a PM memory, a DRAM memory, a region division module, a page description module and a disk flash module, wherein,

the PM memory is used for storing a table file;

the DRAM stores cache pages and page descriptors and writes modified areas to the PM memory according to the disk write module instruction;

the region division module is used for carrying out region division on the cache page;

the page description module is used for setting a corresponding page descriptor for the cache page;

and the disk flash module controls the DRAM memory, and writes the modified area into the PM memory for persistent disk flash.

10. The PM-based database page caching system of claim 9, wherein said region partitioning module partitions each cache page into a plurality of regions of size 1 cache line; the structure of the cache page is divided into a header and a footer, wherein,

the header, comprising:

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: pointer arrays recorded in the pages;

the footer, comprising:

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

11. The PM-based database page caching system of claim 9, wherein said page description module sets page descriptors comprising,

tablespace id: the table space ID where the table is located, namely the storage path of the table file;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: data page number on PM;

cache line bitmap (bitmap table): marking which areas in the cache page are modified.

12. The PM-based database page caching system of claim 9, wherein said disk flush module traverses all page descriptor arrays, controls said DRAM memory, invokes a PMDK interface in parallel, flushes to PM memory in a user-mode, and performs disk flush persistence.

13. An electronic device, comprising:

a processor; and

memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the PM-based database page caching method of any one of claims 1 to 8.

14. A computer readable storage medium storing one or more programs which, when executed, perform the steps of the PM-based database page caching method of any one of claims 1 to 8.

Technical Field

The invention relates to the technical field of data processing, in particular to a database page caching method and system based on a PM.

Background

The existing relational database system is a database system facing a magnetic disk, and the main architecture is on a two-layer storage hierarchy constructed by an HDD and a DRAM. Because the relational database data itself is large, it is impossible to store all of them in the memory, and the memory itself has no persistence, it is necessary to persist the data pages to the disk.

Conventional storage media are Block devices, the minimum unit for reading and writing is a Block (Block), I/O latency varies from millisecond to microsecond, and DRAM is nanosecond. In terms of performance, the storage engine does not directly modify the data pages on the physical disk, but reads the data pages to be modified into a memory buffer (herein, the mapping of the data pages in the buffer is referred to as a cache page), then modifies the cache page, and after the transaction is committed, selects a chance to flush the modified cache pages (referred to as dirty pages) to the disk, thereby completing the persistence. The storage engine introduces a page cache mechanism, and effectively solves the contradiction between the disk slow I/O and the high-speed CPU. However, the write amplification problem is caused by the write granularity of the page unit, even if a transaction only modifies a few bytes of a certain page, the whole dirty page is written to the disk in a persistent mode, and the effective throughput of the disk is severely limited.

Disclosure of Invention

In order to solve the technical problems, the invention provides a database page caching method and system based on a PM (particulate matter), which cache line blocks need to be refreshed according to a bitmap, write amplification is further reduced while the cost of change points of record pages is reduced, and the performance is improved.

In order to achieve the above purpose, the database page caching method based on the PM provided by the present invention comprises the following steps:

mapping the table file to a memory;

carrying out region division on a cache page;

setting a corresponding page descriptor for the cache page after the region division;

acquiring a modification area and modifying a corresponding page descriptor;

and flashing the data of the modification area to a PM memory.

Further, the step of dividing the cache pages into regions further includes dividing each cache page into a plurality of regions of fixed size, where the size of each region is 1 cache line.

Furthermore, the cache page after the partition is divided into a header and a footer, wherein,

the header, comprising:

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: pointer arrays recorded in the pages;

the footer, comprising:

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

Further, the page descriptors, including,

tablespace id: the table space ID where the table is located;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: page number of data page on PM memory;

cache line bitmap: the modified area in the cache page is marked.

Further, the step of obtaining the modification area and modifying the corresponding page descriptor further comprises,

modifying the cache page;

pointing the upper limit pointer of the free space in the page to the last modification record;

pointing a pointer array of records in a page to the modification record;

acquiring a modification area;

modifying bit corresponding to a modification area in the page descriptor mark cache page to be 1;

and modifying the bit corresponding to the next area of the first area and the modified area into 1.

Further, the step of flushing the data in the change modification area to the PM memory further includes traversing all page descriptor arrays, and flushing the area with bit 1 to the PM memory.

Further, the area with bit 1 is flashed to the PM memory, a PMDK interface is called in a parallel mode, and the PM memory is flashed in a user mode to carry out disk-flashing persistence.

Further, the bitmap in the page descriptor is cleared to 0 after the disk brushing persistence is finished.

In order to achieve the above object, the present invention further provides a PM-based database page caching system, comprising a PM memory, a DRAM memory, a region dividing module, a page description module, and a disk flushing module, wherein,

the PM memory is used for storing a table file;

the DRAM stores cache pages and page descriptors and writes modified areas to the PM memory according to the disk write module instruction;

the region division module is used for carrying out region division on the cache page;

the page description module is used for setting a corresponding page descriptor for the cache page;

and the disk flash module controls the DRAM memory, and writes the modified area into the PM memory for persistent disk flash.

Further, the area dividing module divides each cache page into a plurality of areas with the size of 1 cache line; the structure of the cache page is divided into a header and a footer, wherein,

the header, comprising:

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: pointer arrays recorded in the pages;

the footer, comprising:

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

Further, the page descriptor module, which sets the page descriptor, includes,

tablespace id: the table space ID where the table is located, namely the storage path of the table file;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: data page number on PM;

cache line bitmap (bitmap table): marking which areas in the cache page are modified.

Furthermore, the disk flash module traverses all page descriptor arrays, controls the DRAM memory, calls the PMDK interface in a parallel manner, and flashes the PM memory in a user mode to perform disk flash persistence.

In order to achieve the above object, the present invention further provides an electronic device, comprising a processor; and

a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the PM-based database page caching method described above.

In order to achieve the above object, the present invention further provides a computer readable storage medium, which stores one or more programs that, when executed, perform the steps of the above PM-based database page caching method.

The database page caching method and system based on the PM are applied to a database system, and the storage engine system is reconstructed by fully utilizing the new characteristics of non-volatility, byte addressing, large capacity, low delay and the like of the PM; the cache line aware page is designed to bring the following beneficial effects:

the initial position of each page change does not need to be accurately recorded, and only the change area needs to be recorded; all change points can be quickly positioned by only one bitmap, and the management overhead is very low. In the process of persistence of the cache page, which areas need to be persisted can be quickly judged only through the bitmap table; the characteristic of addressing according to bytes is fully utilized, when dirty pages are flushed, the whole data page does not need to be flushed to a disk any more, cache line blocks needing to be flushed are obtained according to a bitmap, the write amplification is further reduced while the change point overhead of the record page is reduced, and the performance is improved; the cache line block is brushed in a parallel mode, so that the disc brushing efficiency is further improved; the service life of hardware is prolonged while IO is reduced by using a cache line block disk brushing mechanism.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

FIG. 1 is a diagram illustrating a Cache page structure based on Cache-line according to the present invention;

FIG. 2 is a diagram illustrating a page descriptor data set structure according to the present invention;

FIG. 3 is a flow chart of a PM-based database page caching method according to the present invention;

FIG. 4 is a flow chart of inserting a record modification data page and flushing the disk in accordance with the present invention;

FIG. 5 is a PM-based database page caching system architecture diagram according to the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.

In the embodiment of the invention, the data page is persisted to the PM disk by utilizing the new characteristics of non-volatility, byte addressing, large capacity, low delay and the like of PM hardware.

In the embodiment of the present invention, it is not necessary to accurately record the start position of each page modification, and only a modification Area (Area) needs to be recorded: each cache page is divided into fixed-size regions, a bitmap is added to the page descriptor, each bit representing a region is set to 1 when the region data is changed (referred to as dirty). The advantage of this approach is that the management overhead is very low and only one bitmap is needed to quickly locate all change points. The smaller the zone is set, the more total number of flushes, and the higher the IOPS requirement for the PM, at a given load. According to the actual measurement result, the random read-write time delay of the Intel Optane DC Persistent Memory is about 300 nanoseconds, and the IOPS requirement can be easily met.

Since the minimum unit for exchanging data between the CPU Cache and the memory (PM is also a memory) is 1 Cache Line (Cache Line), the optimal performance can be obtained by setting the area size to 1 Cache Line.

Fig. 1 is a Cache page structure diagram based on Cache-Line in the present invention, and as shown in fig. 1, assuming that the size of 1 Cache page is 8KB and the size of 1 Cache Line is 64B, each Cache page is divided into 128 regions.

The data page structure is divided into a header and a footer, wherein,

the header includes: checksum, lower, upper, line, wherein,

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: an array of pointers recorded within a page.

The page tail comprises: prepage, nextpage, where,

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

For managing cache pages, each cache page has a corresponding page descriptor, fig. 2 is a schematic diagram of a structure of a page descriptor data group according to the present invention, as shown in fig. 2, descriptors are organized in an array form, and an array index is a page number of a data page in a memory. The page descriptor includes several member variables:

tablespace id: the table space ID where the table is located, namely the storage path of the table file;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: data page number on PM;

there are also 1 bitmap table of 16 bytes size: cache-line area bitmap to mark which areas are dirty. For example, inserting a record into 1 clean cache page, the data storage location spans 9 th and 10 th regions, and since page changes will involve a Header (located in 1 st region) change, the corresponding bitmap value is 0000000000000301. In the persistence process of the cache page, which areas need to be persisted can be quickly judged only through the bitmap table.

Example 1

Fig. 3 is a flowchart of a PM-based database page caching method according to the present invention, and the PM-based database page caching method of the present invention will be described in detail with reference to fig. 3.

First, in step 301, a table file is mapped to a DRAM memory.

In the embodiment of the invention, the table file of the PM memory is mapped into the DRAM memory through the PMDK.

In step 302, a cache page is subject to region partitioning and page description.

In the embodiment of the invention, the division of the Cache pages of the database into regions comprises the step of dividing each Cache page into regions with fixed sizes, wherein the size of each region is set to be 1 Cache Line, and the data pages are divided into a plurality of regions (Cache Line blocks). The structure of the database cache page after the area division is shown in fig. 1, which includes a header and a footer, wherein,

the header includes: checksum, lower, upper, line, wherein,

checksum: a data check value;

lower: a lower limit of free space within a page;

upper: an in-page free space upper limit;

and (4) link: an array of pointers recorded within a page.

The page tail comprises: prepage, nextpage, where,

and (4) preprage: the page number of the previous page;

nextpage: the page number of the next page.

In the embodiment of the invention, the page descriptor corresponding to the cache page setting is subjected to page description, including,

tablespace id: the table space ID where the table is located, namely the storage path of the table file;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: data page number on PM;

cache line bitmap (bitmap table): mark which cache line blocks in the cache page are dirty (regions are modified).

The descriptors are organized in an array form, and the index of the array is the page number of the data page in the memory.

In step 303, the record in the cache page is modified and the upper pointer and corresponding link array are adjusted.

In the embodiment of the invention, the upper pointer is adjusted to point to the last recording head, the recording offset is ptr, wherein ptr is the offset relative to the position of the header; the corresponding link array is adjusted to point to the record.

In step 304, the modified region is looked up and the page descriptor is modified.

In the embodiment of the invention, the cache line block in which the record is positioned is calculated, and the bit corresponding to the page descriptor bitmap table is modified to be 1:

dirty_cache_line=ptr%cache_line_block_size；

setting the bitmap positions corresponding to the first cache line block and the second dirty _ cache _ line +1 blocks as 1;

at step 305, a disk flush is performed based on the modified page descriptor.

In the embodiment of the invention, the flashing step is as follows: and calling a PMDK interface in a parallel mode by using a cache line block with bit 1 corresponding to the bitmap table according to the modified page descriptor, and brushing the disk in a user mode.

In the embodiment of the invention, a plurality of flash processes are started according to the number of CPU cores, and each cache line block is uniformly distributed to each flash process for flash.

In step 306, after the disk is flushed, the bitmap in the page descriptor is cleared to 0.

Example 2

Fig. 4 is a flowchart of inserting a recording modification data page and flushing a disk according to the present invention, and the flow of inserting a recording modification data page and flushing a disk of the present invention will be described in detail with reference to fig. 4. First, in step 401, a record is inserted into a free block of a cache page.

In the embodiment of the invention, scanning is carried out from the tail of the cache page to the head of the cache page, a free space is found in the free space position of the cache page, and a record is inserted.

At step 402, the upper pointer and corresponding link array are adjusted.

In the embodiment of the invention, an upper pointer is adjusted to point to the head of the last record, and the record offset is ptr; the corresponding lnp array is adjusted to point to the record.

In step 403, the position of the record insertion is calculated and the page descriptor is modified.

In the embodiment of the invention, which area (cache line block) the record is located in is calculated, and the bit corresponding to the page descriptor bitmap table is modified to be 1: dirty _ cache _ line = ptr% cache _ line _ block _ size;

and (3) setting the bitmap positions corresponding to the first cache line block and the second dirty _ cache _ line +1 block as 1.

At step 404, a dirty page flush is performed based on the modified page descriptor.

And performing persistent flashing on the PM memory by the area inserted with the record.

In the embodiment of the invention, all page descriptor arrays are traversed, and the cache line block with bit 1 corresponding to the bitmap table is called in a parallel mode to flush the disk by a PMDK interface in a user mode.

In the embodiment of the invention, the kernel mode of the operating system is skipped so as to avoid the extra consumption of switching the kernel mode and the user mode.

In the embodiment of the invention, the concurrent mode is that a plurality of flash processes are started according to the number of the CPU cores, and each cache line block is uniformly distributed to each flash process for flash.

At step 405, the persistent flush is complete and the page descriptor is cleared.

And after the disk brushing persistence is finished, clearing 0 the bitmap in the page descriptor. Example 3

Fig. 5 is a schematic diagram of the PM-based database page caching system according to the present invention, as shown in fig. 5, the PM-based database page caching system of the present invention includes a PM memory 501, a DRAM memory 502, a region dividing module 503, a page description module 504, and a disk flushing module 505, wherein,

and a PM memory 501 for storing the table file.

In the embodiment of the present invention, the PM memory 501 maps the table file to the DRAM memory 502 through the persistent memory development kit PMDK, receives the instruction of the disk refresh module 505, and updates the table file.

The DRAM memory 502 stores the database cache pages and page descriptors and flushes the modified area to the PM memory 501 according to the instructions of the disk flush module 505.

And an area division module 503, which performs area division on the database cache page.

In the embodiment of the invention, the division of the Cache pages of the database into areas comprises the step of dividing each Cache page into areas with fixed sizes, wherein the area size is set to be 1 Cache Line, and the Cache pages are divided into a plurality of areas (Cache Line blocks).

The page description module 504 sets a corresponding page descriptor for the cache page.

In the embodiment of the present invention, the page descriptor set by the page description module 504 includes,

tablespace id: the table space ID where the table is located, namely the storage path of the table file;

database id: the database ID where the table is located;

rel id: the ID of the table;

page id: data page number on PM;

cache line bitmap (bitmap table): marking which cache line blocks in the cache page are dirty.

The disk flush module 505 traverses the page descriptor array, controls the DRAM memory, flushes the modified area into the PM memory 501, and performs persistent disk flushing.

In the embodiment of the present invention, the disk flushing module 505 traverses the page descriptor array, and calls the cache line block with bit 1 corresponding to the bitmap table in a parallel manner to flush the disk in a user mode through the PMDK interface.

In the embodiment of the invention, the kernel mode of the operating system is skipped so as to avoid the extra consumption of switching the kernel mode and the user mode.

In the embodiment of the invention, the concurrent mode is that a plurality of flash processes are started according to the number of the CPU cores, and each cache line block is uniformly distributed to each flash process for flash.

Example 4

Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 6, in a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may also include a non-volatile Memory, such as at least 1 disk Memory; at least 1 PM, and the like. Of course, the electronic device may also include hardware required for other services.

The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.

And the memory is used for storing programs. In particular, the program may include program code comprising computer operating instructions.

The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program, and forms a shared resource access control device on a logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:

carrying out region division on a cache page;

setting corresponding page descriptors for the cache pages after the area division;

acquiring a modification area and modifying the page descriptor;

and flashing the modification area to the PM memory.

Example 5

Embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed, are capable of performing the method of the embodiments, and in particular for performing the above-described PM-based database page caching method.

Although the embodiments of the present invention have been described above, the above description is only for the convenience of understanding the present invention, and is not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.