阅读说明：本技术 一种写入batch group串行化逻辑的优化方法 (Optimization method for writing in batch group serialization logic ) 是由 刘梦真 于 2021-07-26 设计创作，主要内容包括：本发明涉及数据库领域,具体提供了一种写入batch group串行化逻辑的优化方法,每个batch使用独立的线程,基于滑动窗口机制来实现WAL的并发写入,多次并发访问小IO,维护一个WAL目录,每个batch对应生成一个单独的小log文件。与现有技术相比,本发明优化了RocksDB中WAL写入时batch group串行化逻辑导致的batch写入性能瓶颈问题,不再使用RocksDB内部组提交机制,而是基于滑动窗口机制支持实现batch并发写入WAL,可以通过设置窗口大小来控制并发程度,将一次大IO转换为并发访问小IO。充分利用了SSD的高读写速度和具有并发特性的性能优势,很大程度上提升了存储引擎的写性能。(The invention relates to the field of databases, and particularly provides an optimization method for writing in batch group serialization logic. Compared with the prior art, the method optimizes the bottleneck problem of batch write performance caused by batch group serialization logic during WAL write in the RocksDB, does not use an internal group submission mechanism of the RocksDB any more, supports realization of concurrent write of batch in the WAL based on a sliding window mechanism, can control the concurrency degree by setting the window size, and converts one-time large IO into concurrent access small IO. The high read-write speed and the performance advantage with the concurrency characteristic of the SSD are fully utilized, and the write performance of the storage engine is improved to a great extent.)

1. The optimization method for writing in batch group serialization logic is characterized in that each batch uses an independent thread, WAL concurrent writing is realized based on a sliding window mechanism, small IO (input/output) is accessed for multiple times concurrently, a WAL directory is maintained, and each batch correspondingly generates an independent small log file.

2. The method of claim 1, wherein the method comprises the following steps:

s1, taking the WriteBatch object as a carrier;

s2, generating a WriteThread example for each Write thread;

s3, maintaining a window, wherein the Writer objects in the linked list can be divided into 4 types;

s4, the process of setting sequence number for each Writer in the linked list is serial;

s5, when the write thread in the window starts to write WAL concurrently, after the log is generated, whether all the previous batch has been written is needed to be judged;

s6, starting to write MemTable after finishing the WAL writing;

s7, after the MemTable writing is completed, the last _ sequence _ value in VersionSet needs to be updated;

s8, the Writer object exits after completing the MemTable write.

3. The method of claim 2, wherein in step S1, consecutive write requests are flushed in the database system, each write request comprising one or more operations, and a record of the one or more operations is encapsulated in WriteBatch, and each write to RocksDB is carried by a WriteBatch object.

4. The method of claim 3, wherein in step S2, each Write thread generates a WriteThread instance, Write associated with a corresponding WriteBatch object, concatenates multiple Writer objects with a linked list, and thus handles concurrency.

5. The method of claim 4, wherein in step S3, a window is maintained, and the Writer objects in the linked list can be classified into 4 types:

(1) starting to write the transaction log and having received the confirmation;

(2) starting to write the transaction log, but not receiving the confirmation;

(3) transaction log has not been written, but write is allowed;

(4) no transaction log has been written, and no write is allowed;

wherein the type (2) and the type (3) are both located within the window range;

the window size represents the maximum number of threads allowed to concurrently write to the WAL, and the window moves its left boundary after receiving the acknowledgment sequence number.

6. The method of claim 5, wherein in step S4, each Writer sets a sequence number, the sequence number of the first Writer entering the queue is equal to last _ allocated _ sequence _ value in the current VersionSet, and the sequence number of the following Writer is equal to the sequence number of the previous Writer plus the number of records in the batch of the previous Writer;

the process of setting the sequence number for each Writer in the linked list is serial.

7. The method of claim 6, wherein in step S5, when a write thread in a window starts writing WAL concurrently, after generating the log, it needs to be determined whether all previous lots have completed writing:

when there is a batch with a previous sequence number which has not completed writing but has completed writing, no confirmation sequence number is returned, and no confirmation is performed on subsequent writing; at this time, the current log file is not flushed to the disk, but cached, and only after the batch with the current serial number is completely written, the current log file and the log file accumulated in the cache are flushed together, the sequence number is updated to be equal to the original sequence number plus the record number of the batch, and then the new sequence number is returned.

8. The method of claim 7, wherein if all previous batchs have been written to completion, then the log file generated by the current batch is flushed to disk directly and a new sequence number is returned, which is the original sequence number plus the number of records for the batch.

9. The method of claim 8, wherein in step S6, after the window receives the returned sequence number confirmation, the window updates the current last _ allocated _ sequence _ to the returned sequence number, and slides the window to the position of the sequence number. The confirmed Writer object completes WAL writing and starts writing MemTable.

10. The method of claim 9, wherein in step S7, after completing the MemTable write, the last _ sequence _ value in VersionSet needs to be updated:

as with last _ allocated _ sequence _ the last _ sequence _ also needs to ensure its sequentiality, so when writing MemTable concurrently, a sliding window needs to be maintained, the window size represents the maximum number of threads allowed to write MemTable concurrently;

the same sliding window mechanism is maintained when WAL is written concurrently, the window updates last _ sequence _ value and moves the left boundary of the window only after receiving the confirmation sequence number, and when there is no write completed by the batch with the previous sequence number but the write completed by the batch with the next sequence number, the confirmation sequence number is not returned, and the subsequent write is not confirmed.

Technical Field

The invention relates to the field of databases, and particularly provides an optimization method for writing in batch group serialization logic.

Background

Rocksbb is a persistent, embedded KV storage engine written in C + +, consisting primarily of MemTable, Immutable MemTable, SSTable, and WAL. When inserting KV data, firstly, the data needs to be written into the WAL file, and then the WAL file is refreshed to a disk so as to ensure the durability and the safety of the data.

If each commit of a transaction requires a sync call to flush the log to disk, the transaction commit will significantly limit the write performance of the database. It is contemplated that the commitment of several transactions may be combined into one sync to do so, thereby improving the TPS of the database system. So inside the RocksDB, the transaction log is written in batch using the batch-commit mechanism, so that multiple transactions can be committed simultaneously using one sync call. However, since the write of the WAL needs to ensure the sequentiality between different batch groups, the write can be performed only by a single thread. This batch group serialization logic causes the WAL write to become the bottleneck point for the batch write performance of RocksDB.

Disclosure of Invention

The present invention provides an optimization method for writing in batch group serialization logic, which is highly practical for overcoming the defects of the prior art.

The technical scheme adopted by the invention for solving the technical problems is as follows:

the optimization method for writing in batch group serialization logic is characterized in that each batch uses an independent thread, WAL concurrent writing is realized based on a sliding window mechanism, small IO is accessed for multiple times concurrently, a WAL directory is maintained, and each batch correspondingly generates an independent small log file.

Further, the method comprises the following specific steps:

s1, taking the WriteBatch object as a carrier;

s2, generating a WriteThread example for each Write thread;

s3, maintaining a window, wherein the Writer objects in the linked list can be divided into 4 types;

s4, the process of setting sequence number for each Writer in the linked list is serial;

s5, when the write thread in the window starts to write WAL concurrently, after the log is generated, whether all the previous batch has been written is needed to be judged;

s6, starting to write MemTable after finishing the WAL writing;

s7, after the MemTable writing is completed, the last _ sequence _ value in VersionSet needs to be updated;

s8, the Writer object exits after completing the MemTable write.

Further, in step S1, in the database system, successive write requests are flushed, each write request including one or more operations, a record of the one or more operations is packed into WriteBatch, and each write to rocksbb is carried by a WriteBatch object.

Further, in step S2, each Write thread generates a WriteThread instance, associated with a corresponding WriteBatch object, and concatenates multiple Writer objects with a linked list, thereby handling concurrency.

Further, in step S3, a window is maintained, and the Writer objects in the linked list can be classified into 4 types:

(1) starting to write the transaction log and having received the confirmation;

(2) starting to write the transaction log, but not receiving the confirmation;

(3) transaction log has not been written, but write is allowed;

(4) no transaction log has been written, and no write is allowed;

wherein the type (2) and the type (3) are both located within the window range;

the window size represents the maximum number of threads allowed to concurrently write to the WAL, and the window moves its left boundary after receiving the acknowledgment sequence number.

Further, in step S4, each Writer sets a sequence number, the sequence number of the Writer entering the queue first is equal to last _ allocated _ sequence _ value in the current VersionSet, and the sequence number of the following Writer is equal to the sequence number of the previous Writer plus the number of records in the batch of the previous Writer;

the process of setting the sequence number for each Writer in the linked list is serial.

Further, in step S5, when the write thread in the window starts writing WAL concurrently, it is necessary to determine whether all the previous batch has completed writing after log generation:

when there is a batch with a previous sequence number which has not completed writing but has completed writing, no confirmation sequence number is returned, and no confirmation is performed on subsequent writing; at this time, the current log file is not flushed to the disk, but cached, and only after the batch with the current serial number is completely written, the current log file and the log file accumulated in the cache are flushed together, the sequence number is updated to be equal to the original sequence number plus the record number of the batch, and then the new sequence number is returned.

Further, if the previous batch has all written, the log file generated by the current batch is directly flushed to the disk and a new sequence number is returned, i.e. the original sequence number plus the number of records of the batch.

Further, in step S6, after the window receives the returned sequence number confirmation, the current last _ allocated _ sequence _ is updated to the returned sequence number, and the window is slid to the position of the sequence number. The confirmed Writer object completes WAL writing and starts writing MemTable.

Further, in step S7, after the MemTable writing is completed, the value of last _ sequence _ in VersionSet needs to be updated:

as with last _ allocated _ sequence _ the last _ sequence _ also needs to ensure its sequentiality, so when writing MemTable concurrently, a sliding window needs to be maintained, the window size represents the maximum number of threads allowed to write MemTable concurrently;

the same sliding window mechanism is maintained when WAL is written concurrently, the window updates last _ sequence _ value and moves the left boundary of the window only after receiving the confirmation sequence number, and when there is no write completed by the batch with the previous sequence number but the write completed by the batch with the next sequence number, the confirmation sequence number is not returned, and the subsequent write is not confirmed.

Compared with the prior art, the optimization method for writing the batch group serialization logic has the following outstanding advantages:

the invention optimizes the bottleneck problem of batch write performance caused by batch group serialization logic during WAL write in the RocksDB, does not use the internal group submission mechanism of the RocksDB any more, supports the realization of batch concurrent write WAL based on a sliding window mechanism, can control the concurrency degree by setting the window size, and converts one-time large IO into concurrent access small IO. The high read-write speed and the performance advantage with the concurrency characteristic of the SSD are fully utilized, and the write performance of the storage engine is improved to a great extent.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of the batch-commit mechanism within the RocksDB in the present invention;

FIG. 2 is a flow chart of the present invention for implementing WAL concurrent writing based on a sliding window mechanism;

fig. 3 is a schematic diagram of the window moving forward after receiving the confirmation sequence number in the present invention.

Detailed Description

The present invention will be described in further detail with reference to specific embodiments in order to better understand the technical solutions of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

A preferred embodiment is given below:

as shown in fig. 1 to 3, in the optimization method for writing a batch group serialization logic in this embodiment, the method abandons the use of a batch-commit mechanism inside the RocksDB, and does not write the transaction log in a batch manner any more, but each batch uses an independent thread, and implements concurrent writing of the WAL based on a sliding window mechanism, so as to replace a large IO with multiple concurrent accesses of a small IO. At the same time, the write transaction log no longer uses the append mode. Instead, a WAL directory is maintained, with each batch generating a separate small log file.

The specific implementation process is as follows:

s1, taking the WriteBatch object as a carrier:

in a database system, successive write requests may be flooded. Each write request may contain one or more operations, encapsulating a record of the one or more operations to WriteBatch, with each write to RocksDB carried by a WriteBatch object.

S2, each Write thread generates a WriteThread that is an instance of Write:

each Write thread generates a WriteThread, a Write instance, associated with a corresponding WriteBatch object. Multiple Writer objects are chained together with a linked list to handle concurrency.

S3, maintaining a window, wherein the Writer objects in the linked list can be divided into 4 types:

maintaining a window, the Writer objects in the linked list can be divided into 4 types, (1) starting to write the transaction log and receiving confirmation; (2) begin writing to the transaction log, but not receiving an acknowledgement; (3) the transaction log has not been written, but the write is allowed; (4) the transaction log has not been written and no writes are allowed. Wherein both types (2) and (3) are located within the window. The window size represents the maximum number of threads that are allowed to concurrently write to the WAL. The window moves the left boundary of the window after receiving the acknowledgement sequence number.

S4, the process of setting sequence number by each Writer in the linked list is serial:

each Writer sets a sequence number, the sequence number of the first entering queue Writer is equal to the last _ allocated _ sequence _ value in the current VersionSet, and the sequence number of the subsequent Writer is equal to the sequence number of the previous Writer plus the number of records in the previous Writer's batch. The process of setting the sequence number for each Writer in the linked list is serial.

S5, when the write thread in the window starts to write WAL concurrently, after the log is generated, whether all the previous batch has been written is judged to be completed:

when there is no write completed by the previous batch but the write completed by the subsequent batch, the confirmation sequence is not returned and the subsequent write is not confirmed. At this time, the current log file is not flushed to the disk, but is cached.

Only after the batch with the previous sequence number is completely written, the disk refreshing operation is carried out together with the log file accumulated in the cache, the sequence number is updated and is equal to the original sequence number plus the record number of the batch, and then the new sequence number is returned.

If the previous batch has all written, the log file generated by the current batch is directly flushed to disk and a new sequence number is returned, i.e., the original sequence number plus the number of records for the batch.

S6, after finishing WAL writing, starting to write MemTable:

and after receiving the returned sequence number confirmation, the window updates the current last _ allocated _ sequence _ to be the returned sequence number, and slides the window to the position of the sequence number. After confirmation (i.e., sliding out of the window), the Writer object completes the WAL write and starts to write MemTable.

S7, after the MemTable writing is completed, the last _ sequence _ value in VersionSet needs to be updated:

like last _ allocated _ sequence _ the last _ sequence _ also needs to guarantee its sequentiality. Therefore, when concurrent writing to MemTable, a sliding window needs to be maintained, and the window size represents the maximum number of threads allowed to concurrently write to MemTable. The same sliding window mechanism is maintained when concurrent write WAL, and the window will update the value of last _ sequence _ and move the left boundary of the window only after receiving the confirmation sequence number. When there is no write completed by the previous batch but the write completed by the subsequent batch, the confirmation sequence is not returned and the subsequent write is not confirmed.

S8, the Writer object exits after completing the MemTable write.

The above embodiments are only specific cases of the present invention, and the protection scope of the present invention includes but is not limited to the above embodiments, and any suitable changes or substitutions that are consistent with the claims of the monitoring alarm method of the present invention and are made by those skilled in the art should fall within the protection scope of the present invention.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.