阅读说明：本技术 保证原子性和顺序性的数据存储方法和装置 (Data storage method and device for ensuring atomicity and sequency ) 是由 舒继武 廖晓坚 王硕 于 2021-09-09 设计创作，主要内容包括：本申请提出了一种保证原子性和顺序性的数据存储方法,涉及计算机技术领域,其中,该方法包括：在数据存储过程中,通过主机以事务为单位顺序地发送传输命令至设备端的持久性存储区域,并通过提交门铃通知设备处理传输命令；在设备处理完请求时,通过主机以事务为单位顺序地完成请求并通知上层应用,从而保证原子性和顺序性。采用上述方案的本申请在保证原子性和顺序性的同时,降低了数据和命令传输延迟和所消耗的带宽,提升了数据存储的效率。(The application provides a data storage method for ensuring atomicity and sequency, and relates to the technical field of computers, wherein the method comprises the following steps: in the data storage process, sequentially sending a transmission command to a persistent storage area of an equipment end by a host in a transaction unit, and informing the equipment to process the transmission command by submitting a doorbell; when the device finishes processing the request, the host completes the request in order by taking transaction as a unit and informs the upper application, thereby ensuring atomicity and orderliness. By adopting the scheme, atomicity and sequence are guaranteed, data and command transmission delay and consumed bandwidth are reduced, and data storage efficiency is improved.)

1. A data storage method for guaranteeing atomicity and sequency is characterized by comprising the following steps:

in the data storage process, sequentially sending a transmission command to a persistent storage area of an equipment end by a host in a transaction unit, and informing the equipment end of processing the transmission command by submitting a doorbell;

when the device finishes processing the request, the host completes the request in order by taking transaction as a unit and informs the upper application, thereby ensuring atomicity and orderliness.

2. The method of claim 1, wherein the transaction comprises a set of requests that are all executed or that are not executed at all.

3. A method of atomicity and sequency guaranteed data storage as defined in claim 1, wherein the order in which the host completes the requests is the order in which the requests complete as seen by the user.

4. The method of claim 1, wherein the transfer command is a command issued by a computer to a storage device to store data.

5. The method of claim 1, wherein the persistent storage area is a section of storage space that is not lost after power is lost.

6. The method of guaranteed atomicity and sequency data storage according to claim 1, wherein the commit doorbell is an intermediary for a host to notify a device to process a transfer command, comprising a space of the persistent storage area.

7. The data storage method for ensuring atomicity and sequency of claim 1, wherein the notifying device processes the transmission command by submitting a doorbell, specifically:

and writing a specific numerical value into the submission doorbell area after the transmission command of the transaction is written into the persistent storage area by the host so as to inform the device of processing the request.

8. A method of guaranteed atomicity and sequency data storage as defined in claim 1, wherein the atomicity is a state of either full execution or no execution of a request in a transaction.

9. A data storage device for guaranteeing atomicity and sequency is characterized by comprising a host computer and a device end, wherein,

the host is used for sequentially sending transmission commands to a persistent storage area of the equipment end by taking transactions as a unit in the data storage process and informing the equipment end of processing the transmission commands by submitting a doorbell;

the equipment end is used for completing the request in the transmission command;

and the host is also used for sequentially completing the requests by taking transactions as a unit and informing an upper application when the equipment finishes processing the requests, so that the atomicity and the orderliness are ensured.

10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the atomicity and sequency guaranteed data storage method of any one of claims 1 to 8.

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data storage method and apparatus for ensuring atomicity and sequency.

Background

Atomicity of a data store refers to a group of requests either all executed successfully or all executed failed. The sequentiality of a data store describes the particular order in which data falls on a persistent medium. Atomicity and order play a crucial role in storage systems. For example, both logs in a file system and transactions in a database rely on atomicity and a particular access sequence to ensure correctness and consistency of the storage system.

Since existing storage protocols/methods do not guarantee atomicity and sequentiality, the storage system must rely on a logging mechanism to store data sequentially atomically to guarantee correctness and consistency. The existing advanced and widely used Non-Volatile Memory host controller interface specification (NVMe) does not guarantee atomicity and order. The log mechanism is built on NVMe, and high overhead is required for ensuring atomicity and orderliness. For example, to store N data blocks atomically, existing logging mechanisms require at least the host to send 2N +4 MMIO instructions, N +2 block I/O instructions to the device; the device needs at least 2N +4 DMA commands to pull the transfer command from the host and sends N +2 interrupt commands to notify the host of the completion of the data block. These commands consume significant bandwidth traffic of the physical link. In addition, the host-side logging mechanism needs to wait for the completion of these commands synchronously, adding latency to data storage.

Disclosure of Invention

The present application is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, a first objective of the present application is to provide a data storage method for ensuring atomicity and sequency, which solves the technical problems of the existing method that bandwidth traffic of a physical link is consumed and data storage is delayed, reduces data and command transmission delay and consumed bandwidth while ensuring atomicity and sequency, and improves data storage efficiency.

The method and the device send the transmission command by taking the transaction as a unit, so that the CPU has more chances to combine a plurality of transmission commands into one large request, the problem that the single transmission command needs to align the write amplification of the CPU and the minimum transmission unit of the physical link is solved, and meanwhile, the device is informed by taking the transaction as a unit, so that the doorbell writing times of the host are reduced, and the flow consumption is reduced. The method and the device send the transmission commands to the persistent storage area of the device side in sequence, so that the host can track the atomicity and the sequency state of the transaction even after recovery of crash (such as sudden power failure), the atomicity and the sequency are guaranteed, the atomicity and the sequency are separated from the data persistent storage, the host does not need to synchronously wait for the completion of the block I/O, DMA and an interrupt instruction, and data storage delay is reduced. In addition, compared with a log mechanism, the method and the device do not need additional log submitting blocks, so that the flow consumption is reduced, and the data storage efficiency is improved.

A second object of the present application is to propose a data storage device that guarantees atomicity and order.

A third object of the present application is to propose a non-transitory computer-readable storage medium.

In order to achieve the above object, an embodiment of a first aspect of the present application provides a data storage method for guaranteeing atomicity and order, including: in the data storage process, sequentially sending a transmission command to a persistent storage area of an equipment end by a host in a transaction unit, and informing the equipment end of processing the transmission command by submitting a doorbell; when the device finishes processing the request, the host completes the request in order by taking transaction as a unit and informs the upper application, thereby ensuring atomicity and orderliness.

Optionally, in one embodiment of the present application, a transaction includes a set of requests that are all executed, or, alternatively, all not executed.

Optionally, in an embodiment of the present application, the order in which the host completes the requests is the order in which the requests complete as seen by the user.

Optionally, in an embodiment of the present application, the transmission command is a command issued by the computer to the storage device to store data.

Optionally, in an embodiment of the present application, the persistent storage area is a storage space in which data is not lost after power failure.

Optionally, in one embodiment of the application, the submission doorbell provides an intermediary for the host to notify the device to process the transfer command, including a space of persistent storage area.

Optionally, in an embodiment of the present application, the processing of the transmission command by submitting the doorbell notification device is specifically:

and writing a specific numerical value into the submission doorbell area after the transmission command of the transaction is written into the persistent storage area by the host so as to inform the device of processing the request.

Optionally, in one embodiment of the present application, atomicity is the state of a request or a full execution or non-execution in a transaction.

In order to achieve the above object, a second aspect of the present application provides a data storage device for ensuring atomicity and order, which includes a host and a device, wherein,

the host is used for sequentially sending transmission commands to the persistent storage area of the equipment end by taking transactions as a unit in the data storage process and informing the equipment end of processing the transmission commands by submitting a doorbell;

the equipment end is used for completing the request in the transmission command;

and the host is also used for sequentially completing the requests in a transaction unit and informing the upper application when the device finishes processing the requests, so that the atomicity and the orderliness are ensured.

To achieve the above object, a non-transitory computer readable storage medium is provided in a third aspect of the present application, and when executed by a processor, the instructions in the storage medium can perform a data storage method that guarantees atomicity and sequency.

The data storage method for ensuring the atomicity and the sequency, the data storage device for ensuring the atomicity and the sequency and the non-temporary computer readable storage medium solve the technical problems that the bandwidth flow of a physical link is consumed and the data storage is delayed in the existing method, reduce the data and command transmission delay and the consumed bandwidth while ensuring the atomicity and the sequency, and improve the data storage efficiency.

The method and the device send the transmission command by taking the transaction as a unit, so that the CPU has more chances to combine a plurality of transmission commands into one large request, the problem that the single transmission command needs to align the write amplification of the CPU and the minimum transmission unit of the physical link is solved, and meanwhile, the device is informed by taking the transaction as a unit, so that the doorbell writing times of the host are reduced, and the flow consumption is reduced. The method and the device send the transmission commands to the persistent storage area of the device side in sequence, so that the host can track the atomicity and the sequency state of the transaction even after recovery of crash (such as sudden power failure), the atomicity and the sequency are guaranteed, the atomicity and the sequency are separated from the data persistent storage, the host does not need to synchronously wait for the completion of the block I/O, DMA and an interrupt instruction, and data storage delay is reduced. In addition, compared with a log mechanism, the method and the device do not need additional log submitting blocks, so that the flow consumption is reduced, and the data storage efficiency is improved.

Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

fig. 1 is a flowchart of a data storage method for ensuring atomicity and order according to an embodiment of the present disclosure;

FIG. 2a is a schematic diagram of an original data storage method according to an embodiment of the present application;

FIG. 2b is a schematic diagram of a data storage method for ensuring atomicity and order according to an embodiment of the present disclosure;

FIG. 3 is a diagram illustrating atomicity and order assurance during crash recovery for a computer system according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a data storage device for ensuring atomicity and order according to a second embodiment of the present application.

Detailed Description

Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.

The atomicity and order guaranteed data storage method and apparatus of the embodiments of the present application are described below with reference to the accompanying drawings.

Fig. 1 is a flowchart of a data storage method for ensuring atomicity and sequency according to an embodiment of the present disclosure.

As shown in fig. 1, the data storage method for guaranteeing atomicity and sequency includes the following steps:

step 101, in the data storage process, sequentially sending a transmission command to a persistent storage area of an equipment end by a host in a transaction unit, and informing the equipment end of processing the transmission command by submitting a doorbell;

and 102, when the device finishes processing the request, the host finishes the request in sequence by taking transactions as a unit and notifies an upper application, so that the atomicity and the orderliness are ensured.

According to the data storage method for ensuring atomicity and sequency, in the data storage process, transmission commands are sequentially sent to a persistent storage area of an equipment end by a host in a transaction unit, and the equipment end is informed to process the transmission commands by submitting a doorbell; when the device finishes processing the request, the host completes the request in order by taking transaction as a unit and informs the upper application, thereby ensuring atomicity and orderliness. Therefore, the technical problems that the bandwidth flow of a physical link is consumed and the data storage is delayed in the existing method can be solved, the atomicity and the sequence are guaranteed, meanwhile, the data and command transmission delay and the consumed bandwidth are reduced, and the data storage efficiency is improved.

The method and the device send the transmission command by taking the transaction as a unit, so that the CPU has more chances to combine a plurality of transmission commands into one large request, the problem that the single transmission command needs to align the write amplification of the CPU and the minimum transmission unit of the physical link is solved, and meanwhile, the device is informed by taking the transaction as a unit, so that the doorbell writing times of the host are reduced, and the flow consumption is reduced. The method and the device send the transmission commands to the persistent storage area of the device side in sequence, so that the host can track the atomicity and the sequency state of the transaction even after recovery of crash (such as sudden power failure), the atomicity and the sequency are guaranteed, the atomicity and the sequency are separated from the data persistent storage, the host does not need to synchronously wait for the completion of the block I/O, DMA and an interrupt instruction, and data storage delay is reduced. In addition, compared with a log mechanism, the method and the device do not need additional log submitting blocks, so that the flow consumption is reduced, and the data storage efficiency is improved.

Further, in the embodiments of the present application, a transaction includes a set of requests that are all executed, or, alternatively, all not executed.

Further, in the embodiment of the present application, the order in which the host completes the requests is the order in which the requests complete as seen by the user.

Further, in the embodiment of the present application, the transmission command is a command issued by the computer to the storage device to store data.

Further, in the embodiment of the present application, the persistent storage area is a storage space in which data is not lost after power failure.

Further, in an embodiment of the present application, the submission doorbell provides an intermediary for the host to notify the device to process the transfer command, including a space of the persistent storage area.

Further, in this embodiment of the application, the process of the transmission command by submitting the doorbell notification device specifically includes:

and writing a specific numerical value into the submission doorbell area after the transmission command of the transaction is written into the persistent storage area by the host so as to inform the device of processing the request.

Further, in the embodiments of the present application, atomicity is a state of either all or none execution in a transaction.

Fig. 2a is a schematic diagram of an original data storage method according to an embodiment of the present application.

As shown in fig. 2a, in the original data storage method, the host stores data to the device mainly in 7 steps, and the host puts the transmission command into a ring queue at the host; writing the tail pointer of the queue into a submission doorbell at the equipment end; after observing the change of the doorbell, the equipment pulls and submits the transmission command in the queue from the host through the DMA command and processes the transmission command; the device pulls the data block from the host through the block I/O command and stores the data block; the device places a completion command into a completion queue through DMA; and informing the host computer to process through the interrupt request; and after the host processes the completion queue, setting a completion command through doorbell operation and informing a user of the completion state of the data. For a transaction containing N requests, in order to ensure atomicity, the host needs to go through at least more than N +2 processes, wherein 2 is a descriptor block and a commit block in a log mechanism; to ensure sequentiality, the host needs to execute synchronously and wait for more than N +2 processes.

Fig. 2b is a schematic diagram of a data storage method for ensuring atomicity and order according to an embodiment of the present application.

As shown in fig. 2b, in the data storage method for ensuring atomicity and sequency, the host stores data to the device and ensures atomicity and sequency mainly includes 6 steps, first, a commit queue, a commit doorbell, and a commit queue head pointer are all placed in a persistent storage area of the device, and a transmission command of all requests in a transaction is sent to a commit queue of the device in batch, with the transaction as a unit; only one commit doorbell operation needs to be sent for the entire transaction; the device pulls the data block from the host through the block I/O command and stores the data block; the device places a completion command into a completion queue through DMA; and informing the host computer to process through the interrupt request; after the device processes the transaction request, for all requests in the transaction, the host writes the value of a commit queue head pointer to the persistent storage area of the device only, and sets the completion doorbell only once.

The method and the device set the commit doorbell and the complete doorbell in a transaction unit, so that all requests belonging to the same transaction can be received and completed (namely atomicity) by the device at the same time when the system operates normally, and meanwhile, the transactions are put into the commit queue, the commit doorbell and the complete doorbell are put in sequence, so that the sequency of transaction completion when the system operates normally can be guaranteed.

In the application, for a transaction containing N requests, in order to ensure atomicity and sequency, the host only needs MMIO (doorbell is also MMIO) for 4 times, and the device only needs DMA (direct memory access) instructions, block I/O instructions and interrupt instructions for N +1 times, so that the number of required instructions is greatly reduced. Meanwhile, the atomicity and the sequency state of the transaction can be inferred through the persistent submission queue, the submission doorbell and the submission head pointer, and the delay of atomicity and sequency guarantee is greatly reduced.

FIG. 3 is a diagram illustrating atomicity and order assurance of a computer system during crash recovery according to an embodiment of the present disclosure.

FIG. 3 illustrates the completion status of requests and transactions when a computer system recovers from a crash (e.g., a sudden power loss). The requests located before the commit queue head pointer are both atomicity and order guaranteed so no additional processing is required. Requests located between the commit queue head pointer and the commit doorbell may be non-atomic or out of order and need to be processed again. For request 2 and request 3, which belong to the same transaction 2, since there is a request 2 outstanding in transaction 2, the commit queue head pointer is not moved to request 4 before the system crashes, so the atomicity of transaction 2 is not satisfied, and the data requested in transaction 2 is erased. Transaction 3 contains only one request 4 and request 4 has completed (i.e., atomicity has been guaranteed), however, since transaction 2 before transaction 3 is not guaranteed atomicity, the data requested by transaction 3 should also be erased to ensure sequentiality. By combining the above, the requests between the submit queue head pointer and the submit doorbell are all erased to ensure sequentiality and atomicity.

Fig. 4 is a schematic structural diagram of a data storage device for ensuring atomicity and order according to a second embodiment of the present application.

As shown in fig. 4, the atomicity and order guaranteed data storage apparatus includes a host 10 and a device 20, wherein,

the host 10 is used for sequentially sending transmission commands to a persistent storage area of the equipment end by taking transactions as units in the data storage process and informing the equipment end of processing the transmission commands by submitting a doorbell;

the device end 20 is used for completing the request in the transmission command;

the host 10 is further configured to sequentially complete requests in units of transactions and notify upper-layer applications when the device has finished processing the requests, so as to ensure atomicity and sequency.

The data storage device for ensuring atomicity and sequency comprises a host and an equipment end, wherein the host is used for sequentially sending transmission commands to a persistent storage area of the equipment end by taking transactions as a unit in the data storage process and informing the equipment end of processing the transmission commands by submitting a doorbell; the equipment end is used for completing the request in the transmission command; and the host is also used for sequentially completing the requests in a transaction unit and informing the upper application when the device finishes processing the requests, so that the atomicity and the orderliness are ensured. Therefore, the technical problems that the bandwidth flow of a physical link is consumed and the data storage is delayed in the existing method can be solved, the atomicity and the sequence are guaranteed, meanwhile, the data and command transmission delay and the consumed bandwidth are reduced, and the data storage efficiency is improved.

The method and the device send the transmission command by taking the transaction as a unit, so that the CPU has more chances to combine a plurality of transmission commands into one large request, the problem that the single transmission command needs to align the write amplification of the CPU and the minimum transmission unit of the physical link is solved, and meanwhile, the device is informed by taking the transaction as a unit, so that the doorbell writing times of the host are reduced, and the flow consumption is reduced. The method and the device send the transmission commands to the persistent storage area of the device side in sequence, so that the host can track the atomicity and the sequency state of the transaction even after recovery of crash (such as sudden power failure), the atomicity and the sequency are guaranteed, the atomicity and the sequency are separated from the data persistent storage, the host does not need to synchronously wait for the completion of the block I/O, DMA and an interrupt instruction, and data storage delay is reduced. In addition, compared with a log mechanism, the method and the device do not need additional log submitting blocks, so that the flow consumption is reduced, and the data storage efficiency is improved.

In order to implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium on which a computer program is stored, the computer program implementing the atomicity and order guaranteed data storage method of the above embodiments when executed by a processor.

In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.