Storage system write operation processing method, device, equipment and storage medium
阅读说明：本技术 一种存储系统写操作处理方法、装置、设备及存储介质 (Storage system write operation processing method, device, equipment and storage medium ) 是由 张露露 孟祥瑞 于 2019-09-20 设计创作，主要内容包括：本发明公开了一种存储系统写操作处理方法,包括：对硬盘的日志文件进行检测,判断日志文件中是否存在硬盘的错误信息；若日志文件中存在错误信息,则利用日志收集工具收集目标日志文件；保存目标日志文件；该方法由于无需对smart信息进行收集并分析,而是直接利用日志文件判断是否发生故障,因此可以在硬盘发生故障后及时确认发生故障并收集日志,故此可以保证日志收集的及时性,即收集到故障发生后距离故障发生时间最近的日志,为后续的硬盘故障分析提供最有价值的信息,解决了现有技术无法为硬盘故障分析提供最有价值的信息的问题；此外,本发明还提供了一种存储系统写操作处理装置、设备及计算机可读存储介质,同样具有上述有益效果。(The invention discloses a storage system write operation processing method, which comprises the following steps: detecting a log file of the hard disk, and judging whether error information of the hard disk exists in the log file; if the log file has error information, collecting a target log file by using a log collection tool; saving a target log file; according to the method, because smart information does not need to be collected and analyzed, whether a fault occurs or not is judged directly by using the log file, the fault can be confirmed in time and the log can be collected after the hard disk fails, so that the timeliness of log collection can be ensured, namely, the log closest to the fault occurrence time after the fault occurs is collected, the most valuable information is provided for subsequent hard disk fault analysis, and the problem that the most valuable information cannot be provided for hard disk fault analysis in the prior art is solved; in addition, the invention also provides a storage system write operation processing device, equipment and a computer readable storage medium, which also have the beneficial effects.)
1. A storage system write operation processing method is characterized by comprising the following steps:
acquiring write operations aiming at a target object in a cache layer, and judging whether the number of the write operations is greater than a number threshold value;
if the number of the write operations is larger than a number threshold, directly sending all the write operations to a back-end storage;
performing the write operation in the back-end storage.
2. The storage system write operation processing method according to claim 1, wherein executing the write operation in the backend storage includes:
combining all the write operations in the main OSD stored at the back end to obtain an integral write operation;
and executing the overall writing operation.
3. The method for processing write operations of a storage system according to claim 2, wherein merging all the write operations in the main OSD stored in the back end to obtain an overall write operation comprises:
and merging the write operations aiming at the target object into one integral write operation according to a merging rule in a memory of the main OSD stored at the back end.
4. The storage system write operation processing method according to claim 3, wherein the obtaining of the write operation for the target object includes:
acquiring a write request aiming at the target object;
and analyzing the write request to obtain the write operation.
5. A storage system write operation processing apparatus, comprising:
the judging module is used for acquiring the write operation aiming at the target object in the cache layer and judging whether the number of the write operation is greater than a number threshold value or not;
the sending module is used for directly sending all the write operations to the back-end storage if the number of the write operations is larger than a number threshold;
and the execution module is used for executing the write operation in the back-end storage.
6. The storage system write operation processing apparatus according to claim 5, wherein the execution module includes:
a write operation merging unit, configured to merge all write operations in the main OSD stored in the back end to obtain an overall write operation;
and the write operation execution unit is used for executing the whole write operation.
7. The storage system write operation processing module of claim 6, wherein the write operation merge unit comprises:
a rule merging subunit, configured to merge the write operations for the target object into one integral write operation according to a merging rule in a memory of the main OSD stored in the backend.
8. The storage system write operation processing module of claim 7, wherein the determining module comprises:
a write request acquisition unit configured to acquire a write request for the target object;
and the write request analyzing unit is used for analyzing the write request to obtain the write operation.
9. A storage system write operation processing apparatus comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is configured to execute the computer program to implement the storage system write operation processing method according to any one of claims 1 to 4.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the storage system write operation processing method according to any one of claims 1 to 4.
Technical Field
The present invention relates to the field of write operation processing of storage systems, and in particular, to a write operation processing method, a write operation processing apparatus, a write operation processing device, and a computer-readable storage medium for a storage system.
Background
In the existing distributed Storage cluster, a master-slave mode is adopted for read-write operation, and when a client wants to read or write data, only a read request or a write request is sent to a master OSD (Object-based Storage Device).
When the client side needs to write data, the write request is sent to the cache layer, the cache layer sends the request to the main OSD stored at the rear end in batch, and the main OSD analyzes the write request to obtain write operation and then sends the write operation to the auxiliary OSD. The slave OSD executes the write operation, and after the write operation is completed, the slave OSD informs the master OSD so that the master OSD can inform the client of the completion of the write operation, and the consistency of data is ensured. However, when the write request includes an excessive number of write instructions for the same object, the main OSD needs a long time to resolve the write instructions one by one, and thus the thread time-out suicide is caused, and the write operation cannot be executed. Particularly, when the number of write operations for the same object exceeds 65535, the main OSD may parse 65535 write requests in one parsing process, which may cause a write request parsing error and also may not perform the write operations.
Therefore, how to solve the problem that the existing storage system is prone to failure when executing too many write operations to the same object is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a storage system write operation processing method, a storage system write operation processing apparatus, a storage system write operation processing device, and a computer-readable storage medium, which solve the problem that an existing storage system is prone to failure when performing too many write operations for a same object.
In order to solve the above technical problem, the present invention provides a method for processing write operation of a storage system, including:
acquiring write operations aiming at a target object in a cache layer, and judging whether the number of the write operations is greater than a number threshold value;
if the number of the write operations is larger than a number threshold, directly sending all the write operations to a back-end storage;
performing the write operation in the back-end storage.
Optionally, executing the write operation in the backend storage includes:
combining all the write operations in the main OSD stored at the back end to obtain an integral write operation;
and executing the overall writing operation.
Optionally, merging all the write operations in the main OSD stored in the back end to obtain an overall write operation, including:
and merging the write operations aiming at the target object into one integral write operation according to a merging rule in a memory of the main OSD stored at the back end.
Optionally, the obtaining the write operation for the target object includes:
acquiring a write request aiming at the target object;
and analyzing the write request to obtain the write operation.
The invention also provides a storage system write operation processing device, which comprises:
the judging module is used for acquiring the write operation aiming at the target object in the cache layer and judging whether the number of the write operation is greater than a number threshold value or not;
the sending module is used for directly sending all the write operations to the back-end storage if the number of the write operations is larger than a number threshold;
and the execution module is used for executing the write operation in the back-end storage.
Optionally, the execution module includes:
a write operation merging unit, configured to merge all write operations in the main OSD stored in the back end to obtain an overall write operation;
and the write operation execution unit is used for executing the whole write operation.
Optionally, the write operation merging unit includes:
a rule merging subunit, configured to merge the write operations for the target object into one integral write operation according to a merging rule in a memory of the main OSD stored in the backend.
Optionally, the determining module includes:
a write request acquisition unit configured to acquire a write request for the target object;
and the write request analyzing unit is used for analyzing the write request to obtain the write operation.
The invention also provides a storage system write operation processing device, which comprises a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is used for executing the computer program to realize the storage system write operation processing method.
The present invention also provides a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the storage system write operation processing method described above.
The write operation processing method of the storage system obtains the write operation aiming at the target object in the cache layer and judges whether the number of the write operation is larger than the number threshold value or not. And if the number of the write operations is larger than the number threshold, sending all the write operations to the back-end storage. Write operations are performed in back-end storage.
Therefore, when the method determines that the number of the write operations aiming at the target object exceeds the number threshold, the write operations are sent to the back-end storage, so that the back-end storage executes the write operations. The method limits the number of write operations aiming at the target object acquired by the back-end storage each time, avoids the situation that overtime suicide of threads or request analysis errors occur in the back-end storage, and solves the problem that the existing storage system is easy to break down when executing too many write operations aiming at the same object.
In addition, the invention also provides a storage system write operation processing device, a storage system write operation processing device and a computer readable storage medium, and the beneficial effects are also achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for processing a write operation of a storage system according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for processing a write operation in a memory system according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a write operation processing apparatus of a memory system according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a storage system write operation processing device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating a method for processing a write operation of a memory system according to an embodiment of the present invention. The method comprises the following steps:
s101: and acquiring the write operation aiming at the target object in the cache layer, and judging whether the number of the write operation is greater than a number threshold value.
In order to improve the read-write efficiency, in the existing distributed storage system, a cache layer exists between a client and a backend storage. The Cache layer is a Cache layer, and the Cache Memory is a primary Memory existing between the main Memory and the CPU, and generally consists of a Static Random-Access Memory (SRAM), and has a smaller capacity but a much higher speed than the main Memory, which is close to the speed of the CPU. In the hierarchy of computer memory systems, there is a high speed, small capacity memory between the central processor and the main memory. Which together with the main memory constitutes the primary memory.
The write operations for the target object are obtained in the cache layer, and the write operations in the embodiment of the invention are all discontinuous write operations, namely write operations with discontinuous write addresses. Specifically, after generating the write request, the client sends the write request to the cache layer. The target object is an object to which a write operation needs to be performed, and any one of the objects may be the target object. After the write request aiming at the target object is obtained, the write request can be immediately analyzed to obtain write operation; or the write requests can be analyzed uniformly when the number of the write requests reaches a preset number, so that the write operation is obtained. After the write operations for the target object are acquired, whether the number of write operations is greater than a number threshold is determined. The number of write operations included in each write request is not limited in this embodiment, and for example, one write request may include only one write operation. It is determined whether the number of write operations is greater than a number threshold. The specific size of the number threshold is not limited in this embodiment, and generally, may be set to 10000. The embodiment does not limit the operation execution frequency of the write operation, and for example, the write operation can be acquired in real time; or the write operation may be acquired in a preset acquisition cycle.
In the prior art, a cache layer determines at what time to send an operation to a backend storage according to factors such as the use size of the current cache layer, the number of read/write operations stored in the current cache layer, the storage time of the read/write operations, and the like, which may result in a large number of write operations for an object, even exceeding 65535, and further cause the backend storage to have a situation of thread timeout suicide or request parsing error, and thus the write operation for the object cannot be executed. In this embodiment, it is determined whether the number of write operations for each object is greater than the number threshold, where the object currently being determined is the target object, and if the number of write operations for the target object is greater than the number threshold, the process proceeds to step S102; when the number of write operations is not greater than the number threshold, the number of write operations for the target object may be retrieved and determined, or no operation may be performed, and the operation when the number of write operations is not greater than the number threshold is not limited in this embodiment.
S102: all write operations are sent directly to the back-end storage.
When it is determined that the number of write operations for the target object is greater than the quantity threshold, all write operations are sent directly to the back-end storage. Specifically, the write request is acquired from the cache layer and sent to the back-end storage, and the back-end storage analyzes the write request to obtain the write operation. Because the number of the write requests included in the write operation does not exceed the number threshold, the back-end storage does not have the situation of thread overtime suicide or request analysis error in the process of analyzing the write requests to obtain the write operation. In the embodiment of the invention, the structure of the back-end storage is distributed, so that the write request is sent to the main OSD stored in the back-end. It should be noted that in this embodiment, the write operations sent to the backend storage are all write operations for a target object, that is, write operations for the same object and the number of the write operations exceeds a preset threshold. The present embodiment is also not limited to whether to send other operations for other objects when sending a write operation for a target object, and for example, may also send read operations for other objects at the same time. Preferably, in order to enable the backend storage to process the write operation to the target object as soon as possible and prevent the problem of thread timeout suicide, the write operation to the target object is only sent to the backend storage.
S103: write operations are performed in back-end storage.
When a write request is executed in the back-end storage, specifically, the write operation may be obtained by the master OSD, and the write operation is sent to the slave OSDs storing the target objects, and the write operations are obtained from the slave OSDs and executed one by one.
Further, in order to increase the processing speed and reduce the time required for performing the write operation, it is preferable in the present embodiment to perform the merge processing on the write operation. Referring to fig. 2 in detail, fig. 2 is a flowchart of another write operation processing method of a storage system according to an embodiment of the present invention, including:
s201: and combining all the write operations in the main OSD stored at the back end to obtain an integral write operation.
Specifically, all write operations in the main OSD stored at the back end are combined, and this embodiment does not limit the specific method of combining, and reference may be made to related technologies. For example, the write operations for the target object may be merged into one integral write operation according to a merge rule in a memory storing the main OSD at the back end, and the specific content of the merge rule is not limited in this embodiment as long as a plurality of write operations can be merged into one integral write operation according to the merge rule.
S202: the overall write operation is performed.
Performing the overall write operation in the back-end storage, for example, the target object may be read out from the slave OSD into the master OSD, where the overall write operation is performed on the target object; or the entire write operation may be transmitted to the slave OSD storing the target object, where it is performed. By using the write operation execution method, the time for executing the write operation can be reduced, and the operation processing speed can be improved.
By applying the storage system write operation processing method provided by the embodiment of the invention, the write operation aiming at the target object is obtained in the cache layer, and whether the number of the write operation is greater than the number threshold value or not is judged. And if the number of the write operations is larger than the number threshold, sending all the write operations to the back-end storage. Write operations are performed in back-end storage. The method sends the write operation to the back-end storage when determining that the number of the write operations for the target object exceeds a number threshold, so that the back-end storage executes the write operation. The method limits the number of write operations aiming at the target object acquired by the back-end storage each time, avoids the situation that overtime suicide of threads or request analysis errors occur in the back-end storage, and solves the problem that the existing storage system is easy to break down when executing too many write operations aiming at the same object.
In the following, the storage system write operation processing apparatus provided by the embodiment of the present invention is introduced, and the storage system write operation processing apparatus described below and the storage system write operation processing method described above may be referred to correspondingly.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a write operation processing apparatus of a storage system according to an embodiment of the present invention, including:
a determining
a sending
and the
Optionally, the executing
the write operation merging unit is used for merging all write operations in the main OSD stored at the back end to obtain an integral write operation;
and the write operation execution unit is used for executing the whole write operation.
Optionally, the write operation merging unit includes:
and the rule merging subunit is used for merging the write operations aiming at the target object into an integral write operation according to a merging rule in a memory of the main OSD stored at the back end.
Optionally, the determining module includes:
a write request acquisition unit configured to acquire a write request for a target object;
and the write request analyzing unit is used for analyzing the write request to obtain the write operation.
In the following, the storage system write operation processing device provided by the embodiment of the present invention is introduced, and the storage system write operation processing device described below and the storage system write operation processing method described above may be referred to correspondingly.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a storage system write operation processing device according to an embodiment of the present invention, where the storage system write operation processing device includes a memory and a processor, where:
a
the
The following describes a computer-readable storage medium provided by an embodiment of the present invention, and the computer-readable storage medium described below and the storage system write operation processing method described above may be referred to correspondingly.
The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the storage system write operation processing method are implemented.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relationships such as first and second, etc., are intended only to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
The above detailed description is provided for the storage system write operation processing method, the storage system write operation processing apparatus, the storage system write operation processing device, and the computer-readable storage medium, and specific examples are applied herein to explain the principles and embodiments of the present invention, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.