阅读说明：本技术 存储资源处理装置及方法 (Storage resource processing device and method ) 是由 罗正伟 于 2020-05-09 设计创作，主要内容包括：本申请实施方式提供一种存储资源处理装置,包括多个通过网络连接的服务器,所述存储资源处理装置还包括设置模块、第一建立模块、第二建立模块,所述设置模块用于将每一服务器中的多个第一存储装置形成一个仿真硬盘,所述第一建立模块用于将每一服务器中所形成的仿真硬盘之间进行配对并建立成一个虚拟硬盘,当所述服务器的多个第二存储装置中的任一个第二存储装置损坏时,所述第二建立模块用于将所述虚拟硬盘与新的硬盘作配对并建立成一逻辑存储装置,以对所述逻辑存储装置进行数据存取操作。本申请实施方式还提供一种存储资源处理方法。由此,本申请实施方式提供的存储资源处理装置及方法,提高数据的安全性,并提升用户的体验度。(The embodiment of the application provides a storage resource processing device, which comprises a plurality of servers connected through a network, and further comprises a setting module, a first establishing module and a second establishing module, wherein the setting module is used for forming a simulation hard disk by a plurality of first storage devices in each server, the first establishing module is used for pairing the simulation hard disks formed in each server and establishing a virtual hard disk, and when any one of the second storage devices of the servers is damaged, the second establishing module is used for pairing the virtual hard disk and a new hard disk and establishing the virtual hard disk into a logic storage device so as to carry out data access operation on the logic storage device. The embodiment of the application also provides a storage resource processing method. Therefore, the storage resource processing device and method provided by the embodiment of the application improve the data security and improve the user experience.)

1. A storage resource processing apparatus comprising a processor, wherein the storage resource processing apparatus is communicatively coupled to a plurality of servers, the processor comprising:

the setting module is used for forming a plurality of first storage devices in each server into an emulation hard disk;

the first establishing module is used for pairing the simulation hard disks formed in each server and establishing a virtual hard disk; and

and the second establishing module is used for pairing the virtual hard disk with the new second storage device and establishing the virtual hard disk as a logic storage device when any one of the second storage devices in the plurality of second storage devices of the server is damaged so as to perform data access operation on the logic storage device.

2. The storage resource processing apparatus of claim 1, wherein the setup module forms the plurality of first storage devices in each server into the emulated hard disk when the plurality of first storage devices in each server are in an idle state.

3. The storage resource processing apparatus of claim 1, wherein the emulated hard disk has a first storage capacity, the virtual hard disk has a second storage capacity, the second storage capacity being greater than the first storage capacity.

4. The storage resource processing apparatus as claimed in claim 1, wherein the storage resource processing apparatus further comprises a detection module, when the first storage device in the server is damaged, the detection module is configured to detect a location of the damaged first storage device on the server, and the setting module is configured to re-establish an emulated hard disk with the undamaged first storage device using the new first storage device.

5. The storage resource processing apparatus according to claim 4, wherein when the server where the new hard disk is located is damaged, the detection module detects the damaged component in the server, the setting module reconstructs the plurality of first storage devices in the server into the emulated hard disk, and the first establishing module pairs the reconstructed emulated hard disks in each of the servers to reconstruct a virtual hard disk.

6. A storage resource processing method is applied to a storage resource device, wherein the storage resource processing device is in communication connection with a plurality of servers, and the storage resource processing method comprises the following steps:

forming a plurality of first storage devices in each server into an artificial hard disk;

pairing the simulation hard disks formed in each server, and establishing a virtual hard disk; and

when any one of the second storage devices of the server is damaged, the virtual hard disk is paired with the new second storage device and is established into a logic storage device so as to perform data access operation on the logic storage device.

7. The storage resource processing method of claim 6, wherein the plurality of first storage devices in each server are formed into the emulated hard disk when the plurality of first storage devices in each server are in an idle state.

8. The storage resource processing method of claim 6, wherein the emulated hard disk has a first storage capacity, the virtual hard disk has a second storage capacity, and the second storage capacity is greater than the first storage capacity.

9. The storage resource processing method according to claim 6, wherein the storage resource processing method further comprises the steps of:

when a first storage device in a server is damaged, detecting the position of the damaged first storage device on the server;

the damaged first storage device is replaced with the new first storage device, and an emulated hard disk is re-established with the undamaged first storage device.

10. The storage resource processing method according to claim 9, wherein the storage resource processing method further comprises the steps of:

when a server where a new hard disk is located is damaged, detecting a damaged component in the server;

and reforming the first storage devices in the servers into the simulated hard disks, and pairing the reformed simulated hard disks in each server to reestablish the virtual hard disk.

Technical Field

The present disclosure relates to the field of data storage, and in particular, to a storage resource processing apparatus and method.

Background

Due to the increasing Storage demand in recent years, many Storage devices, such as small NAS (Network Attached Storage ), are limited by the lack of sufficient lateral scalability, and have been unable to meet such large data Storage volumes, and thus research has gradually been focused on distributed Storage systems. The distributed storage system can connect the hard disk devices of a plurality of servers in series through a network to form a large-scale storage system. Through the processing of the distributed storage system, the storage capacity and I/O throughput of the whole system can be easily increased by increasing the number of servers and hard disks without being limited by the capacity and I/O upper limit of the conventional storage device.

In the prior art, data is distributed in a plurality of servers, and in order to maintain high availability of the data or avoid damage of hard disks, one copy of the data is copied and stored in the hard disks of different servers. When the hard disk or the server is damaged, the number of copies of some data is reduced, and when the distributed data storage system detects the situation, the distributed data storage system triggers a data backfill (backfile) action. However, as the amount of data stored increases and the size of the hard disk continues to expand to terabytes, the backfilling action will make the data of the system unsafe and time consuming, and further affect the user experience, due to the limitations in the natural physical speed of the hard disk, and the large amount of data that needs to be backfilled, which makes the hard disk take a long time to digest the backfilled data.

Disclosure of Invention

In view of this, a storage resource processing apparatus and method are needed, and the storage resource processing apparatus and method provided in the embodiments of the present application can greatly reduce the risk of data in the backfill process, improve the security of data, and improve the experience of a user.

An embodiment of the present application provides a storage resource processing apparatus, including a processor, the storage resource processing apparatus is in communication connection with a plurality of servers, the processor includes:

the setting module is used for forming a plurality of first storage devices in each server into an emulation hard disk;

the first establishing module is used for pairing the simulation hard disks formed in each server and establishing a virtual hard disk;

and the second establishing module is used for pairing the virtual hard disk with the new second storage device and establishing the virtual hard disk as a logic storage device when any one of the second storage devices in the plurality of second storage devices of the server is damaged so as to perform data access operation on the logic storage device.

The embodiment of the present application further provides a storage resource processing method, which is applied to a storage resource device, where the storage resource processing device is in communication connection with a plurality of servers, and the storage resource processing method includes the following steps:

forming a plurality of first storage devices in each server into an artificial hard disk;

pairing the simulation hard disks formed in each server and establishing a virtual hard disk;

when any one of the second storage devices of the server is damaged, the virtual hard disk is paired with the new second storage device and is established into a logic storage device so as to perform data access operation on the logic storage device.

According to the storage resource processing device and method provided by the embodiment of the application, the plurality of first storage devices in each server form the simulation hard disk, the simulation hard disks formed in each server are paired and established into the virtual hard disk with the second capacity, and the virtual hard disk and the new second storage device are paired and established into the logic storage device so as to perform data access operation on the logic storage device. Therefore, the storage resource processing device and method provided by the embodiment of the application greatly reduce the risk of data in the backfilling process and improve the safety of the data.

Drawings

FIG. 1 is a block diagram of a storage resource processing apparatus according to the present application.

FIG. 2 is a block diagram of a preferred embodiment of the processor of FIG. 1.

FIG. 3 is a diagram of a storage resource processing apparatus according to an embodiment of the present invention.

FIG. 4 is a diagram of another embodiment of a storage resource processing apparatus according to the present application.

FIG. 5 is a flowchart illustrating a storage resource processing method according to an embodiment of the present invention.

Description of the main elements

Storage resource processing apparatus 100

Processor 10

Setup module 101

First setup module 102

Second establishing module 103

Detection module 104

Flash cache module 105

Adjustment module 106

Memory 20

Server 200

First storage device 210

Second storage device 220

Virtual hard disk 230

Logic storage device 300

Caching device 310

Background device 320

Distributed data access system 400

The following detailed description will explain the present application in further detail in conjunction with the above-described figures.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application.

All other embodiments that can be obtained by a person skilled in the art without inventive step based on the embodiments in this application are within the scope of protection of this application.

Referring to fig. 1, a storage resource processing apparatus 100 according to a preferred embodiment of the present application is provided. The storage resource processing apparatus 100 is communicatively connected to a plurality of servers 200, and is configured to manage storage resources of the plurality of servers 200. In the present embodiment, the storage resource processing apparatus 100 is a management server.

The storage resource processing device 100 includes a processor 10 and a memory 20. The storage resource processing device 100 may be in communication connection with the server 200 in a wired or wireless manner.

Referring to fig. 2 to 4, in the embodiment of the present application, the servers 200 are also communicatively connected to each other. Each server 200 includes a plurality of first storage devices 210 and a plurality of second storage devices 220 for storing data. Preferably, the first storage device 210 is a memory, the second storage device 220 is a HDD (Hard disk drive), and the stored data is a program code and/or software data. Thus, the storage resource processing apparatus 100 can concatenate the HDDs on the servers with each other through the network to form a large storage system, that is, the HDDs on the servers are concatenated with each other to form the distributed data access system 400.

As shown in fig. 2, the storage resource processing apparatus 100 may include at least a setting module 101, a first establishing module 102, a second establishing module 103, a detecting module 104, a flash cache module 105, and an adjusting module 106. It is understood that in the present embodiment, the modules are programmable software instructions stored in the memory 20 and called by the processor 10 to be executed. It will be appreciated that in other embodiments the modules may also be program instructions or firmware (firmware) that are resident in the processor 10.

The setup module 101 is configured to form a plurality of first storage devices 210 in each server 200 into an emulated hard disk (not shown).

In the embodiment of the present application, 20 storage resource processing devices and 10 HDDs per storage resource processing device are taken as an example for explanation, and the number of the servers and the HDDs may be adjusted according to actual needs, which is not specifically limited in the present application.

For example, a storage server typically does not need to use much memory space, and a total of 16 memory slots are provided in the server 200, and typically only 4 32GB of memory (128 GB total) are inserted to save hardware cost. The remaining 12 memory slots are also filled with 32GB of memory (384 GB total), and this 384GB of memory space will be reserved for use in subsequent data backfilling. Assuming 20 servers, each server 200 has 10TB hard disks for the storage system. Because the memory of each server 200 is full, each server 200 has more memory space of 384 GB. The setting module 101 can thus establish these memory spaces as a memory emulation hard Disk (RAM Disk) with a storage capacity of 384 GB. Therefore, the 20 servers 200 have a total of 20 memory emulation hard disks of 384 GB.

The first establishing module 102 is configured to pair the emulated hard disks formed in each server 200 and establish a virtual hard disk 230.

For example, the storage resource processing apparatus 100 may use a distributed storage tool to build up 20 reserved memory emulation hard disks with a size of 384GB into a distributed storage system. That is, the first establishing module 102 establishes a virtual hard disk 230 with a storage capacity of 7680GB from the system.

When any one of the second storage devices 220 in the plurality of second storage devices 220 of the server 200 is damaged, the second establishing module 103 is configured to pair the virtual hard disk 230 with the new second storage Device 220 and establish a logical storage Device (Device Mapper Device)300 to perform a data access operation on the newly established logical storage Device 300. Since the newly created logical storage device will use the virtual hard disk as the read/write cache space. And using these newly built logical storage devices instead of HDDs as the basic storage devices of the distributed data access system, the logical storage device 300 can greatly increase the access speed.

In the embodiment of the present application, when the plurality of first storage devices in each server are in an idle state, the plurality of first storage devices in each server may form the emulated hard disk.

In this embodiment, the second establishing module 103 preferably pairs the virtual hard disk 230 with the new second storage device 220 through a flash cache module 105, thereby completing the establishment of the logical storage device 300. The flash cache module 105 may include a Bcache or a Flashcache software package.

If one hard disk is damaged and replaced with a new hard disk, data backfilling can be performed through the first building module 102 and the second building module 103.

Specifically, the logical storage apparatus 300 uses the virtual hard disk 230 as a cache device of a new hard disk, that is, the virtual hard disk 230 is a cache device (cache device)310 in the logical storage apparatus 300, and the new hard disk is a background device (backing device)320 in the logical storage apparatus 300. When the logical storage apparatus 300 is built, the adjusting module 106 adjusts the cache mode (cache mode) to the write-back (writeback) mode, that is, when there is data written to the logical storage apparatus 300, the writing is completed as long as the data is written to the cache device 310.

The data refilled from the remaining hard disk will start to be written to the cache device 310 in the logical storage apparatus 300, that is, the virtual hard disk virtualized by the memory space reserved by all servers. When all the data to be refilled into the new hard disk is written into the cache device 310, the data refilling operation is finished.

After the backfill action is finished, the adjusting module 106 converts the cache mode into a write around (write around) mode, that is, the new write requests are all directly written into the background device 320, and the storage function of the cache device is released from the logical storage device 300, and only the original background device is left to provide the storage service of the distributed data access system. When the cache device 310 is released from storage, the data stored in the cache device 310 is flushed (flush) to the background device 320, but this action is performed in the context of the operating system of the server to which the background device 320 belongs, and the background device 320 can operate independently in the distributed storage system after the flush is completed.

In the embodiment of the present invention, all the memory space reserved by the server 200 is used as the cache space of the newly replaced hard disk, and the cache mode is set as the write-back mode, so that the backfill data from other hard disks can be stored in the cache space virtualized by the memory. Because the data transmission of the memory is through the electronic signal, and is not limited by the speed limit of the physical hard disk rotation with the hard disk, the speed of the cache space virtualized by the memory is at least 100 times faster than that of the hard disk.

For example, if the IO performance of the new hard disk is relied on, it takes about 167 hours to backfill all the data to the new hard disk, so that the data backfill operation of the distributed storage system is completed. If the scheme of the embodiment of the present application is used, the data can be written into the cache space virtualized from the memory only about 1.67 hours, the data refilling operation of the distributed storage system can be finished, the remaining portion of the data written back to the background device 320 from the cache device 310 is executed by the operating system of the server to which the new hard disk belongs, and the writing speed of the new hard disk can reach more than 100 MB/s.

During the experiment, the parameters of the backfill data for the comparative example and the embodiment of the present application were recorded and the results are recorded in table 1 below.

TABLE 1 parameters of backfill data for comparative examples and examples

As can be seen from the above table, the method of the embodiment of the present application can make the time required for data backfill faster than that of the data backfill method in the prior art by more than 9 times.

It can be understood that the memory space (7680GB) reserved by the 20 servers in the embodiment of the present application is just large enough to completely store the data 6TB written by the other 199 hard disk data backfills. When the cluster of the distributed storage system is large enough, the remaining empty slots are quite amazing (each server has 12 empty slots), and if all slots can be inserted with full memory and used as cache for data backfill, the utilization rate of the servers can be higher, and the use of the computer room is more efficient.

In the embodiment of the present invention, if the first storage device 210 (i.e. the memory) of the server 200 is damaged, the storage resource processing device 100 can repair the server according to the following operations.

First, the detection module 104 detects the damaged first storage device 210, confirms its location on the server 200, and then replaces the damaged first storage device 210 with a new first storage device 210. The detection module 104 may include a memory test software package. Then, the setup module 101 will re-establish an emulated hard disk using the new first storage device 210 and the undamaged first storage device 210. Further, the first establishing module 102 re-establishes a virtual hard disk, and adds the virtual hard disk back to the distributed data access system. Then, the second establishing module 103 re-establishes the logical storage device with the new hard disk to be refilled by using tools such as Bcache or Flashcache, and finally re-executes the data refilling operation in the original distributed data access system.

In the embodiment of the present application, if the new hard disk is damaged, the storage resource processing apparatus 100 may repair the server according to the following operations.

Firstly, a new hard disk is used to replace the damaged hard disk, and the detection module 104 performs smart control check on the new hard disk to confirm that the new hard disk has no problem. Then, the second establishing module 103 reestablishes the logical storage device using the Bcache or the Flashcache and the virtual hard disk, adds the logical storage device back to the distributed data access system 400, and performs an action of refilling data.

In the embodiment of the present application, if the server where the new hard disk is located is damaged, the storage resource processing apparatus 100 may repair the server according to the following operations.

After shutdown, the detection module 104 detects a damaged component of the server, replaces the damaged component, and then starts the server to confirm that the damaged component is normal. The first establishing module 102 re-establishes a virtual hard disk in the reserved memory space of the server, and adds the virtual hard disk back to the distributed data access system. Then, the second establishing module 103 re-establishes the logical storage device with the new hard disk to be refilled by using tools such as Bcache or Flashcache, and finally re-executes the data refilling operation in the original distributed data access system.

The storage resource processing device 100 provided in the embodiment of the application can reduce the risk of data in the backfilling process, and improve the security of the data.

Referring to fig. 5, a memory resource processing method according to a preferred embodiment of the present invention can be used in the memory resource processing apparatus 100 of fig. 1 or fig. 2. The storage resource processing method comprises the following steps:

step S501: and forming the plurality of first storage devices in each server into an emulated hard disk.

Step S502: and pairing the simulated hard disks formed in each server, and establishing a virtual hard disk.

Step S503: and pairing the virtual hard disk and the new second storage device and establishing a logic storage device so as to perform data access operation on the logic storage device.

In this embodiment, when any one of the second storage devices of the server 200 is damaged, the virtual hard disk is paired with the new second storage device and is established as a logical storage device, so as to perform data access operation on the logical storage device.

According to the storage resource processing device and method provided by the embodiment of the application, the plurality of first storage devices in each server form the simulation hard disk, the simulation hard disks formed in each server are paired and established into the virtual hard disk with the second capacity, and the virtual hard disk and the new second storage device are paired and established into the logic storage device so as to perform data access operation on the logic storage device. Therefore, the storage resource processing device and method provided by the embodiment of the application greatly reduce the risk of data in the backfilling process and improve the safety of the data.

It should be understood by those skilled in the art that the above embodiments are only for illustrating the present application and are not used as limitations of the present application, and that suitable modifications and changes of the above embodiments are within the scope of the claims of the present application as long as they are within the spirit and scope of the present application.