阅读说明：本技术 一种存储集群系统、数据传输方法和装置 (A kind of storage cluster system, data transmission method and device ) 是由 王孝鹏 于 2019-08-21 设计创作，主要内容包括：本申请实施例公开了一种存储集群系统,包括多个存储控制器、PCIe接口卡和PCIe线缆。其中,多个存储控制器中的每个存储控制器具有PCIe插槽,PCIe接口卡插在存储控制器的PCIe插槽上。PCIe接口卡上提供PCIe插口,PCIe插口与PCIe线缆连接,这样便可利用PCIe线缆通过PCIe插口将分别具有PCIe接口卡的两个存储控制器连接,形成PCIe链路。与通过板载PCIe芯片建立存储控制器间的PCIe链路相比,通过PCIe接口卡提供对外接口,两个存储控制器具有PCIe接口卡,便可通过PCIe线缆将两个存储控制器连接形成PCIe链路,以提高存储控制器间数据传输速率,优化存储集群系统整体性能。(The embodiment of the present application discloses a kind of storage cluster system, including multiple storage controls, PCIe interface card and PCIe cable.Wherein, each storage control in multiple storage controls has PCIe slot, and PCIe interface card insertion is on the PCIe slot of storage control.PCIe socket is provided on PCIe interface card, and PCIe socket is connect with PCIe cable, two storage controls for being respectively provided with PCIe interface card can be connected by PCIe socket using PCIe cable in this way, be formed PCIe link.Compared with the PCIe link established between storage control by onboard PCIe chip, external interface is provided by PCIe interface card, two storage controls have PCIe interface card, two storage controls can be connected to form PCIe link by PCIe cable, to improve message transmission rate between storage control, optimize storage cluster system overall performance.)

1. a kind of storage cluster system, which is characterized in that the system comprises multiple storage controls, PCIe interface card and PCIe Cable:

Each storage control in the multiple storage control has PCIe slot；

The PCIe interface card, for being inserted in the PCIe slot of storage control；PCIe is provided on the PCIe interface card to insert Mouthful；

The PCIe cable, for being connected two storage controls for being respectively provided with PCIe interface card by the PCIe socket It connects, forms PCIe link.

2. system according to claim 1, which is characterized in that the two storages control for being respectively provided with PCIe interface card Device includes the first storage control and the second storage control, and first storage control is located at the first controller box, institute It states the second storage control and is located at second controller cabinet.

3. system according to claim 1 or 2, which is characterized in that the system also includes PCIe switches:

The PCIe switch is connect by the PCIe cable with the storage control with PCIe interface card.

4. system according to claim 1 or 2, which is characterized in that be inserted into a PCIe slot multiple described PCIe interface card.

5. system according to claim 1 or 2, which is characterized in that include at least one PCIe on the PCIe interface card Port.

6. a kind of data transmission method, which is characterized in that the method is applied to the described in any item storage collection of claim 1-5 Group's system, which comprises

Obtain data to be transmitted；The data to be transmitted is the data for needing to be transmitted by the first storage control；

If there are PCIe links between first storage control and the second storage control, transmitted by the PCIe link The data to be transmitted, second storage control remove other of first storage control for the storage cluster system Storage control.

7. according to the method described in claim 6, it is characterized in that, multiple second storage controls if it exists, described PCIe link is multiple PCIe links, described to transmit the data to be transmitted by the PCIe link, comprising:

Target PCIe link is determined from the multiple PCIe link using polling mode；

The data to be transmitted is transmitted by the target PCIe link.

8. according to the method described in claim 6, it is characterized in that, including at least one PCIe port, institute on PCIe interface card It states before transmitting the data to be transmitted by the PCIe link, the method also includes:

Check whether the port resource of port corresponding to the PCIe link is occupied；

The data to be transmitted is transmitted by the PCIe link, comprising:

The resource to be transmitted is sent from the unappropriated port of port resource.

9. according to the method described in claim 8, it is characterized in that, the method also includes:

If the port resource of the corresponding all of the port of the PCIe link is all occupied, the data to be transmitted is stored into pending Send queue.

10. according to the method described in claim 9, it is characterized in that, the method also includes:

If the port resource of the target port of the PCIe link is released, and includes number to be transmitted in the queue to be sent According to passing through the target port and send the data to be transmitted.

11. according to the method described in claim 6, it is characterized in that, the method also includes:

If the PCIe link breaks down, the notification information that the PCIe link disconnects is obtained.

12. a kind of data transmission device, which is characterized in that described device is applied to the described in any item storages of claim 1-5 Group system, described device include:

First acquisition unit, for obtaining data to be transmitted；The data to be transmitted is to need to be transmitted by the first storage control Data；

Transmission unit, if passing through institute for there are PCIe links between first storage control and the second storage control It states PCIe link and transmits the data to be transmitted, second storage control is the storage cluster system except described first deposits Store up other storage controls of controller.

Technical field

This application involves fields of communication technology, more particularly to a kind of storage cluster system, data transmission method and device.

Background technique

In order to meet continuous improvement of the user to storage demand, cluster storage system comes into being, and improves storage resource Capacity, reliability and availability.Include controller box in cluster storage system, includes storage control in controller box Device, for carrying out necessary control to the access of memory according to certain timing planning.

For the rate of improve data transfer, the stabilization of storage control is ensured, current storage control mostly uses greatly Dual control scheme installs two storage controls in an i.e. controller box, and two storage controls are mutually backups, same control Data transmission between two storage controls in device cabinet is all using onboard extension element interconnection (Peripheral Component Interconnect Express, PCIe) chip carried.

However, the link between these controller boxes is if in storage cluster system including multiple controller boxes Fiber channel (Fibre Channel, FC).Since the bandwidth of FC link limits, under the scene of big data quantity, the number of FC link Low according to transmission rate, the FC link between controller box often becomes the bottleneck of storage cluster system overall performance.

Summary of the invention

In order to solve the above-mentioned technical problem, this application provides storage cluster system, data transmission method and devices, improve Message transmission rate between storage control optimizes storage cluster system overall performance.

The embodiment of the present application discloses following technical solution:

In a first aspect, the embodiment of the present application provides a kind of storage cluster system, the system comprises multiple storages controls Device, PCIe interface card and PCIe cable:

Each storage control in the multiple storage control has PCIe slot；

The PCIe interface card, for being inserted in the PCIe slot of storage control；It is provided on the PCIe interface card PCIe socket；

The PCIe cable, for two storage controls of PCIe interface card will to be respectively provided with by the PCIe socket Connection forms PCIe link.

Optionally, two storage controls for being respectively provided with PCIe interface card include the first storage control and second Storage control, first storage control are located at the first controller box, and second storage control is located at the second control Device cabinet processed.

Optionally, the system also includes PCIe switches:

The PCIe switch is connect by the PCIe cable with the storage control with PCIe interface card.

Optionally, multiple PCIe interface cards are inserted into a PCIe slot.

It optionally, include at least one PCIe port on the PCIe interface card.

Second aspect, the embodiment of the present application provide a kind of data transmission method, and the method is applied to first aspect institute The storage cluster system stated, which comprises

Obtain data to be transmitted；The data to be transmitted is the data for needing to be transmitted by the first storage control；

If there are PCIe links between first storage control and the second storage control, pass through the PCIe link The data to be transmitted is transmitted, second storage control is the storage cluster system except first storage control Other storage controls.

Optionally, multiple second storage controls, the PCIe link are multiple PCIe links if it exists, described logical It crosses the PCIe link and transmits the data to be transmitted, comprising:

Target PCIe link is determined from the multiple PCIe link using polling mode；

The data to be transmitted is transmitted by the target PCIe link.

It optionally, include at least one PCIe port on PCIe interface card, it is described by described in PCIe link transmission Before data to be transmitted, the method also includes:

Check whether the port resource of port corresponding to the PCIe link is occupied；

The data to be transmitted is transmitted by the PCIe link, comprising:

The resource to be transmitted is sent from the unappropriated port of port resource.

Optionally, the method also includes:

If the port resource of the corresponding all of the port of the PCIe link is all occupied, by the data to be transmitted store into Queue to be sent.

Optionally, the method also includes:

If the port resource of the target port of the PCIe link is released, and includes to be transmitted in the queue to be sent Data send the data to be transmitted by the target port.

Optionally, the method also includes:

If the PCIe link breaks down, the notification information that the PCIe link disconnects is obtained.

The third aspect, the embodiment of the present application provide a kind of data transmission device, and described device is applied to described in first aspect Storage cluster system, described device includes:

First acquisition unit, for obtaining data to be transmitted；The data to be transmitted is to need by the first storage control The data of transmission；

Transmission unit, if leading to for there are PCIe links between first storage control and the second storage control It crosses the PCIe link and transmits the data to be transmitted, second storage control is the storage cluster system except described the Other storage controls of one storage control.

Optionally, multiple second storage controls, the PCIe link are multiple PCIe links, the biography if it exists Defeated unit, is used for:

Target PCIe link is determined from the multiple PCIe link using polling mode；

The data to be transmitted is transmitted by the target PCIe link.

It optionally, include at least one PCIe port, described device on PCIe interface card further include:

Whether inspection unit, the port resource for checking port corresponding to the PCIe link are occupied；

The transmission unit, is used for:

The resource to be transmitted is sent from the unappropriated port of port resource.

Optionally, described device further include:

Storage unit, if the port resource for the corresponding all of the port of the PCIe link is all occupied, will it is described to Transmission data are put into queue to be sent.

Optionally, described device further include:

Transmission unit, if the port resource of the target port for the PCIe link is released, and the team to be sent Include data to be transmitted in column, the data to be transmitted is sent by the target port.

Optionally, described device further include:

Second acquisition unit obtains the notice letter that the PCIe link disconnects if breaking down for the PCIe link Breath.

The embodiment of the present application has the advantage that it can be seen from above-mentioned technical proposal

Storage cluster system provided by the present application includes multiple storage controls, PCIe interface card and PCIe cable.Wherein, Each storage control in multiple storage controls has PCIe slot, and PCIe interface card can be inserted in storage control On PCIe slot.PCIe socket is provided on PCIe interface card, PCIe socket can connect with PCIe cable, can utilize in this way PCIe cable is connected two storage controls for being respectively provided with PCIe interface card by PCIe socket, forms PCIe link.With Conventional store group system is compared, and the onboard PCIe chip in traditional approach does not provide external interface, passes through onboard PCIe chip PCIe link can not be established between the storage control being located in different controller boxes.And the application is mentioned by PCIe interface card It, can be by PCIe cable by two storage controls if two storage controls have PCIe interface card for external interface Connection forms PCIe link, to improve message transmission rate between storage control, optimizes storage cluster system overall performance.

Detailed description of the invention

In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of application without any creative labor, may be used also for those of ordinary skill in the art To obtain other drawings based on these drawings.

Fig. 1 is a kind of topology example figure of storage cluster system provided by the embodiments of the present application；

Fig. 2 is a kind of flow chart of data transmission method provided by the embodiments of the present application；

Fig. 3 is a kind of structural block diagram of data transmission device provided by the embodiments of the present application.

Specific embodiment

In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is only this Apply for a part of the embodiment, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art exist Every other embodiment obtained under the premise of creative work is not made, shall fall in the protection scope of this application.

In traditional storage cluster system, the data between two storage controls in same controller box, which are transmitted, to be used Onboard PCIe chip is carried, but existing onboard PCIe chip does not provide external interface, be not used to controller machine The interconnection of storage control between case, therefore, the link between different controller boxes is FC link.Due to the band tolerance of FC link System, under the scene of big data quantity, the message transmission rate of FC link is low, and the FC link between controller box often becomes The bottleneck of storage cluster system overall performance.

In order to solve the above-mentioned technical problem, the embodiment of the present application provides a kind of storage cluster system, which includes multiple Storage control, PCIe interface card and PCIe cable.Wherein, each storage control in multiple storage controls has PCIe Slot, PCIe interface card can be inserted on the PCIe slot of storage control.PCIe socket is provided on PCIe interface card, PCIe is inserted Mouth can be connect with PCIe cable, can will be respectively provided with PCIe interface card by PCIe socket using PCIe cable in this way Two storage control connections, form PCIe link.Onboard PCIe core compared with conventional store group system, in traditional approach Piece does not provide external interface, can not be established between the storage control being located in different controller boxes by onboard PCIe chip PCIe link.And the application provides external interface by PCIe interface card, as long as two storage controls have PCIe interface card, Two storage controls can be connected to form PCIe link by PCIe cable, improve data transmission speed between storage control Rate optimizes storage cluster system overall performance.

Next, a kind of storage cluster system provided by the embodiments of the present application will be introduced in conjunction with attached drawing.Referring to figure 1, Fig. 1 shows a kind of structure chart of storage cluster system.The system comprises multiple storage controls 101, PCIe interface card 102 and PCIe cable 103, Fig. 1 are introduced by taking 2 storage controls 101 for including in system as an example.

Each storage control in the multiple storage control 101 has PCIe slot.

The PCIe interface card 102, for being inserted in the PCIe slot of storage control 101, the PCIe interface card 102 Upper offer PCIe socket.

The PCIe cable 103, for two storages of PCIe interface card 102 will to be respectively provided with by the PCIe socket Controller 101 connects, and forms PCIe link.

As shown in Figure 1, two storage controls 101 are respectively provided with PCIe interface card 102 in Fig. 1, the two storage controls Device 101 is connected by PCIe cable 103, to form PCIe link between the two storage controls 101.

It should be noted that two storage controls 101 with PCIe interface card 102 can be located at the same controller Cabinet can also be located at different controller boxes.

For example, two storage controls for being respectively provided with PCIe interface card include the first storage control and the second storage control Device processed, the first storage control can be located at the first controller box, and the second storage control can be located at second controller machine Case, to establish PCIe link between the storage control for being located at different controller boxes.

In some cases, many controller boxes be may include in storage cluster system, in different controller boxes Storage control is interconnected.For example, including 4 controller boxes in storage cluster system, in each controller box Storage control need with other controller boxes in storage control establish PCIe link, so as to cause 4 controllers Need to dispose many PCIe cables between cabinet.Wiring confusion caused by order to avoid excessive as PCIe cable, some In possible implementation, the system also includes PCIe switch, PCIe switch passes through the PCIe cable and has The storage control of PCIe interface card connects.

As such, it is desirable to which the storage control established in the controller box of PCIe link is exchanged by PCIe cable with PCIe Machine reduces the quantity of deployment PCIe cable.

In one possible implementation, multiple PCIe interface cards can be inserted in a PCIe slot, to be connected to A plurality of PCIe link.

It in one possible implementation, include at least one PCIe port, each on the PCIe interface card PCIe link has corresponding PCIe port.For a PCIe link, data can be sent by corresponding PCIe port, from And the transmission of data is realized by the PCIe link.

Storage cluster system provided by the present application includes multiple storage controls, PCIe interface card and PCIe cable.Wherein, Each storage control in multiple storage controls has PCIe slot, and PCIe interface card can be inserted in storage control On PCIe slot.PCIe socket is provided on PCIe interface card, PCIe socket can connect with PCIe cable, can utilize in this way PCIe cable is connected two storage controls for being respectively provided with PCIe interface card by PCIe socket, forms PCIe link.With Conventional store group system is compared, and the onboard PCIe chip in traditional approach does not provide external interface, passes through onboard PCIe chip PCIe link can not be established between the storage control being located in different controller boxes.And the application is mentioned by PCIe interface card It, can be by PCIe cable by two storage controls if two storage controls have PCIe interface card for external interface Connection forms PCIe link, to improve message transmission rate between storage control, optimizes storage cluster system overall performance.

Based on storage cluster system described in Fig. 1 corresponding embodiment, the embodiment of the present application also provides a kind of transmission side data Method.It is shown in Figure 2, which comprises

S201, data to be transmitted is obtained.

It is understood that after the starting of such as the first storage control of the storage control in storage cluster system, bottom Layer driving reports PCIe link information to communication layers.PCIe link information is for reflecting available PCIe link, including PCIe chain The mark for the storage control that road is connected, the mark of PCIe interface card, mark of port etc..Communication layers can recorde PCIe chain Road information distributes port resource for link transmission.

When there is data such as data to be transmitted to need to be transmitted, the available data to be transmitted of communication layers.Wherein, to be passed Transmission of data is the data for needing to be transmitted by the first storage control.

If between S202, first storage control and the second storage control, there are PCIe links, by described PCIe link transmits the data to be transmitted.

Wherein, second storage control is the storage cluster system except other of first storage control are deposited Store up controller.

First storage control can be located in same controller box, can also be located in different controller boxes. In this way, due to also can establish PCIe link between the storage control in different controller boxes, so as to pass through Data are transmitted between PCIe link realizes the storage control in different controller boxes, improve the efficiency of data transmission.

It is understood that just making if PCIe link is not present between the first storage control and other storage controls The data to be transmitted described in other types of link transmission, such as pass through data to be transmitted described in FC link transmission etc..

In some cases, multiple second storage controls, the PCIe link are multiple PCIe links if it exists, Connecting the PCIe link of the first storage control, there are multiple.In this case it is necessary to be selected from multiple PCIe links A PCIe link carries out data transmission out, realizes the load balancing between multiple PCIe links.Therefore pass through the PCIe in S202 A kind of possible implementation of data to be transmitted described in link transmission be using polling mode from the multiple PCIe link Determine target PCIe link；The data to be transmitted is transmitted by the target PCIe link.It is, of course, also possible to using preference+ The mode of poll determines target PCIe link from multiple PCIe links.

In one possible implementation, include at least one PCIe port on PCIe interface card, pass through in S202 Before the PCIe link transmits the data to be transmitted, communication layers can also traverse PCIe link, check the PCIe link Whether the port resource of corresponding port is occupied.If check some port port resource be it is occupied, from port provide The unappropriated port in source sends the resource to be transmitted.If the port resource for checking some port is occupied, under checking Whether the port resource of a port is occupied, and so on.

If after checking out all of the port corresponding to PCIe link, finding the port of the corresponding all of the port of the PCIe link Resource is all occupied, then stores the data to be transmitted and the port resource of port is waited to be released into queue to be sent.

If the port resource of some port such as target port of the PCIe link is released, team to be sent can detecte Whether include data to be transmitted in column, if in queue to be sent including data to be transmitted, institute is sent by the target port State data to be transmitted.

In one possible implementation, some or certain PCIe links may break down, if communication layers are not known Know which or which PCIe link and failure has occurred, then, communication layers still may transmit number to be transmitted by the PCIe link According to, and the PCIe link can not transmit data to be transmitted due to failure, to need to reselect to be used for transmission number to be transmitted According to PCIe link.In this case, in order to avoid the PCIe link that failure has occurred due to selection transmits data to be transmitted, Result in the need for reselecting the PCIe link for being used for transmission data to be transmitted, and caused by the wasting of resources, when PCIe link occurs When failure, the notification information that the available PCIe link disconnects, such as driving layer send notification information to communication layers, thus When selection is used for transmission the PCIe link of data to be transmitted, the interference for the PCIe link directly debugged avoids resource Waste.

Method provided by the embodiments of the present application can be applied in storage cluster system, which includes multiple storage controls Device, PCIe interface card and PCIe cable.Wherein, each storage control in multiple storage controls has PCIe slot, PCIe interface card can be inserted on the PCIe slot of storage control.PCIe socket is provided on PCIe interface card, PCIe socket can To connect with PCIe cable, two storages of PCIe interface card can will be respectively provided with by PCIe socket using PCIe cable Controller connection, forms PCIe link.In this way, when getting data to be transmitted, if the first storage control and the second storage There are PCIe links between controller, can transmit data to be transmitted by the PCIe link.Since PCIe link is to pass through What PCIe interface card and PCIe cable were established, PCIe interface card has external interface, as long as two storage controls are respectively provided with PCIe interface card just has the ability for establishing PCIe link, especially when two storage controls are located at different control In the case where device cabinet, so as to be realized by PCIe link, data are passed between the storage control in different controller boxes It is defeated, improve the efficiency of data transmission.

Based on the data transmission method that Fig. 2 corresponding embodiment provides, the present embodiment also provides a kind of data transmission device, institute Device is stated applied to storage cluster system described in Fig. 1 corresponding embodiment.Referring to Fig. 3, described device includes:

First acquisition unit 301, for obtaining data to be transmitted；The data to be transmitted is to need to be controlled by the first storage The data of device transmission；

Transmission unit 302, if between first storage control and the second storage control there are PCIe link, The data to be transmitted is transmitted by the PCIe link, second storage control is the storage cluster system except described Other storage controls of first storage control.

In one implementation, multiple second storage controls, the PCIe link are multiple PCIe if it exists Link, the transmission unit, is used for:

Target PCIe link is determined from the multiple PCIe link using polling mode；

The data to be transmitted is transmitted by the target PCIe link.

It in one implementation, include at least one PCIe port, described device on PCIe interface card further include:

Whether inspection unit, the port resource for checking port corresponding to the PCIe link are occupied；

The transmission unit, is used for:

The resource to be transmitted is sent from the unappropriated port of port resource.

In one implementation, described device further include:

Storage unit, if the port resource for the corresponding all of the port of the PCIe link is all occupied, will it is described to Transmission data are put into queue to be sent.

In one implementation, described device further include:

Transmission unit, if the port resource of the target port for the PCIe link is released, and the team to be sent Include data to be transmitted in column, the data to be transmitted is sent by the target port.

In one implementation, described device further include:

Second acquisition unit obtains the notice letter that the PCIe link disconnects if breaking down for the PCIe link Breath.

Those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above method embodiment can pass through The relevant hardware of program instruction is completed, and foregoing routine can be stored in a computer readable storage medium, which exists When execution, step including the steps of the foregoing method embodiments is executed；And storage medium above-mentioned can be at least one in following media Kind: read-only memory (English: read-only memory, abbreviation: ROM), RAM, magnetic or disk etc. are various to be can store The medium of program code.

It should be noted that all the embodiments in this specification are described in a progressive manner, each embodiment it Between same and similar part may refer to each other, each embodiment focuses on the differences from other embodiments. For equipment and system embodiment, since it is substantially similar to the method embodiment, so describe fairly simple, The relevent part can refer to the partial explaination of embodiments of method.Equipment and system embodiment described above is only schematic , wherein unit may or may not be physically separated as illustrated by the separation member, it is shown as a unit Component may or may not be physical unit, it can and it is in one place, or may be distributed over multiple networks On unit.Some or all of the modules therein can be selected to achieve the purpose of the solution of this embodiment according to the actual needs. Those of ordinary skill in the art can understand and implement without creative efforts.

The above, only a kind of specific embodiment of the application, but the protection scope of the application is not limited thereto, Within the technical scope of the present application, any changes or substitutions that can be easily thought of by anyone skilled in the art, Should all it cover within the scope of protection of this application.Therefore, the protection scope of the application should be with scope of protection of the claims Subject to.