阅读说明：本技术 用于高速外设部件互连（PCIe）系统的机架式交换机设备 (Rack-mounted switch device for peripheral component interconnect express (PCIe) system ) 是由 C·R·隆 J·布雷克斯通 A·R·海德 B·M·鲁斯特 S·沃尔什 B·施拉姆 于 2018-03-07 设计创作，主要内容包括：本文提供了机架式高速外设部件互连(PCIe)交换机组件。一个示例PCIe交换机组件包括：外壳,其包围所述PCIe交换机组件的元件；第一多个PCIe互连端口,其位于所述PCIe交换机组件的前侧；以及第二多个PCIe互连端口,其位于所述PCIe交换机组件的后侧。提供一个或多个冗余交叉链路端口以利用至少另一PCIe交换机组件处理故障转移通信。PCIe交换机电路通信地联接到形成集群互连PCIe结构的所述第一多个PCIe互连端口和所述第二多个PCIe互连端口。控制处理器被配置为控制至少所述PCIe交换机电路和所述一个或多个冗余交叉链路端口的操作。(A rack-mounted peripheral component interconnect express (PCIe) switch assembly is provided herein. One example PCIe switch component includes: a housing enclosing elements of the PCIe switch component; a first plurality of PCIe interconnect ports located at a front side of the PCIe switch component; and a second plurality of PCIe interconnect ports located on a back side of the PCIe switch component. One or more redundant cross-link ports are provided to handle failover communications with at least another PCIe switch component. PCIe switch circuitry is communicatively coupled to the first plurality of PCIe interconnect ports and the second plurality of PCIe interconnect ports forming a cluster interconnect PCIe fabric. The control processor is configured to control operation of at least the PCIe switch circuit and the one or more redundant cross-link ports.)

1. A peripheral component interconnect express (PCIe) switch assembly, comprising:

A housing enclosing elements of the PCIe switch component;

A first plurality of PCIe interconnect ports located at a front side of the PCIe switch component;

A second plurality of PCIe interconnect ports located on a back side of the PCIe switch component;

One or more redundant crosslink ports configured to handle failover communications with at least another PCIe switch component;

a PCIe switch circuit communicatively coupled to the first plurality of PCIe interconnect ports and the second plurality of PCIe interconnect ports forming a cluster interconnect PCIe fabric; and

A control processor configured to control operation of at least the PCIe switch circuit and the one or more redundant cross-link ports.

2. The PCIe switch assembly of claim 1 wherein said one or more redundant cross-link ports comprises at least a first port configured to couple said control processor of said PCIe switch assembly to PCIe switch circuitry of said another PCIe switch assembly; and

Wherein the one or more redundant cross-link ports further comprise at least a second port configured to couple the PCIe switch circuit of the PCIe switch component to a control processor of the other PCIe switch component.

3. The PCIe switch component of claim 2 wherein said control processor of said another PCIe switch component controls operation of at least said PCIe switch circuitry of said PCIe switch component via said one or more redundant crosslink ports in response to a failure of at least said control processor of said PCIe switch component.

4. The PCIe switch component of claim 1 wherein said one or more redundant cross-link ports comprises at least a first port configured to couple said control processor of said PCIe switch component to a control processor of said another PCIe switch component.

5. The PCIe switch component of claim 4, wherein the control processor of the other PCIe switch component controls operation of at least the PCIe switch circuitry of the PCIe switch component via the one or more redundant cross-link ports in response to a failure of at least the control processor of the PCIe switch component.

6. The PCIe switch assembly of claim 1 wherein the one or more redundant cross-link ports comprises a x4 PCIe port.

7. The PCIe switch component of claim 1, wherein said PCIe switch component and said another PCIe switch component are mounted side-by-side in a rack configuration.

8. The PCIe switch component of claim 1 further comprising:

One or more removable fans located in the enclosure and proximate to the front side of the PCIe switch component to provide airflow for the elements of the PCI switch component;

A fan service cover coupled to a top side of the enclosure and configured to provide service access to the one or more removable fans via the top side of the enclosure; and

The control processor configured to control operation of the one or more removable fans.

9. the PCIe switch component of claim 8 comprising:

the first plurality of PCIe interconnect ports are located on the front side of the PCIe switch component and below a front ventilation feature associated with the one or more removable fans; and

The second plurality of PCIe interconnect ports are located on a rear side of the PCIe switch assembly and under a rear ventilation feature associated with the one or more removable fans.

10. The PCIe switch component of claim 1 further comprising:

A power component removable from the back side of the PCIe switch component and configured to provide power to at least the elements of the PCIe switch component.

11. The PCIe switch component of claim 1 comprising:

The control processor is configured to monitor for failure of one or more PCIe switch elements in the PCIe switch circuit; and

in response to the failure, the control processor is configured to disable one or more PCIe switch elements associated with the failure and instruct at least a spare PCIe switch element of the PCIe switch circuits to handle respective PCIe communications in place of the one or more PCIe switch elements associated with the failure.

12. The PCIe switch component of claim 11 comprising:

the control processor is configured to reset the one or more PCIe switch elements associated with the failure; and

In response to determining that the one or more PCIe switch elements associated with the failure are no longer experiencing the failure, the control processor is configured to instruct the one or more PCIe switch elements associated with the failure to process the respective PCIe communications.

13. The PCIe switch component of claim 11 comprising:

The control processor is configured to store configuration information for the one or more PCIe switch elements in a non-volatile memory device;

In response to determining that the one or more PCIe switch elements associated with the failure are no longer experiencing the failure, the control processor is configured to retrieve the configuration information from the non-volatile memory device and instruct the one or more PCIe switch elements associated with the failure to process the respective PCIe communications based at least on the configuration information.

14. The PCIe switch component of claim 1 further comprising:

A display element located on the front side of the PCIe switch component and configured to display at least a status of the PCIe switch component; and

An angled front portion of the enclosure configured to provide airflow from a fan assembly of the PCIe switch assembly and around the display element on a front side of the PCIe switch assembly.

15. A method of operating a peripheral component interconnect express (PCIe) switch assembly, the method comprising:

Forming a cluster interconnect PCIe fabric between at least a first plurality of PCIe interconnect ports located at a front side of the PCIe switch component and a second plurality of PCIe interconnect ports located at a back side of the PCIe switch component; and

one or more redundant cross-link ports are provided to handle failover communications with at least another PCIe switch component.

16. the method of claim 15, wherein the one or more redundant cross-link ports comprises at least a first port configured to couple a control processor of the PCIe switch component to PCIe switch circuitry of the other PCIe switch component; and

wherein the one or more redundant cross-link ports further comprise at least a second port configured to couple PCIe switch circuitry of the PCIe switch component to a control processor of the other PCIe switch component.

17. the method of claim 16, wherein the control processor of the other PCIe switch component controls operation of at least the PCIe switch circuitry of the PCIe switch component via the second port in response to a failure of at least the control processor of the PCIe switch component.

18. The method of claim 15, wherein the one or more redundant cross-link ports comprises at least a first port configured to couple a control processor of the PCIe switch component to a control processor of the other PCIe switch component, and wherein the control processor of the other PCIe switch component controls operation of at least the PCIe switch circuitry of the PCIe switch component via the first port in response to a failure of at least the control processor of the PCIe switch component.

19. a rack-mounted peripheral component interconnect express (PCIe) switch assembly, comprising:

A housing enclosing elements of the PCIe switch component;

One or more removable fans located in the enclosure and proximate to a front side of the PCIe switch component to provide airflow for the elements of the PCI switch component;

A fan service cover configured to provide service access to the one or more removable fans via a top side of the enclosure of the PCIe switch assembly;

a first plurality of PCIe interconnect ports located on the front side of the PCIe switch component and below a front ventilation feature associated with the one or more removable fans;

A second plurality of PCIe interconnect ports located on a rear side of the PCIe switch component and below a rear ventilation feature associated with the one or more removable fans;

a power component removable from a back side of the PCIe switch component and configured to provide power to the elements of the PCIe switch component;

A PCIe switch circuit communicatively coupled to the first plurality of PCIe interconnect ports and the second plurality of PCIe interconnect ports forming a cluster interconnect PCIe fabric; and

A control processor configured to control operation of at least the PCIe switch circuitry and the one or more removable fans.

20. The chassis PCIe switch assembly of claim 19 further comprising:

One or more redundant crosslink ports configured to handle failover communications with at least another PCIe switch component; and

the control processing system is further configured to control operation of the one or more redundant cross-link ports.

background

Network storage and computing systems have been proposed that store and process large amounts of data in enterprise-level storage environments. These network storage systems typically provide end users or other external systems with access to bulk data storage through one or more network interfaces. In addition to data storage, remote computing systems include a variety of processing systems that can provide remote computing resources to end users. These network storage systems and remote computing systems may be included in high-density devices, such as rack-mounted environments. Some computing devices employ peripheral component interconnect express (PCIe) interfaces to connect peripheral devices and storage devices. However, typical PCIe implementations employ a point-to-point host device architecture.

Disclosure of Invention

A rack-mounted peripheral component interconnect express (PCIe) switch assembly is provided herein. One example PCIe switch component includes: a housing enclosing elements of the PCIe switch component; a first plurality of PCIe interconnect ports located at a front side of the PCIe switch component; and a second plurality of PCIe interconnect ports located on a back side of the PCIe switch component. One or more redundant cross-link ports are provided to handle failover communications with at least another PCIe switch component. PCIe switch circuitry is communicatively coupled to the first plurality of PCIe interconnect ports and the second plurality of PCIe interconnect ports forming a cluster interconnect PCIe fabric. The control processor is configured to control operation of at least the PCIe switch circuit and the one or more redundant cross-link ports.

drawings

Many aspects of this disclosure can be better understood with reference to the following drawings. The components in the figures are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the present disclosure is not limited to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications and equivalents.

Fig. 1 illustrates an embodiment of an exemplary PCIe switch device.

Fig. 2 illustrates an embodiment of an exemplary PCIe switch device.

Fig. 3 illustrates an embodiment of an exemplary PCIe switch device.

Fig. 4 illustrates an embodiment of an exemplary PCIe switch device.

Fig. 5 illustrates an embodiment of an exemplary rack-mounted PCIe switch device.

FIG. 6 illustrates an embodiment of an exemplary PCIe switch circuit.

FIG. 7 illustrates an embodiment of an exemplary PCIe switch circuit.

FIG. 8A illustrates an exemplary PCIe switch control in one embodiment.

FIG. 8B illustrates an exemplary PCIe switch control in one embodiment.

FIG. 9 illustrates an exemplary PCIe switch display in one embodiment.

Detailed Description

examples herein illustrate embodiments of various peripheral component interconnect express (PCIe) switch devices. A PCIe switch device may provide a switching fabric between various PCIe hosts and endpoints. Multiple PCIe compatible ports may be provided to interconnect other devices in the rack-mounted environment using associated PCIe links. Fig. 1-5 illustrate a rack mounted PCIe switch component having 16 front-end PCIe ports and 8 back-end PCIe ports, for a total of 24 PCIe ports. When coupled via associated wiring, the PCIe ports interconnect PCIe elements of a PCIe fabric, which includes other systems not shown in fig. 1.

in fig. 1, two switch assemblies 110 are shown in view 100, view 100 showing a top/front isometric view. The switch assembly 110 includes structural elements for receiving and structurally supporting the elements of the switch assembly 110. The housing 115 may include chassis elements, frames, fastening elements, rack-mount features, ventilation features, and other elements. The size of the housing 115 may be determined from a multiple of the standardized computer rack height, such as 1U or 2U. The housing 115 of the assembly 110 includes various chassis elements including top covers 111, 112, and 113. The cover 111 comprises a removable or top cover over the internal fan of the assembly 110. The cover 111 may be opened to provide service access to the fan within the housing. In many examples, the enclosure 115 also includes fans or other cooling and ventilation elements for providing airflow to the elements of the switch assembly 110. The housing 115 may also include power supply elements to convert external power or provide various forms of electrical power to the elements of the switch assembly 110.

the detailed view 101 shows the front side of the assembly 110. The front side of the assembly 110 includes a first plurality of PCIe ports 120 located below the ventilation feature 116 for the internal fans of the assembly 110. In FIG. 1, four sets of four PCIe ports are shown, including 16 front-end PCIe ports. In this example, the PCIe port includes a micro serial attached scsi (sas) HD connector, a Quad Small Form Factor Pluggable (QSFFP) jack, a QSFP/QSFP + jack, or a zSFP + interconnect, although other ports/jacks are possible.

in the example herein, PCIe ports 120 are grouped into sets of four (4) ports. Each switch assembly has a 4x4 port configuration on the front side and a 2x4 port configuration on the back side, which allows connection to other switches, devices, or servers with up to x16 buses and supporting bus widths of x4 and x 8. Each PCIe port 120 may also have a bottom mounted indicator (e.g., a Light Emitting Diode (LED)) that illuminates through the associated PCIe port connector to help guide customers to connect cables, troubleshoot, and indicate connections and errors.

the cover 111, the handle 114, the housing 115, and the ventilation feature 116 may each comprise various materials, such as metals, alloys, polymers, carbon composites, or other materials. In some examples, the cover 111, the handle 114, the housing 115, and the ventilation feature 116 all comprise aluminum material. Handles 114 are included on each end of the front side of the assembly 110 to facilitate insertion and removal of the assembly 110 into and from the rack system. In some examples, the handle 114 is coupled to a latch mechanism. When the handle 114 is pulled by a user, the latch mechanism retracts the latch feature coupled to the latch mechanism into the associated handle. Each latching feature retains the associated switch assembly 110 in a housing or rack-mounted system.

Also visible in fig. 1 is a redundant cross-link 180 comprising at least one x4 PCIe link coupling the management controllers of two switch assemblies 110 together for failover between the switch assemblies. In the event that a first switch component experiences a failure, control/management of the failed switch component may be taken over by an adjacent switch component coupled by link 180. Links 180 are coupled through one or more redundant cross-link ports 211 on switch component 110, the redundant cross-link ports 211 configured to handle failover communications with at least another PCIe switch component. The redundant cross-link port 211, via link 180, may allow the first PCIe switch component to control the operation of one or more other PCIe switch components, such as by sharing a single management processor between multiple PCIe switch components and multiple PCIe switch circuit entities. In this example, link 180 may be coupled using ports including micro serial attached scsi (sas) HD connectors, Quad Small Form Factor Plug (QSFFP) jacks, QSFP/QSFP + jacks, or zSFP + interconnects, although other ports/jacks are possible.

In fig. 2, switch component 110 is shown in view 200. Similar elements to those of fig. 1 are shown in fig. 2. Other electrical components may be included in fig. 2, but are omitted for clarity. View 200 shows a top/front isometric view with the cover removed to expose the service holes of the fan assembly 130 within the housing 115 and to expose other internal components. Fig. 2 also shows a latch 118 that may hold the associated switch assembly 110 in a housing or rack-mounted system. As described above, the latch 118 may be coupled to the handle 114 and the latch mechanism.

The fan assembly 130 may include a plurality of fan units that are removable for maintenance, cleaning, and replacement. A lid latch may be included to hold the lid 111 closed until an operator opens the lid 111. Spring-loaded or friction fit latches, as well as other latch types, including magnetic fasteners, may be employed.

In fig. 2, the fan assembly 130 includes a rotating fan element for moving air within the housing 115. These rotating fan elements may include one or more fins connected to a central shaft or shaft that rotates in response to an electric motor or driver. The fan assemblies 130 may each include any fan type, such as axial, centrifugal, and cross-flow, or other fan types, including associated ducts, louvers, fins, or other directional elements. The control of the fan assembly 130 may include proportional control features such as proportional-integral-derivative (PID) control for acoustic noise minimization.

Removal of the fan cover 111 is monitored by the control circuitry of the switch assembly 110. Once removal is detected, for example via a sense switch or other sensing circuitry, the control circuitry may trigger one or more maintenance events for the switch assembly 110. For example, if the fan assembly 130 is removed from the main assembly after the cover 111 is opened, the control circuit may: (1) starting a timer, (2) increasing the temperature polling frequency of a temperature sensor within the switch component 110, (3) responding to a time or temperature threshold-alerting the subscriber switch component 110 of an impending power down, and (4) powering down the switch component 110 for security to protect internal components from thermal damage.

The power supply assembly 160 includes an input power port 161 and a mounting feature 163 that retains the power supply assembly 160 in the housing 115. The power component 160 may be removed from the housing 115 by loosening the mounting feature 163. Two power supplies 162 are included in a redundant or parallel configuration to provide power to the internal components of the assembly 110. Each input power port may provide input power to an associated one of the power supplies 162.

Circuit board 155 is used to carry one or more circuit components of assembly 110. Circuit board 155 includes an assembly of various circuits, connectors, materials, interconnects, coatings, labels, fastener features, and other components. Typically, the circuit board 155 includes a separate printed circuit board as a base on which various components are mounted and in which electronic or optical interconnections are formed. The circuit board 155 may be secured to the chassis 115 for structural support and vibration resilience. The circuit board 155 includes various external connectors, such as PCIe ports and ethernet ports.

Circuit board 155 is configured to transfer power and signals between the components of assembly 110. Signaling includes PCIe signaling, sideband signaling, control signaling, discrete signaling, digital or analog signaling, and other signaling. Typically, input power is provided to the power supply 162 from a power supply via the input power port 161. Power control circuitry including a power supply 162 that filters, conditions, converts, and distributes input power to various components of the circuit board 155 via signaling associated with the circuit board 155. Various voltages may be used and converted by the power supply or on the circuit board 155. Circuit board 155 may include additional power control circuitry.

In fig. 2, a control processor 151 and one or more PCIe switch circuits included under a heat sink 153 are included. An optional data storage device 152 may be included to support the operation of control processor 151. Control processor 151 may include a modular CPU component coupled to circuit board 155 via one or more connectors. Further, in addition to the control processor 151, the circuit board 155 may include various microcontrollers (μ C), Field Programmable Gate Arrays (FPGA), or other processing and logic circuits.

fig. 2 also includes a mezzanine circuit board 210. The mezzanine circuit board 210 is coupled to the circuit board 155 via one or more connectors. Mezzanine circuit board 210 includes additional external ports, i.e., ports 211 and 212, for switch assembly 110. Ports 211 comprise redundant cross-link ports that support failover between switch components for link 180 in fig. 1. Ports 212 include control plane interconnects, such as gigabit Ethernet ports (10 GbE). Ports 212 may include small form-factor pluggable plus (SFP +) ports or gigabit interface converter (GBIC) ports, 10GbE, etc. The mezzanine board 210 can switch to support various port configurations, such as a first mezzanine configuration that supports SFP + ports and a second mezzanine configuration that supports GBIC ports, and so on. The control plane couples the processing elements of switch assembly 110 to other control processors and PCIe fabric management processors. Below the mezzanine port is a management port 122. The management port 122 comprises a local ethernet port protected by a firewall of other portions of the switch assembly 110 and may be coupled to a system administrator device or tool through an ethernet cable for inbox management functions and monitoring.

In fig. 3, switch component 110 is shown in view 300. Similar elements to those of fig. 1-2 are shown in fig. 3. Detailed view 301 shows a top/front isometric view with cover 111 laid flat upward to expose service holes for fan assembly 130 within housing 115. The fan assembly 130 may include a plurality of fan units that may be removed for maintenance, cleaning, and replacement. A lid latch 117 may be included to hold the lid 111 closed until an operator opens the lid 111. Spring loaded latches or friction fit latches, among other latch types, may be employed.

In fig. 4, switch component 110 is shown in view 400. Similar elements to those of fig. 1-3 are shown in fig. 4. View 400 shows a top/rear isometric view of power module 160 removed from housing 115. Removal may be performed for maintenance, replacement, or upgrade.

in fig. 5, switch component 110 is shown in view 500. Similar elements to those of fig. 1-4 are shown in fig. 5. View 500 shows a top/front isometric view of two switch assemblies mounted side-by-side via associated latches 118 and in a rack-mounted system comprising front vertical members 170 and rear vertical members 171. The horizontal rails 172 are configured to support two switch assemblies within the rack assembly. In a further example, four switch components 110 may be disposed in a single "tier" of a 1U rack-mount space. As shown in fig. 5, two front switch assemblies may be provided, with the other two switch assemblies facing rearward.

Fig. 6 illustrates an exemplary embodiment of the internal circuit components of the assembly 110. Fig. 6 is a block diagram illustrating a fabric switch assembly 620 as an example of circuitry and elements including electrical and software elements of the switch device 110. It should be understood that the elements of assembly 620 may be combined onto a single module or included in separate modules.

The fabric switch component 620 includes one or more PCIe switches 621, one or more power control modules 622 with associated optional hold circuits 623, a control processing system 624, a network switch 625, data storage 626, and ventilation fans 640. The fabric switch component 620 provides at least a portion of a peripheral component interconnect express (PCIe) fabric including PCIe links 602-604. PCIe links 602 provide external interconnects for devices of the compute/storage cluster to interconnect the various compute/storage rack-mounted modules. PCIe links 603-604 provide internal PCIe communication links for control processing system 624 as well as interconnect one or more PCIe switches 621. The fabric switch component 620 also provides one or more ethernet network links 607 via a network switch 625. Various sideband or auxiliary links 628 may also be used in the fabric switch component 620, such as a system management bus (SMBus) link, a Joint Test Action Group (JTAG) link, an integrated circuit bus (Inter-integrated circuit, I2C, Inter-integrated circuit) link, a Serial Peripheral Interface (SPI), a Controller Area Network (CAN) interface, a universal asynchronous receiver/transmitter (UART) interface, a Universal Serial Bus (USB) interface, or any other communication interface. Additional communication links may be included and are not shown in fig. 6 for clarity.

Each of the PCIe links 602 and 604 may include various widths or lanes of PCIe signaling. PCIe may support multiple bus widths, e.g., x1, x4, x8, x16, etc., where each multiple of bus width includes additional "lanes" for data transfer. PCIe also supports the transmission of sideband signaling, such as SMBus and JTAG, and associated clock, power, and bootstrap, among other signaling. For example, each of the links 602-604 may include a PCIe link having four lane "x 4" PCIe links, a PCIe link having eight lane "x 8" PCIe links, or a PCIe link having 16 lane "x 16" PCIe links, as well as other lane wide links.

One or more redundant and hot-pluggable power control modules 622, such as those shown in fig. 2 and 4, may be included in fabric switch assembly 620 for power supplies 162 and power modules 160. These power control modules may be redundant to power elements of the fabric switch assembly 620 during a failure of one of the power control modules. Power control module 622 receives source input power over link 630 and converts/conditions the input power for use by the elements of fabric switch assembly 620. The power control module 622 distributes power to each element of the fabric switch assembly 620 through an associated power link 631. Power control module 622 may selectively enable/disable power for each element in component 620. The power control module 622 includes circuitry to selectively and individually power any of the elements of the fabric switch assembly 620. The power control module 622 may receive control instructions from the control processing system 624 via an associated PCIe link or sideband link (not shown in fig. 6 for clarity). In some examples, the operation of the power control module 622 is provided by the processing element discussed for controlling the processing system 624.

Power control module 622 may include various power supply electronics, such as power regulators, boost converters, buck-boost converters, power factor correction circuits, and other power electronics. Various magnetic, solid state and other electronic components are typically sized according to the maximum power consumption of a particular application and are secured to an associated circuit board.

Alternatively, a hold circuit may be employed. When employed, the retention circuit 623 includes an energy storage device for storing power received through the power link 630 for use during a power interruption event, such as a loss of input power. The hold circuit 623 may include capacitive storage devices, such as capacitor arrays, as well as other energy storage devices. Excess or remaining retained power may be reserved for future use, shunted into a virtual load, or redistributed to other devices over a PCIe power link or other power link.

the ventilation fans 640 each include a rotating fan element or blower element for moving air within the associated enclosure. The rotating fan elements may include one or more fins coupled to a central shaft or shaft that rotates in response to an electric motor or electric drive. The ventilation fans 640 may each comprise any fan type, such as axial, centrifugal, and cross-flow, or other fan or blower types, including associated ventilation ducts, louvers, fins, or other air directing elements. Thermal sensors or thermal measurement circuits may be employed in the fabric switch assembly 620 to monitor the thermal status of the elements of the fabric switch assembly 620. In fig. 6, various discrete temperature sensors may be distributed throughout the fabric switch assembly 620 to measure temperature or other thermal information associated with the fan 640, the structural material of the enclosure, the ambient temperature outside the enclosure, the temperature inside the enclosure, or components of the fabric switch assembly 620.

the network switch 625 may include one or more ethernet switches to provide an ethernet link 607 including transceivers, transformers, isolation circuits, buffers, and the like. Ethernet link 607 may include a gigabit Ethernet (1GbE), a gigabit Ethernet (10GbE), a 40 gigabit Ethernet (40GbE) or a 100 gigabit Ethernet (100GbE), as well as other speed and format network links. The network switch 625 may carry ethernet communications, as well as any associated Internet Protocol (IP) and Transmission Control Protocol (TCP) communications, as well as communications of other network communication formats and protocols.

The control processing system 624 includes one or more microprocessors or microcontrollers and any associated memory 626. In some examples, control processing system 624 may be an implementation of portions of control processor 151. Control processing system 624 communicates over PCIe link 604 and ethernet link 607 and optional sideband communications 628 (including USB link, I2C link or serial link, etc., as described above).

The control processing system 624 may accumulate statistics and usage information for PCIe fabric switch components. The external system may retrieve this statistics or usage information via the network switch 625 or via the cluster interconnect 602 over the ethernet link 607. Fabric switch component 620 also provides an extension of the control plane for cluster control plane interconnection over ethernet link 607.

Control processing system 624 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that cooperate in executing program instructions. Examples of control processing system 624 include a general purpose central processing unit, a microprocessor, a special purpose processor and a logic device, and any other type of processing device. The control processing system 624 may include one or more non-transitory memory devices 626, such as RAM, solid State Storage Devices (SSD), or other memory, to store instructions that are executable by the control processing system 624 to operate as discussed herein. The control processing system 624 may monitor usage statistics, communication status, or other usage information of the fabric switch component 620 via links 604 and 607. Control processing system 624 may track this usage information during normal operations and data transfers within fabric switch component 620. In some examples, the memory devices 626 include one or more m.2 type Solid State Drive (SSD) memory devices.

The elements of memory device 626 may collectively comprise a non-transitory data storage system, although variations are possible. The memory device 626 may include any storage medium readable by the control processing system 624 and capable of storing software. The memory device 626 may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. The memory device 626 may include non-volatile storage media such as solid state storage media, flash memory, phase change memory, or magnetic memory, including combinations thereof. The memory device 626 may be implemented as a single storage device, but may also be implemented over multiple storage devices or subsystems. The memory device 626 may include additional elements, such as a controller, capable of communicating with the control processing system 624.

Software stored on or in memory device 626 may include computer program instructions, firmware, or some other form of machine-readable processing instructions having a process that, when executed, directly controls processing system 624 to operate as described herein. For example, software drivers control the processing system 624 to control the failover process and power modules of the PCIe switch, monitor operational statistics and status of the storage devices, monitor power status, and instruct the power circuitry 622 to control the flow of optional retention power or operational power, among other operations. The software may also include user software applications. The software may be implemented as a single application or as multiple applications. In general, software, when loaded into a processing system and executed, may transform the processing system from a general-purpose device to a special-purpose device customized as described herein.

The software modules 650 and 653 each include executable instructions that are executable by the control processing system 624 or the power control module 622 for operating the elements of the fabric switch component 620 in accordance with the processes discussed herein. In particular, failover manager 650 monitors the operational status of at least PCIe switch 621 to determine when one or more PCIe switches or associated links experience errors, failures, or other problems. Failover manager 650 may control the failover of communications handled by a first PCIe switch to a second PCIe switch, e.g., control communications partitioning, logical partitioning, non-transparent port configuration, logical domain configuration, or other configuration and partitioning between PCIe switches. When redundant PCIe switches are employed, the failover manager 650 can configure the backup PCIe switch to handle communications for other non-functional primary PCIe switches and ensure that communications processing can be maintained over the PCIe fabric and cluster interconnect.

the failover manager 650 restarts individual PCIe switches by cycling power to the associated PCIe switch. In other examples, a reset command may be issued to the PCIe switch. In still other examples, PCIe commands are issued to affected PCIe switches that command a reset of those particular PCIe switches. Failover manager 650 can read and store configuration information in SSD626, including configuration state or PCIe routing information for the affected PCIe switch, and reconfigure the PCIe switch using the stored state or PCIe routing information once the PCIe switch is reset. The configuration state or PCIe routing information may include an identification of the PCIe switch, an address of the PCIe switch, a logical partition or non-transparent port configuration of the PCIe switch, and other state, configuration, and information.

During the restart process, any system communicatively coupled via the affected PCIe switch may be isolated from the restart process using a redundant or standby PCIe switch such that the system is unaware of the failure, problem, or associated restart. If the reset process fails to address a particular PCIe switch, that PCIe switch may be powered down. During the reset/power cycle of the affected PCIe switches and when any particular PCIe switch fails, the remaining PCIe switches may continue to operate through the associated PCIe interfaces. Thus, enhanced operation may be achieved by continuing to operate the PCIe fabric during individual PCIe switch failures, and those individual PCIe switch failures may be addressed by the failover manager 650 without external/host system involvement or interruption through the use of the above-described processes.

The statistics monitor 651 monitors usage status or usage statistics of elements of the fabric switch assembly 620. The usage statistics include data transmission rate of the link, error rate of the link, cumulative number of link errors, and other statistics. Usage statistics may be collected and stored by control processing system 624 in a data structure, such as a database or table, and stored in memory device 626 or other storage element.

Power controller 652 monitors power statistics, power state statistics, power activity states, voltage levels, phase measurements, current consumption, hold circuit states or levels, power module insertion status, thermal levels, and other statistics during the course of operation. In response to the statistics monitor 651 and other signals, such as discrete signals monitored by the power control module 622, the power controller 652 instructs the power circuit to power up or power down the associated memory device or module. Responsive to an operational status of an associated PCIe switch, network switch; a command received over any of a PCIe, Ethernet, or sideband interface; or other factors, the power controller 652 may power up or power down the elements of the fabric switch assembly 620.

Thermal controller 653 controls the operation of fan 640. Thermal controller 653 may adjust the fan speed and fan operation of fan 640 based on the operating state, thermal state, temperature thresholds, or statistical information of fabric switch assembly 620. During operation of the fabric switch assembly 620, various temperature sensors may monitor the shell temperature of the fabric switch assembly 620. The airflow provided by the one or more fans 640 may be adjusted to change the airflow conditions of the enclosure. For example, the current fan speed and the current temperature within the enclosure may be monitored, and the fan 640 may be adjusted to adjust the airflow rate by changing the rotational speed of the fan 640. In examples where louvered or finned airflow apertures are used, the louvers or fins may be adjusted to vary the airflow. Thermal controller 653 may adjust fan speed and may receive feedback from fan 640 or from other sensors to indicate the current fan speed and associated temperature.

The software modules 650 & 653 may reside in a RAM portion of the memory device 626 during execution and operation by the control processing system 624, and may reside in non-volatile storage space of the memory device 626 in a powered-off state, as well as elsewhere during other states. The software modules 650 and 653 can be loaded into RAM during a boot or boot process as described for the computer operating system and applications. The non-volatile storage portion of the memory device 626 may include one or more storage systems including flash memory (such as NAND flash or NOR flash), phase change memory, magnetic memory, and other solid state storage technologies.

Control processing system 624 is generally intended to represent a computing system in which at least a software module 650 and 653 are deployed and executed to present or otherwise carry out the operations described herein. However, control processing system 624 may also represent any of the following computing systems: at least these software modules 650-653 can be staged across the computing system, and the software modules 650-653 can be distributed from the computing system, transmitted, downloaded or otherwise provided to yet another computing system for deployment and execution, or otherwise distributed. It should be appreciated that one or more of the software modules 650 and 653 may be stored, deployed, executed, and operated by any of the processing elements controlling the processing system 624 or the power control module 622.

each PCIe switch 621 comprises one or more PCIe crosspoint switches that logically interconnect respective ones of the associated PCIe links based at least on communications carried by the associated PCIe links. Each PCIe switch 621 establishes switched connections between any PCIe interfaces handled by each PCIe switch 621. In some examples, each PCIe switch 621 includes a PLX/Broadcom/Avago PEX 879624 port, a 96-lane PCIe switch chip, a PEX 872510 port, a 24-lane PCIe switch chip, a PEX97xx chip, a PEX9797 chip, or other PEX87xx/PEX97xx chips. In some examples, redundancy is established via one or more PCIe switches 621, such as having primary and secondary/backup PCIe switches in the PCIe switch. Failover from the primary PCIe switch to the secondary/standby PCIe switch may be handled by at least the control processing system 624. In some examples, redundant PCIe links linked to different PCIe switches may be used to provide primary and secondary functions in different PCIe switches. In other examples, the primary function and the secondary function may be provided in the same PCIe switch using redundant links linked to the same PCIe switch.

PCIe switches 621 each include a cluster interconnect interface 602, the cluster interconnect interface 602 for interconnecting modules of additional computing and storage systems in additional enclosures. The cluster interconnect provides a PCIe interconnect between external systems (e.g., computing and storage systems) through associated external connectors and external wiring. These connections may be PCIe links provided by any included PCIe switches as well as other PCIe switches not shown, for interconnecting other modules of the storage system via PCIe links. PCIe links for the cluster interconnect may terminate at external connectors or jacks, such as micro Serial Attached Scsi (SAS) connectors HD, quad small form-factor pluggable (QSFFP), QSFP/QSFP +, or zSFP + interconnects, which are used to carry PCIe signaling through associated wiring (e.g., micro SAS or QSFFP wiring). In a further example, a MiniSAS HD cable driving 12Gb/s is used, compared to a standard SAS cable driving 6 Gb/s. 12Gb/s can support at least PCIe generation 3.

The PCIe link 602-604 may also carry NVMe (non-volatile memory host controller interface Specification, NVM Express) communications issued by a host processor or host system. Nvme (nvm express) is an interface standard for mass storage devices, such as hard disk drives and solid state storage devices. NVMe can replace the serial ata (sata) interface to interface with mass storage devices in personal computer and server environments. However, these NVMe interfaces are limited to a one-to-one host drive relationship, similar to SATA devices. In examples discussed herein, a PCIe interface may be employed to transport NVMe communications and a multi-drive system including a number of storage drives is presented as one or more NVMe Virtual Logical Unit Numbers (VLUNs) on the PCIe interface.

Fig. 7 illustrates an exemplary embodiment of the internal electrical components of the switch assembly 110. Fig. 7 is a block diagram illustrating a switch component 700. The switch assembly 700 includes a housing or enclosure 710 and a main Printed Circuit Board (PCB) 711. Various sub-modules are also included and coupled to the circuit board 711 via one or more individual connectors 633-636. These sub-modules include a mezzanine circuit board 720, a computing module (CPU)721, and two vertically stacked m.2ssd 722. Including a removable power module 162 and a fan assembly/module 130. The power module 162 receives input power via the external connector 630 and provides converted internal power via the internal connector 631. The fan module 130 is coupled to the circuit board 711 via a connector 632, which connector 632 provides power and control signaling.

Mezzanine circuit board 720 may include the elements discussed above with respect to mezzanine circuit board 210 of fig. 2. Mezzanine circuit board 720 includes additional external ports, i.e., ports 607 and 680, for switch assembly 700. Ports 680 comprise redundant cross-link ports that support failover between switch components for link 180 in fig. 1. The port 607 includes a control plane interconnect, such as a gigabit Ethernet port (10 GbE). Ports 607 may include small form-factor pluggable plus (SFP +) or gigabit interface converter (GBIC) ports, 10GbE, etc. Mezzanine board 720 can switch to support various port configurations, such as a first mezzanine configuration that supports SFP + ports and a second mezzanine configuration that supports GBIC ports, and so on. The control plane couples the processing elements of switch assembly 700 to other control processors and PCIe fabric management processors. Below the mezzanine ports are a management port 640, a Universal Serial Bus (USB) type C connector 628, and a PCIe port 602.

fig. 8A and 8B illustrate exemplary embodiments of the internal electrical components of the switch assembly 110. Fig. 8A is a block diagram illustrating a switch component configuration 800. Fig. 8B is a block diagram illustrating a switch component configuration 801.

In the configuration 800, two switch components 620A-620B are presented in a redundant configuration using a processing system 624, which processing system 624 may include a CPU or other microprocessor. The x1 PCIe interface is coupled between each processing system 624 and PCIe switch 621 for PCIe communication snooping, switch management, and switch/communication flow control. Two x4 PCIe redundant links are included as PCIe cross links 880. If the processing system 624 of one of the switch components fails, the processing system of the other switch component can act as a failover processor to control the operation of at least the PCIe switch 621 in the failed switch component.

in configuration 801, two switch assemblies 620A-620B are presented in a redundant configuration using an FPGA824, which FPGA824 can include programmable logic or discrete logic with an embedded microprocessor or microcontroller. The x1 PCIe interface is coupled between each FPGA824 and PCIe switch 621 for PCIe communication snooping, switch management, and switch/communication flow control. Including an x4 PCIe redundant link as PCIe cross link 881, which x4 PCIe redundant link directly couples FPGA824 rather than the cross configuration shown in fig. 8A. If one of the switch components 'FPGAs 824 fails, the other switch component's FPGA824 can act as a failover processor to control the operation of at least PCIe switch 621 in the failed switch component.

In fig. 9, two stacked switch assemblies 110 are shown in view 900. Similar elements to those of fig. 1-2 are shown in fig. 9. Figure 9 is provided to emphasize the airflow into each switch assembly. In some examples, the upper/top switch component is omitted or includes another upper module that does not include the switch component 110. When stacked or otherwise blocked by modules above the lower switch assembly 110, air flow 930 is still provided to the internal fan assembly components located below the fan cover 111. This is provided in part by the angled front ventilation feature 116.

Also visible in fig. 9 is a user display 910. A user display 910 may optionally be attached to the front face of each switch assembly 110 and provide various visual or graphical information to the user, such as via an alarm, sound, graphical user interface, or other interface. The user display 910 may be attached using various fasteners, latches, or other mechanisms. In some examples, the user display 910 is coupled via magnetic fasteners and allows the user display 910 to be easily removed from the switch assembly 110 for user inspection, user manipulation, or quick/easy removal/replacement.

In yet a further example, the user display 910 includes an Organic Light Emitting Diode (OLED) array or display screen that can display any number of graphical user interface elements. The display may include indicators located near ports on the front face of the switch assembly 110 that may display status, communication indications, alarms, errors, or other information. In addition, status of the entire assembly, communication indications, alarms, errors, or other information may also be displayed. Various graphical icons or symbols may be employed to display information to a user. Also, a near field (e.g., NFC) or personal area network (e.g., bluetooth) transceiver may be included to allow a user to send/receive management information with the portable user device or to configure the display itself. Further, when the user display 110 is removable for handheld use by a user, a wireless link may be established that couples the user display 110 to the administrative state or information of the switch assembly 110.

the description and drawings are included to depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the invention. Those skilled in the art will also appreciate that the above-described features may be combined in various ways to form multiple embodiments. Accordingly, the present invention is not limited to the specific embodiments described above, but only by the claims and their equivalents.