阅读说明：本技术 模块化电子系统中的液体冷却分发 (Liquid cooled dispensing in modular electronic systems ) 是由 拉凯什·乔普拉 曼迪·欣·拉姆 M·巴里斯·多格鲁兹 于 2019-08-30 设计创作，主要内容包括：在一个实施例中,一种装置包括：机箱,该机箱包括用于接纳多个模块的多个插槽,该多个模块中的第一组模块在第一方向被接纳,并且该多个模块中的第二组模块在与所述第一方向正交的第二方向被接纳；以及冷却剂分发模块,该冷却剂分发模块在第一方向插入多个插槽中的一个插槽,以用于将冷却剂分发到第二组模块中的至少一个模块。本文还公开了一种用于将冷却剂分发到模块的方法。(In one embodiment, an apparatus comprises: a chassis including a plurality of slots for receiving a plurality of modules, a first set of the plurality of modules being received in a first direction and a second set of the plurality of modules being received in a second direction orthogonal to the first direction; and a coolant distribution module inserted into one of the plurality of slots in a first direction for distributing coolant to at least one module of the second set of modules. A method for distributing coolant to modules is also disclosed herein.)

1. An apparatus, comprising:

a chassis including a plurality of slots for receiving a plurality of modules, a first set of the modules being received in a first direction and a second set of the modules being received in a second direction orthogonal to the first direction; and

a coolant distribution module inserted into one of the plurality of slots in the first direction for distributing coolant to at least one module of the second set of modules.

2. The apparatus of claim 1, further comprising: a second coolant distribution module inserted into one of the plurality of slots in the second orientation for distributing coolant to at least one module of the first set of modules.

3. The apparatus of claim 2, wherein the second coolant distribution module receives coolant from the coolant distribution module inserted into one of the plurality of slots in the first direction.

4. The apparatus of any of claims 1-3, wherein one of the first and second sets of modules comprises a line card and the other of the first and second sets of modules comprises a network card.

5. The apparatus of any of claims 1-4, wherein the second set of modules comprises line cards that plug horizontally into the chassis.

6. The apparatus of any of claims 1-5, wherein the second set of modules comprises a network card that is inserted vertically into the chassis.

7. The apparatus of any of claims 1-6, wherein at least one of the first set of modules and the second set of modules is configured only for air cooling by a fan in the chassis.

8. The apparatus of any one of claims 1 to 7, wherein the coolant distribution module comprises at least one external connector for receiving the coolant.

9. The apparatus of any one of claims 1 to 8, wherein the coolant distribution module comprises a plurality of quick disconnect connectors for mating with coolant distribution lines on a plurality of modules in the second set of modules.

10. The apparatus of any one of claims 1 to 9, wherein the coolant passes through a heat exchanger external to the chassis.

11. The apparatus of any one of claims 1 to 10, wherein the coolant distribution module comprises internal connectors for mating with connectors on the second set of modules receiving the coolant, the internal connectors being offset from the location of orthogonal connectors on an air cooling module.

12. The apparatus of any one of claims 1 to 11, wherein the coolant distribution module comprises an internal connector for mating with a connector on the second set of modules receiving the coolant, the internal connector operable to move between: a recessed position when mated with an air-cooled module and an engaged position when mated with the connector on the second set of modules.

13. An apparatus, comprising:

a coolant distribution module for insertion into a chassis of a network device in a first direction for delivering coolant to one or more modules inserted into the chassis in a second direction orthogonal to the first direction, the coolant distribution module comprising:

a plurality of internal connectors for connecting to the one or more modules; and

a distribution line for delivering coolant to the plurality of internal connectors.

14. The apparatus of claim 13, wherein the coolant distribution module further comprises: at least one external connector for receiving the coolant from an external source.

15. The apparatus of claim 13 or 14, wherein at least one of the internal connectors is configured to distribute the coolant to a second coolant distribution module inserted into the chassis in the second orientation for distribution to one or more modules inserted into the chassis in the first orientation.

16. The apparatus of any of claims 13-15, wherein the one or more modules comprise a line card.

17. The apparatus of any of claims 13-16, wherein the one or more modules comprise a plurality of modules, at least one of the plurality of modules configured only for air cooling by a fan in the chassis.

18. A method, comprising:

receiving coolant at a coolant distribution module that is inserted into a slot of a chassis in a first orientation; and

coolant is delivered to a plurality of modules inserted into a plurality of slots in the chassis in a second direction orthogonal to the first direction for cooling components on the modules.

19. The method of claim 18, further comprising: delivering coolant to a second coolant distribution module for delivery to a plurality of modules inserted into slots of the chassis in the first orientation, the second coolant distribution module being inserted into one of the slots in the second orientation.

20. The method of claim 18 or 19, further comprising: air-cooling one or more modules inserted into the chassis and configured for air-cooling only.

Technical Field

The present disclosure relates generally to modular electronic systems, and more particularly to liquid cooling distribution in modular electronic systems.

Background

Over the past few years, the demand for higher performance communication networks has increased dramatically. The increased performance requirements result in increased energy usage, which increases the heat dissipation of the assembly. As power usage increases, conventional air cooling may no longer be sufficient to cool the network equipment, and thus liquid cooling may be required.

Modular electronic systems are designed to provide flexibility in configuring the system according to user needs. These systems include a number of slots to accommodate various modules (e.g., line cards, network cards, etc.). Orthogonal systems with horizontally and vertically inserted cards may present additional challenges for liquid cooling, as the cards are typically connected to the front and back of the system.

SUMMARY

Aspects of the invention are set out in the independent claims and preferred features are set out in the dependent claims. Features of one aspect may be applied to each aspect individually or in combination with the other aspects.

In one embodiment, an apparatus generally comprises: a chassis including a plurality of slots for receiving a plurality of modules, a first set of the plurality of modules being received in a first direction and a second set of the plurality of modules being received in a second direction orthogonal to the first direction; and a coolant distribution module inserted into one of the plurality of slots in a first direction for distributing coolant to at least one module of the second set of modules.

In another embodiment, an apparatus generally includes a coolant distribution module for insertion into a chassis of a network device in a first direction for delivering coolant to one or more modules inserted into the chassis in a second direction orthogonal to the first direction. The coolant distribution module includes a plurality of internal connectors for connecting to one or more modules and a distribution line for delivering coolant to the internal connectors.

In yet another embodiment, a method generally comprises: receiving coolant at a coolant distribution module that is inserted into a slot of a chassis in a first orientation; and delivering coolant to a plurality of modules inserted into a plurality of slots in the chassis in a second direction orthogonal to the first direction for cooling components on the modules.

A further understanding of the nature and advantages of the embodiments described herein may be realized by reference to the remaining portions of the specification and the attached drawings.

Example embodiments

The following description is presented to enable any person skilled in the art to make and use the embodiments. Descriptions of specific embodiments and applications are provided only as examples and various modifications will be apparent to those skilled in the art. The general principles described herein may be applied to other applications without departing from the scope of the embodiments. Thus, the embodiments are not limited to those shown, but are to be accorded the widest scope consistent with the principles and features described herein. For the sake of clarity, details relating to technical material that is known in the technical fields related to the embodiments have not been described in detail.

Network devices are evolving from traditional midplane designs to orthogonal-based systems that may include, for example, horizontally inserted line cards and vertically inserted network cards, which results in the orthogonal orientation of the line cards and network cards, where each network card is connected to all line cards and all line cards are connected to all network cards. This direct attachment eliminates the need for a chassis midplane.

As power consumption on processors and system components increases, conventional air cooling may no longer be sufficient and liquid cooling may be required. Liquid cooling provides improved efficiency and superior thermal performance compared to air cooling, and may be necessary to cool higher power processors required to meet future bandwidth requirements. Since the high power components of the circuit board are connected to the front and back of the quadrature-based system, the system may present additional challenges to the implementation of liquid cooling. Additional challenges are also presented if retrofitting of the air cooling system is desired.

Embodiments described herein provide a system that can operate in a liquid-cooled (or liquid-cooled and air-cooled) manner using one or more coolant distribution modules inserted in an orthogonal-based modular electronic system. Liquid cooling may be distributed for use with one or more modules inserted horizontally, inserted vertically, or both horizontally and vertically. One or more embodiments may support interchangeable air and liquid cooling modules in new chassis designs, or may adapt the air cooling system to achieve liquid cooling of high heat density electronics without managing separate liquid distribution (plumbing) connections at each module. In accordance with one or more embodiments, a system configured for air cooling may be modified to support a centralized liquid cooling distribution system. For example, the embodiments described herein may be used to convert an air-cooled chassis to a liquid-cooled chassis without requiring external plumbing connections for each field replaceable unit. In one or more embodiments, liquid cooling distribution may be provided by replacing one or more field replaceable units in an existing air-cooled chassis, and the system may use both air cooling and liquid cooling. Thus, embodiments allow both liquid cooling modules and air cooling modules to be used together in one chassis. As described in detail below, horizontal adapter cards (coolant distribution modules) may be used to provide liquid cooling to vertical modules (e.g., network cards), vertical adapter cards (coolant distribution modules) may be used to provide liquid cooling to horizontal modules (e.g., line cards), or both horizontal and vertical adapter cards may be used to provide liquid cooling to horizontal and vertical modules. Centralized coolant distribution eliminates the need for plumbing to manually connect and disconnect OIRs (on-line installation and removal) of field replaceable units (e.g., line cards, network cards). In one or more embodiments, a minimal amount of external plumbing connections may be used with a central coolant distribution system that manages the individual field replaceable units. In one or more other embodiments, a separate cooling system may be used to off-load the heat load from the line or network cards without any external piping.

The embodiments described herein may operate in the context of a data communications network comprising a plurality of network devices. A network may include any number of network devices that communicate through any number of nodes (e.g., routers, switches, gateways, controllers, edge devices, access devices, aggregation devices, core nodes, intermediate nodes, or other network devices) that facilitate the transfer of data across one or more networks. One or more network devices may include a modular electronic system that includes a liquid-cooled dispensing system as described herein. A network device may include one or more processors, memory, and network interfaces, where one or more of these components (e.g., line cards, network cards) on a module may be removably inserted into the network device.

It should be understood that the term "module" as used herein may refer to any modular electronic component, field replaceable unit, line card, network card, service card, or other card, element, or component configured for insertion into and removal from a chassis of a modular network device. In the examples described herein, line cards are inserted horizontally into the chassis and network cards are inserted vertically (orthogonal to the line cards) into the chassis, however, any type of module (e.g., line cards, network cards, or other cards) may be placed orthogonal to any other type of module. The term "orthogonal" as used herein generally describes a relationship between two geometries, wherein the two geometries are arranged substantially at 90 ° to each other. The terms "horizontal" and "vertical" are used herein as relative terms, and if a chassis is mounted on its side, the horizontal module will be mounted vertically, while the vertical module will be mounted horizontally.

Further, it should be understood that the term "liquid cooling" as used herein may refer to any coolant fluid (liquid, gas, or multi-phase) to distinguish it from "air cooling" provided by conventional fans. The coolant may include, for example, a liquid (e.g., water/glycol (e.g., 80% water, 20% glycol)), a non-conductive fluid/liquid, a gaseous refrigerant, or a mixed phase coolant (changing from liquid to gas in part along the coolant loop). The coolant may also be a compressed gas that is delivered to cool the module and then released into the open environment, thereby eliminating the need for a return line. The coolant may be provided by a source of cryogenic supply coolant that is routed through distribution pipes, as described below, and routed through elements within the network device, such as components on the modules (line cards, network cards, or other field replaceable units). The warm coolant may be collected through a return manifold where it removes heat from the coolant loop through a heat exchanger to an external cooling device, and then the cycle is repeated. The heat exchanger may be a liquid-to-liquid heat exchanger or a liquid-to-air heat exchanger, for example, provided with a fan to exhaust waste heat to the atmosphere. The heat exchanger may be located within the network device, proximate to the network device, or remote from the network device at a central location that serves any number of network devices. For example, the heat exchanger may be located in the same cabinet as the network equipment, or the system may be connected to a building-wide liquid cooling distribution system. In one or more embodiments, the heat exchanger may include two isolated fluid passages. If coolant flow from one channel stops, the other channel may provide enough coolant to keep the critical component operational. Isolation may be provided to prevent pressure losses in one fluid circuit from also affecting the pressure in the redundant circuit.

The pump for coolant distribution may be located external to the network device or within the modular electronic system. Additional pumps may also be placed within the coolant loop (e.g., on the coolant distribution module, on the liquid cooling card, or other location) as desired. In one or more embodiments, various sensors may monitor the total and individual branch coolant temperature, pressure, flow rate, or any combination thereof at strategic points around the circuit to identify loss of coolant or cooling.

In one or more embodiments, an apparatus comprises: a chassis including a plurality of slots for receiving a plurality of modules, a first set of the modules being received in a first direction and a second set of the modules being received in a second direction orthogonal to the first direction; and a coolant distribution module configured for insertion into one of the plurality of slots in a first direction and distributing coolant to at least one of the second set of modules for cooling components (electronic, optical) thereon. A second coolant distribution module may be inserted into one of the plurality of slots in a second orientation for distributing coolant to at least one module of the first set of modules.

Referring now to the drawings, and initially to fig. 1, a plurality of modules (e.g., line cards 10, network cards 12) are shown arranged orthogonal to one another (i.e., the edges/longitudinal surfaces of the line cards and network cards are orthogonal (vertical) to one another). In this example, the line cards 10 are horizontally positioned and the network cards 12 are vertically positioned. For simplicity, only four line cards 10 and seven network cards 12 are shown, and the chassis supporting these modules is not shown. In addition, the orthogonal data connectors between the horizontal line cards and the vertical network cards are not shown for simplicity. It should be understood that the system may include any number or type of modules 10, 12, wherein a first set of modules is inserted in a first direction and a second set of modules is inserted in a second direction orthogonal to the first direction. As shown in the simplified schematic diagram of fig. 1, each line card 10 may include any number of electronic components 13 (e.g., ASICs (application specific integrated circuits) or other integrated circuits, chips, processors, or high heat density electronic components), optical components 14 (e.g., optical transceivers), or other components in which the heat dissipation capacity of the components is insufficient to regulate their temperature. Each module 10, 12 may include any number of components 13, 14, wherein liquid cooling is provided to one or more of these components. As previously described, air cooling (e.g., provided by one or more fans) may also be provided to cool one or more components or modules that do not receive liquid cooling, or to provide additional cooling in conjunction with liquid cooling.

In the example shown in fig. 1, a vertical adapter card (also referred to herein as a coolant distribution module) 15 receives coolant at a connector 16 (external plumbing connection) and distributes the coolant via distribution lines 17 to each horizontal line card 10 through a coupler 18 (e.g., a quick disconnect connector or other pluggable liquid cooling connection), where the coupler 18 connects to distribution lines 19 that pass through the electronic components 13, 14. It should be understood that the location of the vertical adapter card 15 shown in FIG. 1 is merely an example, and that a vertical adapter card may be placed in any vertical slot in place of any network card 12, and that more than one vertical adapter card 15 may be used. The connectors 18 and conduits 19 may thus be placed at different locations along the edge of the line card 10 to mate with coolant distribution modules 15 inserted into different vertical slots within the chassis.

Referring now to fig. 2, an example is shown in which liquid cooling is provided to vertical cards 12 through the use of horizontal adapter cards (also referred to herein as coolant distribution modules) 25. A supply of liquid coolant is received at an external plumbing connection 26 and provided to each vertical network card 12 via a distribution line 27 at a coupler 28 (e.g., a quick disconnect connector or other pluggable liquid cooling connection) connected to a line 29 to cool the electronic components 23 on the network card 12. As previously noted with reference to fig. 1, the horizontal adapter card 25 may be inserted into any horizontal slot, and more than one card 25 may be used.

Fig. 3 shows an example of providing liquid cooling for both horizontal cards 10 and vertical cards 12. In this example, one horizontal module (line card) 10 and one vertical module (network card) 12 are replaced with coolant distribution modules 25, 35, respectively. External plumbing connections 26 distribute fluid to vertical cards 12 through horizontal adapter cards 25 (as described above with respect to fig. 2) and supply coolant to vertical adapter cards 35 at connectors 38, with vertical adapter cards 35 distributing coolant through distribution lines 17 to couplers 18 connected to each horizontal card 10 (as described above with respect to fig. 1). This eliminates the need for external plumbing connections at the vertical adapter card 35. As previously described, more than one horizontal adapter card 25 or vertical adapter card 35 may be used to provide redundant or additional cooling capabilities.

It should be understood that the liquid-cooled dispensing systems shown in fig. 1, 2, and 3 are merely examples, and that other dispensing configurations may be used without departing from the scope of embodiments. For example, any number of horizontal line cards 10 (e.g., none, one, some, or all) may be configured to receive liquid cooling, and any number of vertical modules 12 (e.g., none, one, some, or all) may be configured to receive liquid cooling, while any remaining cards are air cooled by fans. Thus, conventional line cards or network cards that are air cooled by fans and not configured for liquid cooling may be suitably used with the coolant distribution modules. The chassis may be configured with dedicated slots for the coolant distribution modules 15, 25, 35, or a conventional chassis may use one or more existing line cards or network card slots to receive the coolant distribution modules.

In the example shown herein, the external pipe connections 26 are provided on the front panel of the horizontal adapter card 25. The external plumbing connections may also be located on the sides, rear, or top of the chassis. For example, in one or more embodiments, external plumbing for the vertical adapter cards may be provided from the rear to provide cooling to the horizontal line cards 10 and horizontal adapter cards, which in turn provide cooling to the vertical network cards 12. Coolant distribution may be provided from the rear of the chassis, for example, by passing the coolant lines through a fan tray designed to provide space for the coolant lines. Also, for example, one or more fans may be removed to provide space for coolant distribution, or wiring may pass through space within a fan tray, between fans, above fans, or below fans. The coolant distribution lines may also pass through access panels (access panels) on the sides or top of the enclosure.

FIG. 4 is a fluid schematic diagram illustrating an overview of coolant distribution within horizontal adapter cards (coolant distribution modules) 40 and vertical adapter cards (coolant distribution modules) 42 according to one embodiment. In this example, the horizontal adapter card 40 includes two external plumbing connections, one for delivering coolant to the vertical adapter card 42(LC main supply/return) and the other for delivering coolant to the vertical network card (FC main supply/return). The LC main supply/return provides coolant to the vertical adapter card 42, which in turn, the vertical adapter card 42 delivers the coolant to one or more horizontal line cards (lc0.. LCn). The FC main supply/return provides coolant to one or more network cards (fc0.. FCn) through the horizontal adapter card 40.

The distribution pipe includes a supply line (cold fluid) and a return line (hot/warm fluid). Separate tubing may be provided for the supply line and return line, as shown in fig. 4, or a single tube with separate conduits for supply and return may be used, with sufficient thermal isolation between the supply line and return line, which will reduce the number of external and internal fluid connectors. As described in the examples below, the distribution lines may pass over or to the side of the orthogonal connector or within the space of the connector such that the coolant lines pass between the horizontal and vertical cards. In one example, the external connectors include a supply coupler for receiving cold fluid and a return coupler for returning hot/warm fluid after cooling the components on the line cards and network cards. The external connector may comprise two separate connectors for coupling two separate lines (supply line and return line) or a manifold comprising separate connections and internal channels with sufficient spacing between the supply channel and the return channel.

In the example shown in FIG. 4, the LC main supply/return and the FC main supply/return are each in fluid communication with an exterior heat exchanger 46. As described below, the horizontal adapter card 40 may include two pairs of external connectors (supply/return for line card cooling and supply/return for network card cooling). Fluid dispensing lines may also be grouped within a horizontal adapter card and use only one pair of connectors (supply/return). Further, the horizontal line cards 40 and the vertical adapter cards 42 may each have their own external connections in fluid communication with one or more heat exchangers 46. Each coolant loop may include one or more pumps 48 located outside the chassis or on the coolant distribution module. In another example, the coolant distribution module may include a pump to off-load the heat load from the module without the need for external piping.

The size of the fluid distribution lines may be determined based on the number of modules to be cooled and the thermal capacity of the modules. For example, different capacities of coolant distribution modules may be used based on the number of modules to be cooled using liquid cooling (e.g., one to seventeen (or more) line cards and one to seven (or more) network cards). The flow network modeling may be performed in consideration of the number of pumps, quick disconnects, fluid lines, and modules. Flow network modeling may also be performed on these cards, taking into account the cooling distribution system on the card (including fittings, tees, elbows, cold plates, or any other part of the distribution system) as well as the flow pressure and temperature distribution on the card. The hydraulic and thermal resistances of the cold plate and the card can be determined and optimized. The heat exchanger may be sized to adequately remove heat generated by the card through the coolant distribution system.

FIG. 5 is a front perspective view of chassis 50 with horizontal card adapters (coolant distribution modules) 52 and vertical card adapters (coolant distribution modules) 54 installed. Chassis 50 may include one or more frames or structures configured to support components and slidably receive any number of removable modules (adapter cards, line cards, network cards, or other removable modules). Chassis frame 50 may be formed from any suitable material including, for example, aluminum, steel, or any other metal, non-metal, or composite material.

As described above with respect to fig. 4, in one or more embodiments, the horizontal coolant distribution module 52 may include two pairs of external connectors 53, each pair including a supply line connector and a return line connector. The connector 53 may be a quick disconnect connector or any other suitable connector that prevents fluid loss when disconnected from an associated external cooling line. In this example, two external connectors 53 are coupled to the supply and return lines 55 for supplying coolant to the vertical adapter cards 54 and returning coolant from the vertical adapter cards 54 via distribution lines 66. The vertical adapter cards 54 distribute coolant to and receive return coolant from horizontal line cards (not shown) at connectors 58 (internal connectors) via coolant distribution lines 59. Two other external connectors 53 are coupled to coolant distribution lines 51, the coolant distribution lines 51 being attached to a manifold 56, the manifold 56 being configured to supply (distribute) coolant to and receive (aggregate) return coolant from a vertical network card (not shown) via lines 57. As previously mentioned, lines 51, 55, 57, 59 are appropriately sized to provide sufficient flow to cool the components on the module. As described above, the supply and return lines may be combined into a single line including isolated supply and return conduits.

Fig. 6 is a rear perspective view of the cabinet 50 shown in fig. 5. The vertical adapter card 54 includes a manifold 60, the manifold 60 for distributing supply coolant and aggregate return coolant via lines 59, the lines 59 coupled to the connectors 58 for attachment to horizontal line cards (not shown). The coolant lines 57 of the horizontal adapter card 52 are coupled at their open ends to connectors 62 for attachment to a vertical network card (not shown). As shown in the examples of FIGS. 5 and 6, horizontal adapter cards 52 and horizontal line cards are mounted into slots 65 from the front of chassis 50, and vertical adapter cards 54 and vertical network cards are mounted into slots 67 from the rear of chassis 50.

It should be appreciated that the type, number, and arrangement of slots 65, 67 and adapter cards 52, 54 shown in fig. 5 and 6 are merely examples, and a chassis may include any number of slots for receiving any number or type of modules arranged in any format (e.g., horizontally or vertically disposed), including, for example, network cards, line cards, service cards, combination cards, controller cards, processor cards, high density line cards, high power line cards, or high density and power line cards. Also, as previously described, the adapter cards 52, 54 may be inserted into any of the slots 65, 67 within the chassis 50.

As previously described, one or more line cards or network cards may not be configured for liquid cooling, but only for air cooling. Thus, the coolant distribution lines 57, 59 and the internal connectors 58, 62 are preferably configured so as not to interfere with orthogonal connectors on conventional air-cooled cards, and may be offset or recessed as described below.

Fig. 7 and 8 are perspective and side views, respectively, of chassis 50 shown in fig. 5 and 6 with walls of the chassis removed to show additional details of horizontal adapter cards 52 and vertical adapter cards 54, including distribution lines 51, 55, 57, 59, 66, external connectors 53, and internal connectors 58, 62. As shown in fig. 7, the coolant distribution lines (supply and return) 55 in fluid communication with the vertical adapter cards 54 may have a larger diameter than the coolant lines 57, with the coolant lines 57 being sized for coolant distribution to and return from the various vertical network cards. Similarly, coolant lines 66 coupled to lines 51 may have a larger diameter than lines 59, with lines 59 sized for coolant distribution to and return of coolant from the various horizontal line cards. Line 51 provides coolant to or receives coolant from manifold 56, and line 51 is also sized for higher flow rates.

Fig. 9 is a front view of chassis 50 showing external connectors 53 for horizontal adapter cards 52 and internal connectors 58 for coolant flow into and out of horizontal line cards (not shown). Fig. 10 is a rear view of the chassis 50 showing the internal connectors 62 for distributing coolant to and receiving return flow from the vertical network cards (not shown).

Fig. 11 is a top cut-away view of a horizontal line card (module) 110 mounted in a slot of a chassis 50. In this example, the line cards 110 are mounted in slots directly above the horizontal adapter cards. The line cards 110 receive coolant at supply lines 59, the coolant is delivered to the electronic assemblies 114 via lines 118, and the hot/warm coolant is returned to the vertical adapter cards 54 via lines 119. The line cards 110 include quick disconnect connectors for mating with the internal connectors 59 of the vertical adapter cards 54. The line cards 110 may include one or more electronic circuits and components 114 on a printed circuit board 116 and a data interface (quadrature connector) 115 for mating with a vertical network card. The line cards 110 may include a plurality of ports or other openings configured to receive connectors, cables, or pluggable transceiver modules (e.g., small form-factor pluggable (SFP) transceiver modules, optical transceiver modules, etc.). The line cards 110 may include one or more handles (arms) 112 for inserting the line cards into the chassis 50 or removing the line cards from the chassis 50.

Fig. 12A, 12B, 13A, 13B, 14, 15A, and 15B show details of coolant distribution modules (horizontal adapter cards 52, vertical adapter cards 54) according to one embodiment. As previously described, the coolant distribution module is configured for insertion into the chassis 50 of the network device in a first direction for delivering coolant to at least one module (line card, network card) inserted into the chassis in a second direction orthogonal to the first direction to cool components on the module. The coolant distribution module includes a plurality of internal connectors for connecting to one or more modules and a coolant distribution line for delivering coolant from the external connectors to the internal connectors.

Fig. 12A and 12B are perspective views of horizontal adapter card 52, fig. 13A is a top view of horizontal adapter card 52, and fig. 13B is a front view of horizontal adapter card 52 (e.g., the front of chassis 50). As previously described, the horizontal adapter card 52 includes two pairs of external connectors (quick disconnect connectors) 53 and lines 51 connected to a manifold 56, wherein the two pairs of external connectors 53 are attached to lines 55 to be in fluid communication with the vertical adapter card and the manifold 56 is also attached to a plurality of lines 57 to be in fluid communication with the vertical network card. In this example, an external quick disconnect connector 53 is located on the front panel of the adapter card 52 for ease of access. As previously mentioned, the external connectors may also be located on the back, top, or side panels of the chassis.

According to one embodiment, FIG. 14 is a perspective view of a vertical adapter card 54, FIG. 15A is a front view of the vertical adapter card 54 (e.g., the front of a chassis), and FIG. 15B is a side view of the vertical adapter card 54. In this example, the manifold 60 receives a supply of coolant from the horizontal adapter card at supply lines 66 and distributes the coolant to the horizontal line cards at supply lines 59. The return coolant returns to the manifold at return distribution lines 59 where the warm/hot coolant collects and is routed from return lines 66 to the horizontal adapter card 52. As previously described, coolant may also be supplied to the vertical adapter card 54 at an external connector separate from the horizontal adapter card.

It should be understood that the coolant distribution modules 52, 54 shown in fig. 12A, 12B, 13A, 13B, 14, 15A, and 15B are merely examples, and that other configurations may be used without departing from the scope of the embodiments. For example, as previously described, the dispense lines 51, 55, 57, 59, 66 may be configured to carry both supply and return fluid conduits within one tube, which will reduce the number of lines and connectors in the fluid dispense system. Moreover, the coolant distribution system may be independently contained on one or more coolant distribution modules 52, 54 and include pumps and heat exchangers or radiators to off-load heat loads from the line cards or network cards without any external piping.

As previously described, more than one coolant distribution module 52, 54 may be used to provide cooling path redundancy or additional cooling capacity. Each coolant distribution module may include its own external plumbing connections so that loss of liquid cooling flow from a single external plumbing connection will not cause possible thermal damage or shut down the system. Furthermore, this will allow one coolant distribution module to be repaired while another coolant distribution module carries the full system load. The dual redundant cooling circuits may be managed with check valves or may be completely isolated from each other.

In one or more embodiments, the coolant distribution modules 52, 54 may also include sensors and control components (flow valves, pumps, throttlers) for meeting coolant flow requirements for each individual branch (line card, network card). The control system may be, for example, an open loop system provided with manual valves at the time of system configuration, or a closed loop system using one or more servo valves regulated by feedback from a temperature sensor or coolant return temperature. The control system may also include a leak detection system (e.g., humidity sensor, flow sensor, pressure sensor) configured to shut off coolant flow if a leak is detected.

As previously mentioned, the liquid cooling distribution system may be used with conventional air cooling provided by one or more fans. In one or more embodiments, a conventional line card or network card designed for air cooling only may be used with modules configured for liquid cooling to provide backward compatibility and allow for a mix of air-cooled modules and liquid-cooled modules (or air and liquid-cooled modules). The horizontal and vertical coolant distribution modules 52, 54 described herein may also be inserted into a chassis where all modules (horizontal and vertical cards) are air cooled and not configured for liquid cooling. These modules may be replaced later with modules configured for liquid cooling. If horizontal or vertical coolant distribution modules are used with one or more conventional modules (air-cooled line cards or network cards) that are not configured for liquid cooling, various mechanisms or designs may be implemented to avoid damage to the orthogonal connectors. Fig. 16, 17A, 17B, 17C, 18A, 18B, 19A, and 19B illustrate examples of design configurations or mechanisms that may be used to prevent damage to orthogonal connectors located at the interface between an air cooling module (e.g., a conventional line card or network card) and a cooling distribution module.

Fig. 16 is a partial side view of the chassis 50 with one wall removed to show detail. In this example, one liquid cooled line card 110 is inserted into a slot directly above the horizontal line card adapter 52. Two conventional line cards 162 inserted above the liquid cooled line card 110 are configured for air cooling only (i.e., no coolant lines). The liquid cooled connector 58 engages the liquid cooled card 110 upon card insertion and is recessed from the orthogonal connector 115 on the conventional air cooled card 162 at the interface between the line card 162 and the vertical adapter card (connector 58). The conventional line cards 162 are inserted into the slots without disturbing the coolant connectors 58 and distribution lines 59 of the vertical adapter cards. The cooling assembly (connector 58) on the vertical adapter card is recessed from the conventional line card mating connector 115 so there is no interference between the vertical adapter card and the line card 162.

Similar to the example described above with respect to fig. 16, a conventional vertical network card (module) that is not configured for liquid cooling may be inserted into a chassis having a horizontal coolant distribution module for distributing coolant to at least one vertical card.

Fig. 17A-17C illustrate an example of a keying member 170 disposed at an interface between a horizontal module and a vertical module, according to one embodiment. The keying member 170 serves to align the modules and protect the quadrature cables from damage. In the example shown in fig. 17A-17C, the jack block option on the horizontal line cards 175 may determine keying features. Figure 17A shows two keyed guide pins 171 extending from a vertical adapter card and inserted into keyed members 170 on line card 175. A second guide pin is mounted on the vertical adapter card for horizontal slot at the pipe fitting. Fig. 17B is a rear perspective view of line card 175 showing keyed member 170 having opening 172 for receiving guide pin 171 when guide pin 171 is inserted into the chassis. Figure 17C shows blocked lead receptacle 174 on keying member 170 on a line card without liquid cooling.

Fig. 18A and 18B illustrate a liquid cooling interface operable to be recessed from its normal operating position when not engaged (i.e., not inserted into a chassis or inserted into a card that does not support liquid cooling). Shown in fig. 18A are horizontal line cards 180 that are not configured for liquid cooling, which are installed in a chassis with vertical coolant distribution modules 182, the vertical coolant distribution modules 182 including coolant infrastructure (lines/connectors) 184. In this example, the liquid cooling infrastructure 184 is recessed when not engaged to prevent interference with the orthogonal connector 185. The remaining orthogonal connectors 185 connect to a vertical network card 186. The vertical coolant distribution module 182 includes a sensor 188 with a retractable duct infrastructure 184, the sensor 188 detecting when it is inserted into a slot that interfaces with a card with a compatible liquid cooling infrastructure, thereby preventing damage to the orthogonal connector 185 on the air cooling module 180. The sensor 188 may include various types of mechanisms including, for example, a keying feature, a button, or a near field proximity sensor. Once the vertical adapter cards 182 are engaged with the corresponding horizontal liquid cooling modules 189 (fig. 18B), the coolant infrastructure 184 extends to engage with the coolant infrastructure 187 on the liquid cooling modules.

Fig. 19A and 19B show another example in which the coolant lines and connectors 194 are offset from the orthogonal connector 195, where sufficient space is available. This allows the liquid-cooled infrastructure to coexist with an orthogonal chassis without interference and enables a liquid-cooled card or a conventional air-cooled card to interface with the coolant distribution module without risk of damage. Fig. 19A shows the air cooling module 190 inserted into a chassis having a vertical coolant distribution module 192 and a vertical card 196. Fig. 19B shows a liquid cooling module 199 having cooling distribution lines 197 aligned with distribution lines 194 on a vertical coolant distribution module 192.

As previously noted, embodiments described herein may operate in the context of a network device. In one embodiment, the network device is a programmable machine that may be implemented as hardware, software, or any combination thereof. Network device 120 includes one or more processors 122, memory 124, and network interfaces (line cards, network cards) 126 (fig. 20). One or more components (e.g., processors, memories, interfaces (data, electrical, optical, cooling) may reside on a line or network card, and one or more components may reside on a coolant distribution module (e.g., cooling interface).

The memory 124 may be a volatile memory or a non-volatile storage device that stores various applications, operating systems, modules, and data for execution and use by the processor 122. Network device 120 may include any number of memory components.

Logic may be encoded in one or more tangible media for execution by processor 122. For example, the processor 122 may execute code stored in a computer-readable medium, such as the memory 124. The computer-readable medium may be, for example, electronic (e.g., RAM (random access memory), ROM (read only memory), EPROM (erasable programmable read only memory)), magnetic, optical (e.g., CD, DVD), electromagnetic, semiconductor technology, or any other suitable medium. In one example, the computer-readable medium includes a non-transitory computer-readable medium. Network device 120 may include any number of processors 122.

It should be understood that network device 120 shown in fig. 20 and described above is merely a simplified example and that the embodiments described herein may be implemented in network devices of different configurations. For example, network device 120 may further include any suitable combination of hardware, software, algorithms, processors, devices, components, or elements.

Fig. 21 is a flow diagram illustrating an overview of a process for distributing coolant in a modular electronic system, according to one embodiment. At step 130, coolant is received at a coolant distribution module (horizontal or vertical adapter card) that is inserted into the chassis in a first orientation. Coolant is delivered to modules (line cards, network cards) that are inserted into the chassis in a second direction orthogonal to the first direction of the coolant distribution modules (step 132). The flowing coolant is received back from the module (step 134) and the warm (or hot) coolant is returned to the heat exchanger (step 136). The coolant continues to flow while the network device is operational and supplied with coolant, or until the coolant distribution module is removed. As described above with respect to fig. 3, a second coolant distribution module may be inserted into one of the slots in the second orientation for distributing coolant to at least one of the modules in the first orientation. The same process described above will be repeated to deliver coolant from the second coolant distribution module to one or more modules in the first direction.

As previously described, quick disconnect connectors are used to allow for OIR (online insertion and removal) of any module (line card, network card) without interrupting the operation of the network equipment or the delivery of coolant to the remaining liquid cooling modules. The coolant distribution system is configured to ensure equal pressure when removing horizontal or vertical cards and to ensure proper pressure to avoid condensation from card type to card type. As previously described, the chassis may also include one or more modules (e.g., line cards or network cards) that are only air cooled.

It should be understood that the process shown in fig. 21 and described above is merely an example, and steps may be added, combined, modified, or removed without departing from the scope of embodiments.

Although the method and apparatus have been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations made to the embodiments without departing from the scope of the embodiments. It is intended, therefore, that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

Drawings

FIG. 1 illustrates liquid cooling distribution for horizontal modules in an orthogonal system according to one embodiment.

FIG. 2 illustrates liquid cooling distribution for vertical modules in an orthogonal system according to one embodiment.

FIG. 3 illustrates liquid cooling distribution for both horizontal and vertical modules in an orthogonal system, according to one embodiment.

FIG. 4 is a schematic diagram illustrating liquid cooling distribution in a coolant distribution module of the quadrature system of FIG. 3, according to one embodiment.

Fig. 5 is a partial front perspective view illustrating a coolant distribution module inserted into a chassis according to one embodiment.

Fig. 6 is a partial rear perspective view of the chassis and coolant distribution module of fig. 5.

Fig. 7 is a front perspective view of the cabinet of fig. 5 with a portion of the cabinet removed to show detail.

Fig. 8 is a side view of the enclosure and coolant distribution module shown in fig. 7.

Fig. 9 is a front view of the enclosure and coolant distribution module shown in fig. 5.

Fig. 10 is a rear view of the enclosure and coolant distribution module shown in fig. 6.

Fig. 11 is a cross-sectional view showing liquid cooling line cards inserted into a chassis having coolant distribution modules.

Fig. 12A is a front perspective view of a horizontal coolant distribution module.

Fig. 12B is a rear perspective view of the horizontal coolant distribution module.

Fig. 13A is a top view of a horizontal coolant distribution module.

Fig. 13B is a front view of a horizontal coolant distribution module.

Fig. 14 is a perspective view of a vertical coolant distribution module.

Fig. 15A is a front view of a vertical coolant distribution module.

Fig. 15B is a side view of a vertical coolant distribution module.

Fig. 16 is a partial side view of a chassis and coolant distribution module having both air-cooled line cards and liquid-cooled line cards with a portion of the chassis removed to show detail, according to one embodiment.

Figure 17A is a partial top view of a line card connected to a vertical coolant distribution module at a connection interface through keyed guides according to one embodiment.

Figure 17B is a partial front perspective view of the line card shown in figure 17A.

Figure 17C is a partial front perspective view of a line card configured for air cooling only in the event that the keying guides are blocked.

FIG. 18A is a schematic top view of an interface between an air-cooled module and a coolant distribution module having a recessed liquid-cooled connection according to one embodiment.

Fig. 18B is a top schematic view of a bonded connection between the liquid cooling module and the coolant distribution module of fig. 18A.

FIG. 19A is a schematic top view illustrating the liquid cooling assembly of a coolant distribution module being offset relative to an orthogonal connector on an air cooling module according to one embodiment.

Fig. 19B is a schematic top view illustrating a connection between the liquid cooling assembly and the liquid cooling module of the coolant distribution module of fig. 19A.

FIG. 20 is a block diagram of a network device in which embodiments described herein may be implemented.

FIG. 21 is a flow diagram showing an overview of a coolant distribution process for use in orthogonal modular electronic systems, according to one embodiment.

Corresponding reference characters indicate corresponding parts throughout the several views of the drawings.

Detailed Description