阅读说明：本技术 基于高弹性部署的电子机架冷却系统 (Electronic rack cooling system based on high elasticity deployment ) 是由 高天翼 于 2020-12-16 设计创作，主要内容包括：本文公开了一种用于诸如IT机架或PoD的数据中心的电子机架的冷却系统。该系统包括通过流体冷却回路联接至电子机架的机架歧管的冷却剂分配单元(CDU)。CDU供应分配给IT机架或PoD的冷却流体,并从IT机架或PoD接收返回的温/热流体。该系统还包括增强冷却单元,以接收冷却流体的第一分配部分的第一部分,并通过增强冷却回路,将冷却流体的第一分配部分的第一部分进一步冷却到比供应的冷却流体中的一个更低的温度值。该系统还包括第一外部冷却单元,该第一外部冷却单元通过冷却流体回路连接至CDU,以将冷却流体供应至CDU。该系统还包括空气冷却单元,以接收冷却流体的第二分配部分,并使用冷却流体的第二分配部分来冷却用于IT机架或PoD的气流。该系统布置成五个部分,并且流体和冷却单元在分配回路内在各部分之间共享。(A cooling system for electronics racks of a data center, such as an IT rack or a PoD, is disclosed herein. The system includes a Coolant Distribution Unit (CDU) coupled to a rack manifold of the electronics rack by a fluid cooling circuit. The CDU supplies cooling fluid that is distributed to the IT racks or the pods and receives return warm/hot fluid from the IT racks or the pods. The system also includes an enhanced cooling unit to receive a first portion of the first distributed portion of cooling fluid and further cool the first portion of the first distributed portion of cooling fluid to a lower temperature value than one of the supplied cooling fluids through an enhanced cooling loop. The system also includes a first external cooling unit connected to the CDU by a cooling fluid circuit to supply cooling fluid to the CDU. The system also includes an air cooling unit to receive the second distribution portion of the cooling fluid and to cool an airflow for the IT racks or the PoD using the second distribution portion of the cooling fluid. The system is arranged in five sections, and the fluid and cooling units are shared between the sections within the distribution circuit.)

1. A cooling system for electronics racks of a data center, comprising:

a Coolant Distribution Unit (CDU) coupled to a rack manifold of the electronics rack by a fluid cooling loop, the CDU supplying cooling fluid distributed to the rack manifold and receiving warm/hot fluid returned from the rack manifold;

an enhanced cooling unit for receiving a first portion of the first distributed portion of the cooling fluid and further cooling the first portion of the first distributed portion of the cooling fluid to a temperature lower than a temperature of the cooling fluid supplied through a refrigeration circuit;

a first external cooling unit connected to the CDU through a cooling fluid circuit to supply the cooling fluid to the CDU; and

an air cooling unit to receive the second portion of the distribution of cooling fluid and to cool the electronics rack using the second portion of the distribution of cooling fluid.

2. The cooling system of claim 1, further comprising:

a first three-way valve for receiving a supply of cooling fluid and distributing the supply of cooling fluid into the first and second distribution portions; and

a second three-way valve to receive the first split portion of the cooling fluid and to deliver the first portion of the first split portion of the cooling fluid to the augmented cooling unit and to deliver a second portion of the first split portion of the cooling fluid to the rack manifold through the fluid cooling circuit.

3. The cooling system of claim 2, wherein the augmented cooling unit supplies a further cooled first portion of the first distributed portion of the cooling fluid to a fluid distribution circuit.

4. The cooling system of claim 2, further comprising a condensing unit connected to the augmented cooling unit by the refrigeration circuit.

5. The cooling system of claim 2, wherein the augmented cooling unit and the first and second three-way valves are included in the CDU.

6. The cooling system of claim 5, wherein the CDU further comprises a heat exchanger that exchanges heat between the returned warm/hot fluid and the supplied cooling fluid and a liquid pump that delivers the supplied cooling fluid into the first three-way valve.

7. The cooling system of claim 6, wherein the CDU includes a plurality of inlet and outlet ports coupled to the augmented cooling unit, the second three-way valve, the heat exchanger.

8. The cooling system of claim 7, wherein the plurality of inlet ports and the outlet port comprise:

connecting the augmented cooling unit to a first outlet port of the rack manifold,

connecting the second three-way valve to a second outlet port of the rack manifold,

connecting the rack manifold to a first inlet port of the heat exchanger,

a third outlet port and a second inlet port connecting the cooling fluid circuit between the heat exchanger and the first external cooling unit, an

A fourth outlet port and a third inlet port connecting the refrigeration circuit between the enhanced cooling unit and a second external cooling unit.

9. The cooling system of claim 7, wherein the plurality of inlet ports and the outlet port comprise:

connecting the augmented cooling unit and the second three-way valve to a first outlet port of the rack manifold,

connecting the rack manifold to a first inlet port of the heat exchanger,

a second outlet port and a second inlet port connecting the cooling fluid circuit between the heat exchanger and the first external cooling unit, an

A third outlet port and a third inlet port connecting the refrigeration circuit between the enhanced cooling unit and a second external cooling unit.

10. A cooling system for electronics racks of a data center, comprising:

a plurality of Coolant Distribution Units (CDUs) coupled to rack manifolds of the electronics racks, the plurality of CDUs comprising:

a first CDU supplying a first cooling fluid distributed to the rack manifold and receiving a return warm/hot fluid from the rack manifold, the first CDU including an enhanced cooling unit receiving a first portion of a first distributed portion of the first cooling fluid and further cooling the first portion of the first distributed portion of the first cooling fluid via a refrigeration loop, and

a second CDU for supplying a second cooling fluid distributed to the rack manifolds on a fluid cooling distribution loop and receiving a return warm/hot fluid from the rack manifolds, the second CDU including an enhanced cooling unit to receive a first portion of a first distribution portion of the second cooling fluid and further cool the first portion of the first distribution portion of the second cooling fluid via a cold fluid distribution loop, and

an air cooling unit to receive a second distribution portion of the first cooling fluid and a second distribution portion of the second cooling fluid and to cool the electronics rack with an airflow using the second distribution portion of the first cooling fluid and the second distribution portion of the second cooling fluid.

11. The cooling system of claim 10, further comprising a first external cooling unit connected to the first CDU by a cooling fluid circuit to supply the first cooling fluid to the first CDU, wherein the first external cooling unit is further connected to the second CDU by a cooling fluid distribution circuit to supply the second cooling fluid to the second CDU.

12. The cooling system of claim 11, further comprising a second external cooling unit connected to the augmented cooling unit of the first CDU by the refrigeration circuit, wherein the second external cooling unit is also connected to the augmented cooling unit of the second CDU by the cold fluid distribution circuit.

13. The cooling system of claim 10, wherein the first CDU further comprises a first three-way valve to receive a supply of first cooling fluid and distribute the supply of first cooling fluid into the first and second distribution portions of the first cooling fluid.

14. The cooling system of claim 13, wherein the first CDU further comprises a second three-way valve to receive the first split portion of the first cooling fluid and deliver the first portion of the first split portion of the first cooling fluid to the enhanced cooling unit of the first CDU and deliver a second portion of the first split portion of the first cooling fluid to the rack manifold.

15. The cooling system of claim 14, wherein the second CDU further comprises a first three-way valve to receive a supply of a second cooling fluid and distribute the supply of the second cooling fluid into the first and second distribution portions of the second cooling fluid.

16. The cooling system of claim 15, wherein the second CDU further comprises a second three-way valve to receive the first distributed portion of the second cooling fluid and deliver the first portion of the first distributed portion of the second cooling fluid to the enhanced cooling unit of the second CDU and deliver a second portion of the first distributed portion of the second cooling fluid to the rack manifold on the fluid cooling distribution circuit.

17. A data center system, comprising:

a plurality of independently deployable portions, comprising:

a housing portion comprising: (i) an electronics rack housing a plurality of server chassis, each server chassis corresponding to one or more servers, wherein the electronics rack includes a rack manifold to receive cooling fluid and return warm/hot fluid from the electronics rack, and (ii) an air cooling unit to cool the electronics rack,

a secondary distribution portion comprising: (i) a fluid cooling distribution loop for delivering the cooling fluid to the electronics racks, (ii) a system return distribution loop for delivering a return warm/hot fluid from the electronics racks, and (iii) an air cooling distribution loop for delivering a distributed portion of the cooling fluid to the air cooling unit, and

an enhanced cooling portion comprising: (i) a first Coolant Distribution Unit (CDU) for supplying a first portion of the cooling fluid to the rack manifold, receiving a first portion of the warm/hot fluid from the rack manifold, and supplying another distributed portion of the cooling fluid to the air cooling unit via the air cooling distribution loop, and (ii) a second CDU for supplying a second portion of the cooling fluid to the rack manifold via the fluid cooling distribution loop, receiving a second portion of the warm/hot fluid via the system return distribution loop, and supplying the distributed portion of the cooling fluid to the air cooling unit via the air cooling distribution loop.

18. The data center system of claim 17, wherein the plurality of independently deployable portions further comprises:

a main loop portion comprising: (i) a cooling fluid distribution circuit connecting the second CDU to a first external cooling unit, and (ii) a cold fluid distribution circuit connecting the second CDU to a second external cooling unit,

wherein the first external cooling unit supplies a first cooling fluid to the first CDU and a second cooling fluid to the second CDU, and the second external cooling unit supplies a first cold fluid having a lower temperature than the first cooling fluid to the first CDU and a second cold fluid having a lower temperature than the second cooling fluid to the second CDU.

19. The data center system of claim 18, wherein the plurality of independently deployable portions further comprises a cooling unit portion comprising the first external cooling unit and the second external cooling unit.

20. The data center system of claim 17, wherein the rack portion is a point of delivery (PoD).

Technical Field

Embodiments of the present invention generally relate to data center cooling. More particularly, embodiments of the present invention relate to a highly resilient deployment-based cooling system for electronics racks of Information Technology (IT) components.

Background

Cooling is a prominent factor in the design of computer systems and data centers. The number of high performance electronic components, such as high performance processors packaged within servers, is steadily increasing, thereby increasing the amount of heat generated and dissipated during ordinary operation of the servers. If the environment in which the servers are allowed to operate increases in temperature over time, the reliability of the servers used within the data center decreases. Maintaining a proper thermal environment is critical to the proper operation of these servers in a data center, as well as server performance and life, energy efficiency, and cost, especially in the case of cooling these high performance servers, which requires more efficient and effective cooling solutions.

For high-density electronics racks of IT components, liquid cooling has become an important technology for high power density servers and electronic devices such as Central Processing Units (CPUs)/Graphics Processing Units (GPUs)/Application Specific Integrated Circuits (ASICs). Such cooling systems need to be able to provide liquid distribution to those components. For other components or IT equipment, they may continue to be air cooled. This means that the racks are cooled by mixing air and liquid at the same time, so that air cooling is still required. This means that the system needs to be able to circulate both air and liquid. Therefore, the cooling system needs to provide cooling/cold fluid to the air cooling device.

In general, conventional cooling systems have primarily focused on liquid cooling, or have been optimized for either liquid cooling or air cooling. They cannot support different types of configurations or changes of racks or IT equipment without any system modifications, and are not cost effective. In addition, they are directed only to energy efficiency improvements on the infrastructure side, not on the rack or IT side. For example, those conventional cooling systems may be used to increase plant-side energy efficiency as operating temperatures increase. Unfortunately, this does not support IT and rack-side performance and power optimization. Therefore, facility and IT hardware performance and energy efficiency co-design are crucial in modern data centers.

Disclosure of Invention

According to an aspect of the present application, there is provided a cooling system for electronics racks of a data center, which may include:

a Coolant Distribution Unit (CDU) coupled to a rack manifold of the electronics rack by a fluid cooling loop, the CDU supplying cooling fluid distributed to the rack manifold and receiving warm/hot fluid returned from the rack manifold;

an enhanced cooling unit for receiving a first portion of the first distributed portion of the cooling fluid and further cooling the first portion of the first distributed portion of the cooling fluid to a temperature lower than a temperature of the cooling fluid supplied through a refrigeration circuit;

a first external cooling unit connected to the CDU through a cooling fluid circuit to supply the cooling fluid to the CDU; and

an air cooling unit to receive the second portion of the distribution of cooling fluid and to cool the electronics rack using the second portion of the distribution of cooling fluid.

According to another aspect of the present application, there is provided a cooling system for electronics racks of a data center, which may include:

a plurality of Coolant Distribution Units (CDUs) coupled to rack manifolds of the electronics racks, the plurality of CDUs comprising:

a first CDU supplying a first cooling fluid distributed to the rack manifold and receiving a return warm/hot fluid from the rack manifold, the first CDU including an enhanced cooling unit receiving a first portion of a first distributed portion of the first cooling fluid and further cooling the first portion of the first distributed portion of the first cooling fluid via a refrigeration loop, and

a second CDU for supplying a second cooling fluid distributed to the rack manifolds on a fluid cooling distribution loop and receiving a return warm/hot fluid from the rack manifolds, the second CDU including an enhanced cooling unit to receive a first portion of a first distribution portion of the second cooling fluid and further cool the first portion of the first distribution portion of the second cooling fluid via a cold fluid distribution loop, and

an air cooling unit to receive a second distribution portion of the first cooling fluid and a second distribution portion of the second cooling fluid and to cool the electronics rack with an airflow using the second distribution portion of the first cooling fluid and the second distribution portion of the second cooling fluid.

According to still another aspect of the present application, there is provided a data center system, which may include:

a plurality of independently deployable portions, comprising:

a housing portion comprising: (i) an electronics rack housing a plurality of server chassis, each server chassis corresponding to one or more servers, wherein the electronics rack includes a rack manifold to receive cooling fluid and return warm/hot fluid from the electronics rack, and (ii) an air cooling unit to cool the electronics rack,

a secondary distribution portion comprising: (i) a fluid cooling distribution loop for delivering the cooling fluid to the electronics racks, (ii) a system return distribution loop for delivering a return warm/hot fluid from the electronics racks, and (iii) an air cooling distribution loop for delivering a distributed portion of the cooling fluid to the air cooling unit, and

an enhanced cooling portion comprising: (i) a first Coolant Distribution Unit (CDU) for supplying a first portion of the cooling fluid to the rack manifold, receiving a first portion of the warm/hot fluid from the rack manifold, and supplying another distributed portion of the cooling fluid to the air cooling unit via the air cooling distribution loop, and (ii) a second CDU for supplying a second portion of the cooling fluid to the rack manifold via the fluid cooling distribution loop, receiving a second portion of the warm/hot fluid via the system return distribution loop, and supplying the distributed portion of the cooling fluid to the air cooling unit via the air cooling distribution loop.

Drawings

Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

FIG. 1 is a block diagram illustrating an example of a data center facility, according to one embodiment.

FIG. 2 is a schematic diagram illustrating an example of a cooling system according to one embodiment.

FIG. 3 is a schematic diagram illustrating another example of a cooling system according to one embodiment.

FIG. 4 is a schematic diagram illustrating an example of a cooling system having an alternative Coolant Distribution Unit (CDU) according to one embodiment.

FIG. 5 is a schematic diagram illustrating yet another example of a cooling system according to one embodiment.

FIG. 6 is a schematic diagram illustrating an example of a cooling system with different data center portions, according to one embodiment.

Detailed Description

Various embodiments and aspects of the inventions will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present inventions.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

According to some embodiments, a cooling system with innovative cooling units for a data center having a plurality of high density racks is provided. The cooling system may be used in a next generation cooling architecture for electronic racks of IT components that are cooled by a mixture of air cooling and liquid cooling, as previously described. The cooling system also meets the requirements of the dynamic configuration of the rack or IT equipment by providing different liquid/fluid cooling and air cooling configurations. Embodiments of the present invention provide a flexible solution that is highly desirable in the modern data center industry and is an important trend. In addition, embodiments of the present invention address problems present in conventional cooling systems, such as dynamic rack configurations with hybrid (liquid-air) cooling technology, dynamic changes in power density and IT equipment, flexibility and other requirements, dynamic changes in operating conditions due to performance considerations. Embodiments of the present invention can also provide a large temperature range for the liquid cooling circuit, as for certain electronic components (e.g., CPU, ASIC, or GPU), better energy sufficiency or performance enhancement may require lower temperatures while maintaining the entire cooling system at a lower Total Cost of Ownership (TCO). In addition, embodiments of the present invention provide a powerful dwelling for different types of scenarios, e.g., from a component level to a system-wide level.

In some embodiments, the cooling systems disclosed herein are used in data center IT equipment. Electronics racks of IT components are cooled using a mixture of both cooling water and cooling air. The cooling system provides cooling fluid to the liquid cooling device and the air cooling device through the use of one or more cooling distribution units. An enhanced cooling circuit is also used in a system for a liquid/fluid cooling circuit. It is used to provide an enhanced thermal environment for the liquid cooled components when desired. The method may be understood as designing and operating a system to provide the most desirable enhancements typically used for most types of IT equipment or clusters. This is to increase the flexibility and flexibility of the system design. The enhanced cooling unit is connected to a refrigeration cycle loop or similar loop that provides similar functionality over a period of time. In one embodiment, the enhanced cooling unit may also be embedded in the cooling distribution unit. Also disclosed herein is a system-level architecture that includes a secondary distribution section and a primary loop section for use between IT and cooling units, and an enhanced cooling section.

According to one aspect, a cooling system for electronics racks of a data center is described. The system includes a Coolant Distribution Unit (CDU) coupled to a rack manifold of the electronics rack by a fluid cooling circuit. The CDU supplies cooling fluid that is distributed to the rack manifolds and receives warm/hot fluid back from the rack manifolds. The system also includes an augmented cooling unit to receive the first portion of the first distributed portion of cooling fluid and further cool the first portion of the first distributed portion of cooling fluid to a temperature that is lower than a temperature of the cooling fluid supplied through the refrigeration circuit. The system also includes a first external cooling unit connected to the CDU by a cooling fluid circuit to supply cooling fluid to the CDU. The system also includes an air cooling unit to receive the second distribution portion of the cooling fluid and to cool hot exhaust air for the electronics rack using the second distribution portion of the cooling fluid.

In one embodiment, the system includes a first three-way valve to receive the supply of cooling fluid and distribute the supply of cooling fluid to the first distribution portion and the second distribution portion, and a second three-way valve to receive the first distribution portion of cooling fluid and deliver a first portion of the first distribution portion of cooling fluid to the augmented cooling unit and a second portion of the first distribution portion of cooling fluid to the rack manifold on the fluid distribution circuit. The enhanced cooling unit may supply a further cooled first portion of the first distributed portion of cooling fluid to the fluid cooling circuit. The system also includes a condensing unit connected to the enhanced cooling unit by a refrigeration circuit. An augmented cooling unit and first and second three-way valves are included in the CDU. The CDU may also include a heat exchanger to transfer heat between the returned warm/hot fluid and the supplied cooling fluid, and a liquid pump to deliver the supplied cooling fluid into the first three-way valve. The CDU also includes a plurality of inlets and outlets coupled to the augmented cooling unit, the second three-way valve, the heat exchanger, and the augmented cooling unit.

According to another aspect, a cooling system for electronics racks of a data center is described. The system includes a plurality of Coolant Distribution Units (CDUs) coupled to rack manifolds of the electronics racks. The CDU includes a first CDU to supply a first cooling fluid that is distributed to the rack manifold and to receive a return warm/hot fluid from the rack manifold. The first CDU includes an enhanced cooling unit to receive a first portion of the first distributed portion of the first cooling fluid and further cool the first portion of the first distributed portion of the first cooling fluid via a refrigeration loop. The CDU also includes a second CDU for supplying a second cooling fluid distributed to the rack manifold by a fluid cooling distribution loop and receiving a return warm/hot fluid from the rack manifold. The second CDU includes an enhanced cooling unit to receive the first portion of the first distributed portion of the second cooling fluid and further cool the first portion of the first distributed portion of the second cooling fluid via the cold fluid distribution loop. The system also includes an air cooling unit to receive the second distribution portion of the first cooling fluid and the second distribution portion of the second cooling fluid and to cool the electronics rack using the second distribution portion of the first cooling fluid and the second distribution portion of the second cooling fluid. In one embodiment, there may be a plurality of air cooling units receiving cooling fluid sources from a plurality of second distribution portions of cooling fluid from a plurality of CDUs.

In one embodiment, the system further comprises a first external cooling unit connected to the first CDU by a cooling fluid circuit to supply a first cooling fluid to the first CDU. The first external cooling unit may also be connected to the second CDU by a cooling fluid distribution loop to supply a second cooling fluid to the second CDU. The system may also include a second external cooling unit connected to the augmented cooling unit of the first CDU by a refrigeration circuit. The second external cooling unit may also be connected to the enhanced cooling unit of the second CDU by a cooling fluid distribution loop.

In one embodiment, the first CDU further comprises a first three-way valve to receive the supplied first cooling fluid and distribute the supplied first cooling fluid into first and second distribution portions of the first cooling fluid. The first CDU can further include a second three-way valve to receive the first split portion of the first cooling fluid and deliver a first portion of the first split portion of the first cooling fluid to the augmented cooling unit of the first CDU and a second portion of the first split portion of the first cooling fluid to the rack manifold via the fluid cooling distribution loop.

In one embodiment, the second CDU further comprises a first three-way valve to receive the supplied second cooling fluid and distribute the supplied second cooling fluid into a first distribution portion and a second distribution portion of the second cooling fluid. The second CDU can further include a second three-way valve to receive the first split portion of the second cooling fluid and deliver a first portion of the first split portion of the second cooling fluid to the augmented cooling unit of the second CDU and deliver a second portion of the first split portion of the second cooling fluid to the rack manifold on the fluid cooling distribution loop.

According to yet another aspect, a data center system is described. The system includes an independently deployable portion including a frame portion having: (i) an electronics rack housing a plurality of server chassis, each server chassis corresponding to one or more servers, wherein the electronics rack includes a rack manifold for receiving cooling fluid and returning warm/hot fluid from the electronics rack, and (ii) an air cooling unit for cooling an airflow for the electronics rack. The system also includes a second distribution section, the second distribution section including: (i) a fluid cooling distribution circuit for delivering cooling fluid to the electronics racks; (ii) a system return distribution loop for conveying return warm/hot fluid from the electronics rack and the air cooling unit; and (iii) an air-cooling distribution circuit for delivering a distributed portion of the cooling fluid to the air-cooling unit. The system further includes an enhanced cooling portion comprising: (i) a first Coolant Distribution Unit (CDU) for supplying a first portion of the cooling fluid to the rack manifold, receiving a first portion of warm/hot fluid from the rack manifold, and supplying another distributed portion of the cooling fluid to the air cooling unit through an air cooling distribution loop and receiving the warm/hot fluid through a system return distribution loop; and (ii) a second CDU that supplies a second portion of the cooling fluid to the rack manifold via the fluid cooling distribution loop to receive a second portion of the warm/hot fluid via the system return distribution loop, and supplies the distributed portion of the cooling fluid to the air cooling unit via the air cooling distribution loop and receives the warm/hot fluid via the system return distribution loop.

FIG. 1 is a block diagram illustrating an example of a data center or data center unit according to one embodiment. In this example, fig. 1 shows a top view of at least a portion of a data center. Referring to FIG. 1, according to one embodiment, a data center system 100 includes one or more rows 102 of electronic racks (IT) of IT components, equipment, or tools 101, such as computer servers or computing nodes that provide data services to various clients over a network (e.g., the Internet). In this embodiment, each row includes an array of electronics racks, such as electronics racks 110A-110N. However, more or fewer rows of electronics racks may be implemented. Typically, the rows 101-102 are arranged in parallel with their front ends facing each other and their rear ends facing away from each other, thereby forming a channel 103 between the rows to allow management personnel to walk through. However, other configurations or arrangements may also be applied. For example, two rows of electronics racks may face each other back-to-back without forming an aisle between them, while their front ends face back to back with respect to each other. The rear end of the electronics rack may be coupled to a room cooling liquid manifold. The present methods set forth in this disclosure may be implemented in different configurations, including configurations not discussed herein.

In one embodiment, each of the electronics racks (e.g., electronics racks 110A-110N) includes a housing to house a plurality of IT components arranged in a stack for operation therein. The electronics rack may include a cooling liquid manifold, a plurality of server bays (e.g., standard racks or chassis configured with the same or similar form factors), and a plurality of server chassis (also referred to as blade servers or server racks) that can be inserted into and removed from the bays. Each server chassis represents a computing node having one or more processors, memory, and/or permanent storage (e.g., hard disks), where the computing node may include one or more servers operating therein. At least one of the processors is attached to a liquid cold plate (also referred to as a cold plate assembly) to receive a cooling liquid. Additionally, one or more optional cooling fans are associated with the server chassis to provide air cooling to the computing nodes housed therein. Note that cooling system 120 may be coupled to multiple data center systems, such as data center system 100. There may be multiple cooling systems coupled to a data center or cluster.

In one embodiment, the cooling system 120 includes an external liquid loop connected to a cooling tower or dry chiller external to the building/containment vessel. The cooling system 120 may include, but is not limited to, evaporative cooling, free air, rejection of large thermal masses, and chiller systems of waste heat recovery design. The cooling system 120 may include or be coupled to a source of cooling liquid that provides cooling liquid.

In one embodiment, each server chassis is modularly coupled to a cooling liquid manifold such that the server chassis may be removed from the electronics rack without affecting the operation of the remaining server chassis on the electronics rack and the cooling liquid manifold. In another embodiment, each server chassis is coupled to the cooling liquid manifold by a quick-release coupling member having a server liquid outlet connector and a server liquid inlet connector coupled to a flexible hose to distribute cooling liquid to the processors. The server liquid inlet connector will receive heat removal liquid from a heat removal liquid manifold mounted on the rear end of the electronics rack through the rack liquid inlet connector. The server liquid outlet connectors are used to emit warmer or hotter liquid carried from the processors exchanged to the cooling liquid manifold via the rack liquid outlet connectors back to the Coolant Distribution Unit (CDU) within the electronics rack.

In one embodiment, a cooling liquid manifold is disposed on the rear end of each electronics rack and is coupled to a liquid supply line 132 (also referred to as a room supply manifold) to receive cooling liquid from the cooling system 120. The cooling liquid is distributed through a liquid distribution circuit attached to a cold plate member on which a processor is mounted to remove cooling from the processor. The cold plate is configured to resemble a heat sink having a liquid distribution tube connected or embedded therein. The resulting warmer or warmer liquid carrying the heat exchanged from the processors is transferred back to the cooling system 120 via a liquid return line 131 (also referred to as a room return manifold).

The liquid supply/return lines 131-132 are referred to as data center or room liquid supply/return lines (e.g., bulk liquid supply/return lines), and the liquid supply/return lines 131-132 supply cooling liquid to all of the electronics racks of the rows 101-102. The liquid supply line 132 and the liquid return line 131 are coupled to the heat exchangers of the CDUs located within each of the electronics racks, thereby forming a primary loop. A second loop of the heat exchanger is coupled to each of the server chassis in the electronics rack to deliver the cooling fluid to the cold plate of the processor.

In one embodiment, the data center system 100 further includes an optional airflow delivery system 135 to generate an airflow to pass through the air spaces of the server chassis of the electronics rack to exchange heat generated by the computing nodes (e.g., servers) due to their operation, and to discharge the heat exchanged via the airflow to an external environment or cooling system (e.g., an air-to-liquid heat exchanger) external to the enclosure/room/building to reduce the temperature of the airflow. For example, the air supply system 135 generates a cool/cool air flow to circulate from the aisles 103 through the electronics racks 110A-110N to carry away the exchanged heat.

The cold airflow enters the electronics rack through the front end of the electronics rack and the warm/hot airflow exits the electronics rack from the rear end of the electronics rack. Warm/hot air with exchanged heat is exhausted from the room/building or cooled using a separate cooling system such as a liquid-to-air heat exchanger. Thus, the cooling system is a hybrid liquid-air cooling system, wherein a portion of the heat generated by the processor is removed by the cooling liquid through, for example, a corresponding cold plate, while the remainder of the heat generated by the processor (or other electronic device or processing means) is removed by airflow cooling.

FIG. 2 is a schematic diagram illustrating an example of a cooling system according to one embodiment. Referring to fig. 2, the electronics rack 201 is coupled to the cooling system 200 to receive cooling fluid from the cooling system 200. The electronics rack 201 is representative of any of the electronics racks shown in fig. 1, e.g., the electronics racks 110A-110N. As shown, the electronics rack 201 may include a server chassis 211 and a rack (or cooling liquid) manifold 213. The server chassis 211 may include a liquid cold plate 212 (also referred to as a cold plate assembly) coupled to the manifold 213 to receive cooling fluid from the manifold 213 and return warm/hot fluid to the manifold 213. The cold plate 212 (the IT component on which the server chassis 211 is mounted) may remove heat generated by the IT component using cooling fluid received from the manifold 213.

With continued reference to fig. 2, the cooling system 200 may include, but is not limited to, an enhanced cooling unit 221, a condensing unit (or condenser, or external cooling unit) 222, a CDU 223, an air cooling unit 224, and an external cooling unit 225. The CDU 223 supplies cooling fluid to the racks 201 by supplying fluid to the rack manifold 213 via the three-way valves 242 and 243, wherein the rack manifold 213 delivers the fluid to the cold plate 212. The CDU 223 may also supply cooling fluid to the air cooling unit 224 (liquid-to-air heat exchanger) such that the air cooling unit 224 may use the supplied cooling fluid to cool the recirculated air for the racks 201. For example, in one embodiment, the coolant flows through the air cooling unit 224 and exchanges heat with warm/hot air exhausted from the rack 201 (server chassis 211) and is circulated into the air cooling unit 224. Due to the heat exchange within the air cooling unit, the hot air is cooled and cool/cold air may be generated and supplied to the racks 201 to cool the racks 201, while warm/hot fluid is discharged from the air cooling unit 224 to be mixed with the warm/hot fluid generated by the manifold 213. In one embodiment, the air cooling unit 224 may be integrated as part of the rack 201. It is to be noted that the cooling air flow is not shown in the figure. Cooling air is supplied to the racks 201, and then hot exhaust air enters the air cooling unit 224. The air cooling unit 224 supplies cooling air back to the racks 201.

In some embodiments, the three-way valve 242 is configured to distribute different amounts of fluid to the rack manifold 213 and the air cooling unit 224. That is, valve 242 may be used to regulate the delivery of cooling capacity to different cooling sections, such as an air cooling section and a liquid cooling section. For example, on a liquid/fluid distribution circuit attached or coupled to the rack manifold 213, a three-way valve 243 is used to direct all or a portion of the cooling fluid to the augmented cooling unit 221 (e.g., a liquid-to-liquid heat exchanger). The enhanced cooling unit 221 may be used to enhance the cooling capacity on the liquid distribution circuit or the entire cooling circuit. Within the augmented cooling unit 221, the cooling fluid may be further cooled to a much lower temperature as the augmented cooling unit 221 is cooled by a refrigeration circuit connected or coupled to the condensing unit 222. The cold/cold fluid delivered to the enhanced cooling unit 221 from the condensing unit 222 may then be delivered to the fluid cooling circuit 250 via the two-way valve 244 for delivery to the rack manifold 213 (the valve 244 preferably has a unit in the system 200, although it may be optional in some embodiments, but may also provide additional features for enhanced system control). Note that the condensing unit 222 may be any kind of commercially available or custom-made condensing unit or (condenser). Thus, the details of the condensing unit 222 will not be described here. In one embodiment, the cool/cold fluid supplied by the augmented cooling unit 221 may be mixed with the cold fluid supplied by the CDU 223 on the fluid cooling circuit 250 and the fluid mixture delivered to the manifold 213.

In one embodiment, the CDU 223 may include a heat exchanger 231 and a liquid pump 232. The CDU 223 includes a pump controller (not shown), as well as some other components, such as a reservoir, a power supply, monitoring sensors, and the like. Note that CDU 223 can be any variety of commercially available or custom CDUs. Thus, the details of the CDU 223 will not be described herein. The heat exchanger 231 may be connected to the external cooling unit 225 via a cooling fluid circuit. The heat exchanger 231 may receive cooling fluid from the external cooling unit 225 via a liquid pump 241. In one embodiment, heat from the warm fluid delivered by the manifold 213 (and by the air cooling unit 224) may be transferred to the cold fluid while passing through the heat exchanger 231 before being extracted to the external cooling unit 225. A liquid pump 232 is used to transfer cooling fluid from the external cooling unit 225 onto the liquid distribution circuit. In one embodiment, the heat exchanger 231 may be a liquid-to-liquid heat exchanger. In one embodiment, the external cooling unit 225 may be a cooling tower (open or closed loop), evaporative cooling coil, dry cooler, or the like, external to the building/containment vessel.

Note that FIG. 2 shows only a simplified representation of the cooling system 200 and corresponding critical structures, with only one of each component (units 221 and 225) shown. As previously described, a data center system (data center system 100) may include multiple rows of electronics racks of IT components. Thus, the system 200 may include each of the plurality of units 221-225 for cooling the electronics racks of those rows.

FIG. 3 is a schematic diagram illustrating another example of a cooling system according to one embodiment. In the cooling system 300 of FIG. 3, the enhanced cooling unit 221 is integrated into the CDU 223. A three-way valve 342-343 is also included in the CDU 223 and is used to manipulate the fluid. For example, valve 342 is used in a similar manner as valve 242 by distributing different amounts of fluid to rack manifold 213 and air cooling unit 224. Thus, the valve 342 may be used to regulate the delivery of cooling capacity to different cooling sections, such as an air cooling section and a liquid cooling section. Like valve 243, three-way valve 343 is used to direct the full or partial cooling fluid dispensed by valve 342 to augment cooling unit 221 (as previously described and therefore not described herein). However, in FIG. 3, the CDU 223 includes a plurality of inlet and outlet ports 361 and 367. For example, outlet port 361 connects the augmented cooling unit 221 to the rack manifold 213, and outlet port 362 connects valve 343 to the rack manifold 213. Ports 361-362 may independently supply cool/cold fluid to manifold 213 at different or the same fluid temperatures at the same time, although they may provide fluid in an alternating manner at different or the same fluid temperatures. In some embodiments, the fluids supplied by ports 361 and 362 mix in manifold 213. In one embodiment, inlet port 363 connects manifold 213 to heat exchanger 231 to receive returned warm/hot fluid from manifold 213 and air cooling unit 224. The outlet port 364 and the inlet port 365 connect the cooling circuit between the heat exchanger 231 and the external cooling unit 325A as previously described in fig. 2 with respect to the heat exchanger 231 and the external cooling unit 225. The outlet port 366 and the inlet port 367 connect the refrigeration circuit between the enhanced cooling unit 221 and the external cooling unit 325B. In the cooling system 300, within the enhanced cooling unit 221, the cooling fluid may be further cooled to a lower temperature as the enhanced cooling unit 221 is cooled by a refrigeration circuit connected or coupled to the external cooling unit 325B. In some embodiments, the external cooling unit 325B may be a condensing unit (e.g., the condensing unit 222 of fig. 2, or any other type of cooling unit). In one embodiment, each of the external cooling units 325A-B is the external cooling unit 225 of FIG. 2. Thus, the details of the external cooling units 325A-B will not be described here.

FIG. 4 is a schematic diagram illustrating an example of a cooling system with alternating Cooling Distribution Units (CDUs) according to one embodiment. In fig. 4, cooling system 400 is similar to cooling system 300 of fig. 3. However, the ports 361-362 of FIG. 3 are combined in the CDU 223 based on internal design variations within the CDU 223. That is, the three-way valve 442 may distribute the cool/cold fluid to the air cooling unit 224 and the three-way valve 443 in a similar manner as the valve 342 of fig. 3. However, in cooling system 400, valve 443 is configured to distribute fluid back to augmented cooling unit 221 (similar to valve 343 of fig. 3) and also to port 461 (rather than directly to manifold 213). Thus, the port 461 may carry a fluid mixture of the fluid supplied through the augmented cooling unit 221 and the fluid supplied through the valve 443. The configuration of ports 462 through 467 is similar to ports 362 through 367 of FIG. 3, and thus, the details of ports 462 through 467 will not be described here. In some embodiments, the corresponding design of the rack manifold 213 in fig. 4 is different from the design of the rack manifold 213 in fig. 3. For example, the manifold 213 of FIG. 3 may require an additional inlet port to receive the cooling fluid of the CDU 223.

FIG. 5 is a schematic diagram illustrating yet another example of a cooling system according to one embodiment. In the cooling system 500 of fig. 5, an air-cooled distribution circuit 503, a cooling fluid distribution circuit 504 and a cold fluid distribution circuit 505 are introduced. This will increase the flexibility of the cooling system design. In FIG. 5, multiple CDUs 523A-B may be used to distribute fluid to the fluid cooling distribution loop 501. For example, in the system 500, the coolant distribution loop may include a fluid cooling distribution loop 501, a system return distribution loop 502, and an air cooling distribution loop 503. Loops 501-503 are used between the electronics rack 201 and the CDUs 523A-B. Connection ports are assembled on these distribution circuits and connect cooling units, CDUs and other fluid circuits to these distribution circuits. A cooling fluid distribution loop 504 and a cold fluid distribution loop 505 are used between the CDUs 523A-B and the external cooling units 325A-B. In some embodiments, the fluids used in each of the circuits 501-505 may be different from each other, and thus have different temperatures from each other. However, in other embodiments, the fluid used in some or all of the circuits 501 and 505 may be the same fluid. Note that additional control or check valves may be used on the dispensing circuit not shown in the figures.

Note that in system 500, CDUs 523A-B and external cooling units 325A-B are detachably coupled, and loop 501-505 serves as a reservoir for distributing and receiving fluids. External cooling units 325A-B and CDUs 523A-B may exchange fluids through the pools. Thus, this not only provides flexibility in redundant design, but is also easy to service, while improving infrastructure cost efficiency.

FIG. 6 is a schematic diagram illustrating an example of a cooling system and data center system layout with different portions, according to one embodiment. In fig. 6, the system 500 is divided into five sections, namely a rack section (or IT section) 601, an auxiliary distribution section 602, an enhanced cooling section 603, a main circuit section 604, and a cooling unit section 605. This provides boundaries for different sections or modules in the system 500. This classification of sections (or sections/modules) will simplify the system design and deployment process of system 500, as sections 601 and 605 can be designed and deployed individually and independently, thereby increasing efficiency. This is also important for developing system standardization, as only capacity, or connection methods, and certain key parameters or specifications are needed for the modules to be able to connect to the entire system and function properly. In addition, this would significantly improve reliability, since most of the functionality is to move components enclosed within the CDU (CDU 523A-B). The CDU can be easily replaced and serviced in the system 500 without causing significant interruption to the continuous operation of the system. With the primary loop portion 604, the external cooling unit 325B may be used to provide enhanced cooling capacity to the server chassis 211 in the CDU 523A or the CDU 523B, or both. Similar functionality is provided by portion 602 for different other cooling units in the cooling circuit.

In some embodiments, the secondary distribution section 602 and the primary loop section 604 serve as a source pool, thereby increasing the flexibility of the system. As an example, a refrigeration circuit (between the CDU and the external cooling unit) may be used for different enhancement circuits in the enhanced cooling portion 603 in a multiple CDU configuration. Valves (three-way valves 442 and 443) may be used on the distribution portion 602 and the main circuit portion 604 to control the flow. Fig. 6 also provides a concept of a compact and efficient method for designing a cooling system for a hybrid (liquid-air) cooling rack. For example, the frame portion 601 may be considered a point of delivery (PoD). Thus, the system 500 may be used for computing PoDs, storing PoDs, or heterogeneous high performance computing PoDs, and may use a general enhanced cooling section (enhanced cooling section 603) dedicated to a single PoD or multiple PoDs.

The deployment of each portion may be independent of each other. This means that the respective designs and specifications of each part can be integrated and work properly even though they may be provided by different suppliers.

In the foregoing specification, embodiments of the invention have been described with reference to specific exemplary embodiments thereof. It will, of course, be evident that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.