阅读说明：本技术 数据中心交付点布局和配置 (Data center delivery point layout and configuration ) 是由 高天翼 于 2020-12-16 设计创作，主要内容包括：本文公开了支持数据中心的房间气流、服务器机架和冷却输送子系统的布局、布置和配置,数据中心具有带分离的计算和存储交付点(PoD)的系统体系结构。冷却行布置在容纳PoD的机架的后侧中,以向PoD供应冷却输送子系统,该冷却输送子系统包括向PoD供应冷却流体并将温热流体返回至冷却源的再循环回路。冷却行的模块化冷却单元包括用于冷却离开机架的气流的冷却盘管。供应至用于PoD的冷却输送子系统的冷却液的冷却能力可在反馈回路中基于PoD由冷却盘管冷却后的空气温度来独立地调节和控制,其中该空气温度为诸如来自容纳PoD的冷却行的出口空气温度)。(Disclosed herein are layouts, arrangements, and configurations of room airflows, server racks, and cooling transport subsystems supporting a data center having a system architecture with separate computing and storage point of delivery (PoD). A cooling row is disposed in a rear side of the rack housing the PoD to supply a cooling delivery subsystem to the PoD, the cooling delivery subsystem including a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to the cooling source. The modular cooling units of the cooling rows include cooling coils for cooling the airflow exiting the racks. The cooling capacity of the cooling fluid supplied to the cooling delivery subsystem for the PoD may be independently adjusted and controlled in a feedback loop based on the air temperature of the PoD after it is cooled by the cooling coil, such as the outlet air temperature from the cooling row housing the PoD).)

1. A liquid cooling arrangement for a data center, comprising:

a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply cooling fluid received from a cooling fluid source to a liquid-to-air heat exchanger and return warmed fluid from the liquid-to-air heat exchanger to the cooling fluid source; and

a plurality of cooling coils of the liquid-to-air heat exchanger, each of the cooling coils configured to receive cooling liquid from a respective one of the modular cooling units to remove heat from an airflow of a heat load, wherein a cooling capacity of the cooling liquid supplied to the liquid cooling device is controlled based on a temperature of the airflow after the liquid-to-air heat exchanger.

2. The liquid cooling device according to claim 1, wherein a cooling capacity of the cooling liquid supplied to the liquid cooling device is controlled independently of a cooling capacity of the cooling liquid supplied from the cooling liquid source to a second liquid cooling device.

3. The liquid cooling apparatus of claim 1, wherein the thermal load comprises a point of delivery (PoD) of the data center, wherein the PoD comprises one of a computing PoD, a storage PoD, and a combination of the computing PoD and the storage PoD.

4. The liquid cooling device of claim 1, wherein the plurality of cooling coils are positioned above the source of the thermal load.

5. The liquid cooling device of claim 1, wherein controlling the cooling capacity of the cooling liquid based on the temperature of the airflow after the liquid-to-air heat exchanger comprises: the volume or temperature of the cooling liquid supplied to the liquid cooling device.

6. The liquid cooling device of claim 1, wherein a plurality of cooling coils from the liquid-to-air heat exchanger of a plurality of adjacent ones of the modular cooling units combine to remove heat from an airflow of a plurality of adjacent heat loads.

7. The liquid cooling device of claim 1, wherein the fluid distribution subsystem is further configured to supply a cooling liquid to the source of the thermal load, to remove additional heat from the thermal load using a liquid-to-liquid heat exchanger, and to return warm liquid from the liquid-to-liquid heat exchanger to the cooling liquid source.

8. The liquid cooling device of claim 7, wherein the cooling capacity of the cooling liquid supplied to the liquid cooling device is further controlled based on the temperature of the warm liquid returned from the liquid-to-liquid heat exchanger.

9. A data center system, comprising:

one or more rows of a plurality of electronics racks, each of the electronics racks housing a plurality of electronic components; and

one or more rows of row cooling units, each row of the row cooling units associated with a respective row of electronics racks, wherein each row of cooling units comprises:

a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply cooling fluid received from a cooling fluid source to a liquid-to-air heat exchanger and return warmed fluid from the liquid-to-air heat exchanger to the cooling fluid source; and

a plurality of cooling coils of the liquid-to-air heat exchanger, each of the cooling coils configured to receive cooling liquid from a respective one of the modular cooling units to remove heat from an airflow from one of the electronics racks, wherein a cooling capacity of the cooling liquid supplied to the row cooling units is controlled based on a temperature of the airflow from the row cooling unit after the airflow from the corresponding row of electronics racks transfers heat to the liquid-to-air heat exchanger, and wherein a layout of the system is configured such that the airflow from the row cooling unit flows into an adjacent row of the electronics racks and its associated row cooling unit for the entire area or a portion of the data center in the same direction as an inlet airflow.

10. The system of claim 9, wherein the cooling capacity of the cooling fluid supplied to the row of cooling units is controlled independently of the cooling capacity of the cooling fluid supplied from the cooling fluid source to the second row of cooling units.

11. The system of claim 9, wherein the row cooling unit and the respective row of electronics racks comprise a point of delivery (PoD) of the data center, wherein the PoD comprises one of a compute PoD, a storage PoD, and a combination of the compute PoD and the storage PoD.

12. The system of claim 9, wherein the plurality of cooling coils of the row of cooling units are positioned higher than the electronics racks of the respective row.

13. The system of claim 9, wherein controlling the cooling capacity of the cooling liquid based on the temperature of the airflow from the row cooling unit after the liquid-to-air heat exchanger comprises: the volume or temperature of the cooling liquid supplied to the row of cooling units.

14. The system of claim 9, wherein a plurality of cooling coils of the liquid-to-air heat exchanger from a plurality of adjacent ones of the modular cooling units combine to remove heat from airflow from a plurality of adjacent ones of the electronics racks.

15. The system of claim 9, wherein each of the electronics racks includes a rack manifold having a rack liquid supply line to a liquid-to-liquid heat exchanger and a rack liquid return line from the liquid-to-liquid heat exchanger, and wherein the fluid distribution subsystem is further configured to supply the cooling liquid to the rack liquid supply line of one of the electronics racks to remove additional heat from the electronics rack using the liquid-to-liquid heat exchanger, and to return warm liquid from the liquid-to-liquid heat exchanger to the cooling liquid source through the rack liquid return line.

16. The system of claim 15, wherein the cooling capacity of the cooling liquid supplied to the row cooling units is further controlled based on the temperature of the warm liquid returned from the liquid-to-liquid heat exchanger.

17. The system of claim 9, wherein one of the electronics racks is configured to control airflow in the electronics rack based on a temperature of the airflow in the electronics rack.

18. The system of claim 9, further comprising a plurality of rows of the electronics racks and a corresponding plurality of rows of the row cooling units arranged in the same direction in the data center.

19. The system of claim 9, further comprising a plurality of rows of the electronics rack and a corresponding plurality of rows of the row cooling units, wherein each pair of the plurality of rows of the row cooling units are arranged to face each other.

20. The system of claim 9, further comprising a plurality of rows of the electronics rack and a corresponding plurality of rows of the row cooling units, wherein each of the row cooling units comprises an upper row cooling unit and a lower row cooling unit, and wherein one or more of the upper row cooling units of the plurality of rows of row cooling units are arranged in one direction and one or more of the lower row cooling units of the plurality of rows of row cooling units are arranged in an opposite direction.

Technical Field

Embodiments of the invention generally relate to data center configurations. More particularly, embodiments of the invention relate to the design and layout of thermal management and facility infrastructure to support the deployment and operation of electronic racks in data centers having different point of delivery (PoD) configurations of computing and storage.

Background

Computing clusters and data centers serve today's information-efficient computing, storage, and other Information Technology (IT) needs. With the rapid development of technology and the increasing demands on performance, new applications and workloads, such as machine learning, video streaming, high performance computing, etc., separate new computing architectures, such as computing and storage resources or PoDs, have evolved significantly. Such computing architectures with separate compute and store PoDs are of great importance to the design and layout of the data center facilities and hardware infrastructure required to support them. For example, hardware configurations such as rack layouts, system capacities, power ratings, cooling methods, networking configurations, and the like may be different. As a result, traditional data center infrastructures may not be able to efficiently and cost-effectively support the deployment and operation of new computing architectures. The data center infrastructure may also not be readily adaptable to support computing architectures with different compute and storage separation configurations (e.g., different sizes and ratios of compute and storage PoDs).

Heat dissipation or thermal management is an important consideration in data center design. As the number of high performance electronic components, such as high performance processors and memory devices, in computing and storage containers has steadily increased, the amount of heat generated and dissipated during ordinary operation of the PoD has also increased. The reliability of the electronic components used within a data center decreases if the ambient temperature at which the data center is allowed to operate increases over time. Maintaining a proper thermal environment is critical to the proper operation of these pods in a data center, as well as their performance and lifetime.

Thermal management of high performance PoD with increasingly higher power densities in data centers may use a combination of liquid and air cooling systems. However, it is difficult to flexibly adjust and scale the cooling infrastructure of a data center to meet the changing hardware configuration of separate compute and storage pous. IT is desirable to upgrade and update IT equipment and deploy new computing architectures with separate compute and storage pods at a much faster rate than the lifecycle of the facilities that house the data center, which makes IT challenging to exploit the full potential of the new computing architectures.

Disclosure of Invention

According to an aspect of the present application, there is provided a liquid cooling apparatus of a data center, which may include:

a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply cooling fluid received from a cooling fluid source to a liquid-to-air heat exchanger and return warmed fluid from the liquid-to-air heat exchanger to the cooling fluid source; and

a plurality of cooling coils of the liquid-to-air heat exchanger, each of the cooling coils configured to receive cooling liquid from a respective one of the modular cooling units to remove heat from an airflow of a heat load, wherein a cooling capacity of the cooling liquid supplied to the liquid cooling device is controlled based on a temperature of the airflow after the liquid-to-air heat exchanger.

According to another aspect of the present application, there is provided a data center system, which may include:

one or more rows of a plurality of electronics racks, each of the electronics racks housing a plurality of electronic components; and

one or more rows of row cooling units, each row of the row cooling units associated with a respective row of electronics racks, wherein each row of cooling units comprises:

a plurality of interconnected modular cooling units, each of the modular cooling units having a fluid distribution subsystem configured to supply cooling fluid received from a cooling fluid source to a liquid-to-air heat exchanger and return warmed fluid from the liquid-to-air heat exchanger to the cooling fluid source; and

a plurality of cooling coils of the liquid-to-air heat exchanger, each of the cooling coils configured to receive cooling liquid from a respective one of the modular cooling units to remove heat from an airflow from one of the electronics racks, wherein a cooling capacity of the cooling liquid supplied to the row cooling units is controlled based on a temperature of the airflow from the row cooling unit after the airflow from the corresponding row of electronics racks transfers heat to the liquid-to-air heat exchanger, and wherein a layout of the system is configured such that the airflow from the row cooling unit flows into an adjacent row of the electronics racks and its associated row cooling unit for the entire area or a portion of the data center in the same direction as an inlet airflow.

Drawings

Embodiments of the invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

Fig. 1 is a block diagram illustrating an example of a data center facility.

Fig. 2 is a block diagram illustrating an example of an electronics rack, according to one embodiment.

FIG. 3 is a block diagram illustrating an example of a cold plate configuration according to one embodiment.

FIG. 4 is a top layout view of cooling rows and associated rack rows showing the cooling rows arranged in the same direction throughout the data center, according to one embodiment.

FIG. 5 is a top layout view of cooling rows and associated racks in a data center arranged such that each pair of cooling rows face each other, according to one embodiment.

FIG. 6 is a top layout view of a cooling row and associated racks in a data center arranged such that upper and lower portions of the cooling row face in opposite directions, according to one embodiment.

FIG. 7 is a side or rear view of a cooling row showing fluid connections to each cooling coil for the liquid supply and return circuits, each cooling coil being used to individually cool each rack, according to one embodiment.

Fig. 8 is a side or rear view of a cooling row showing the supply and return circuits for liquid to each cooling coil group that is shared between two adjacent racks according to one embodiment.

Fig. 9 is a top layout view of a cooling row and associated racks configured such that the temperature of the air flow from the cooling row is used to independently control the cooling capacity of the cooling liquid supplied to the cooling row, according to one embodiment.

Fig. 10 is a top view of a cooling row connected to a facility infrastructure to form a cooling liquid supply loop and a warm liquid return loop of a liquid cooling system before attachment of a rack, according to an embodiment.

Detailed Description

Various embodiments and aspects of the inventions will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the invention. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments of the present inventions.

Reference in the specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

Disclosed herein are layouts, arrangements, and configurations of room airflows, server racks, and cooling transport subsystems supporting a data center having a system architecture with separate computing and storage point of delivery (PoD). The compute or storage PoD, which may be referred to as an IT PoD, and the cooling delivery subsystem servicing the IT PoD are arranged such that thermal management of the IT PoD may be flexibly supported by the infrastructure of the data center housing the IT PoD. Since the number, ratio, and configuration of compute and store pouds may be different to serve different IT applications, the compute and store pouds may have different power loads, power densities, etc., requiring different power and cooling delivery subsystems.

In one embodiment, the cooling delivery subsystem may include liquid cooling, or a combination of liquid cooling and air cooling, to flexibly support the thermal management functionality of the IT PoD. The cooling capacity of the liquid supplied to the cooling delivery subsystem of the PoD may be independently adjusted and controlled in a feedback loop based on the air temperature of the PoD after it is cooled by the cooling liquid, such as the inlet and/or outlet air temperature from the rack housing the PoD. The change in cooling capacity may include a change in temperature or volume of the cooling fluid. Advantageously, compute and store PoDs can have different numbers of racks with different power loads and still have independent temperature control, thereby enabling IT PoDs with different power densities to be managed with a consistent thermal management solution.

In one embodiment, cooling rows are arranged in the rear side of each row of racks housing the PoD to supply the PoD with a cooling transport subsystem. The cooling delivery subsystem may include a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to the cooling source. The design of the cooling rows eliminates the traditional hot and cold aisle layout or any containment structure of the data center by effectively treating the entire space of the data center as a cold aisle. The cooling row includes modular cooling units arranged in rows to supply liquid cooling circuits to cool the row frame. The main unit in each cooling unit may be a cooling coil for cooling the airflow exiting one or more racks. The cooling coil is supplied with a cooling fluid and is used as a heat exchanger to cool the temperature of the air flow heated by the electronic components housed within the rack.

In one embodiment, a liquid-to-liquid cooling coil (e.g., a rack-level liquid cooling unit, a rack manifold, a submersion cooling tank, etc.) may be connected to the cooling distribution subsystem in addition to or in place of the liquid-to-gas cooling coil. Different types of IT racks (e.g., computer racks, storage racks, networking equipment racks) may be disposed anywhere in the PoD. Any IT space of the data center may be configured as a compute PoD, a store PoD, or a heterogeneous compute and store PoD. The cooling row design may be flexibly adapted to meet the cooling requirements of various layouts, arrangements, thermal loads, and configurations of the PoD.

In one embodiment, the cooling rows disposed in the rear side of the rack rows may be disposed in the same direction throughout the data center room. After being cooled by the cooling coils of the cooling rows, the air from the racks flows in substantially the same direction. The cooling capacity of the cooling liquid supplied to each row may be independently controlled based on the temperature of the air flow from the cooling row. Thus, the air temperature from different cooling rows may be controlled to be relatively uniform regardless of the power load of the racks associated with the cooling rows. Each location in the room may be considered identical in terms of thermal management due to the uniformly directed airflow from the racks and the uniform temperature of the airflow.

In one embodiment, even when the cooling rows disposed in the rear side of the rack row face each other such that air from each of the pair of cooling rows flows toward each other, the air temperatures from the two cooling rows can be independently controlled to be relatively uniform. This layout is similar to the alternating hot aisle cold aisle arrangement of a conventional PoD, but does not experience a temperature gradient between the hot and cold aisles of a conventional PoD due to the independent control of the air temperature of the cooling rows.

In one embodiment, the cooling row and associated racks may be divided into upper and lower portions. The upper portion of all rows may be in one direction and the lower portion of all rows may be in the opposite direction. Thus, air from both parts of the row may flow in opposite directions. This arrangement may also enable itself to be thermally managed by independent temperature control of the rows, removing any temperature gradients between locations within the room, or keeping the room temperature uniform.

In one embodiment, one of the critical components of the cooling row includes a modular cooling unit with integrated fluid distribution piping that may be interconnected to form a cooling distribution subsystem that supplies cooling fluid to cooling coils to carry away heat generated by electronic components housed within the racks associated with the cooling row. The components of the modular cooling unit may be connected to a cooling fluid supply and return loop of the data center facility to distribute cooling fluid to the electronic components of the racks and return heated fluid for heat dissipation. The cooling capacity of the cooling rows may be controlled independently. The network of fluid distribution pipes integrated into the cooling unit enables the configuration of the cooling distribution subsystem independent of the fixed infrastructure of the facility, thereby providing scalability, serviceability, ease of maintenance, while improving the efficiency, flexibility, availability, and reliability of the liquid cooling system.

Fig. 1 is a block diagram illustrating an example of a data center or data center unit. In this example, fig. 1 shows a top view of at least a portion of a data center. Referring to FIG. 1, according to one embodiment, a data center system 100 includes one or more rows 102 of electronic racks of Information Technology (IT) components, equipment, or tools 101, 102, such as computer servers, computing nodes, or storage nodes that provide data services to various clients over a network (e.g., the Internet). In this embodiment, each row includes an array of electronics racks, such as electronics racks 110A-110N, arranged in row 101 and row 102. However, more or fewer rows of electronics racks may be implemented. Each row may be part of a compute PoD, a store PoD, or a heterogeneous compute and store PoD. Typically, rows 101-102 are aligned in parallel with the front ends facing toward each other (or away from each other; e.g., if multiple systems as shown in FIG. 1 are deployed, two of the racks can be understood to be facing away from each other and the back ends facing away from each other, depending on how the racks are grouped, thereby forming a channel 103 therebetween to allow management personnel to walk and perform services therein.

In one embodiment, each of the electronics racks (e.g., electronics racks 110A-110N) includes a housing to house a plurality of electronics racks of IT components operating therein. The electronics rack may include a thermal fluid manifold, a plurality of server slots, and a plurality of blade servers that can be inserted into and removed from the blade servers or the server slots. Each blade server represents a compute node having one or more processors, memory, and/or persistent storage (e.g., hard disks). At least one of the processors may be connected to a liquid cold plate (also referred to as a cold plate assembly) to receive a cooling liquid. In addition, one or more optional cooling fans are associated with the blade servers to provide air cooling to the compute nodes housed therein. Note that heat dissipation system 120 may be coupled to multiple data center systems, such as data center system 100.

In one embodiment, the heat dissipation system 120 includes an external liquid circuit connected to a cooling tower or dry cooler outside the building/containment vessel. The heat dissipation system 120 may include, but is not limited to, evaporative cooling, free air, rejection of large thermal mass, waste heat recovery designs, or chiller systems with active refrigeration cycles. The heat dissipation system 120 may include or be coupled to a source of cooling fluid that provides the cooling fluid.

In one embodiment, each blade server is modularly coupled to the heat sink fluid manifold such that the blade server can be removed from the electronics rack without affecting the operation of the remaining blade servers on the electronics rack and the heat sink fluid manifold. In another embodiment, each blade server is coupled to a heat sink liquid manifold (also referred to as a coolant manifold) by a quick release coupling assembly having a first liquid inlet connector and a first liquid outlet connector coupled to flexible hoses to distribute the heat sink liquid to the processors. The first fluid inlet connector is adapted to receive a heat dissipating fluid through the second fluid inlet connector from a heat dissipating fluid manifold mounted at a rear end of the electronics rack. The first liquid outlet connector is used to emit warmer or hotter liquid carrying heat exchanged from the processor to the heat sink liquid manifold via the second liquid outlet connector back to a Coolant Distribution Unit (CDU) within the electronics rack.

In one embodiment, a heat sink fluid manifold is disposed at the rear end of each electronics rack and is coupled to the fluid supply line 131 to receive heat sink fluid (also referred to as coolant) from the heat sink system 120. The heat sink liquid is distributed through a liquid distribution loop attached to a cold plate assembly on which the processor is mounted to remove heat from the processor. The cold plate is configured similar to a heat sink with liquid distribution tubes attached or embedded therein. The resulting warmer or warmer liquid carrying the heat exchanged from the processor is transferred back to heat dissipation system 120 via liquid return line 132.

The liquid supply/return lines 131-132, referred to as data center or room liquid supply/return lines (e.g., global liquid supply/return lines), supply the heat sink liquid to all of the electronics racks of rows 101-102. As will be discussed, in one embodiment, the liquid supply/return lines 131 and 132 may be integrated within modular cooling units coupled to the electronics racks 110A-110N. Electronics racks 110A-110N may be assembled to form rows 101 or 102 by interconnecting the liquid supply/return lines 131 and 132 of adjacent modular cooling units to form cooling rows and connecting the liquid supply/return lines 131 and 132 of the modular cooling units of a cooling row to the cooling fluid manifolds of electronics racks 110A-110N to complete the cooling fluid supply and return loops with the supply/return lines of the data center facility. The liquid supply line 131 and liquid return line 132 are coupled to the heat exchangers of the CDUs located within each of the electronics racks, forming a primary loop. A secondary loop of the heat exchanger is coupled to each blade server in the electronics rack to deliver cooling fluid to the cold plate of the processor.

In one embodiment, the data center system 100 further includes an optional airflow delivery system 135 to generate an airflow to pass through the air spaces of the blade servers of the electronics rack to exchange heat generated by the compute nodes (e.g., servers) due to their operation and to discharge the heat exchanged via the airflow to the external environment 108 outside the house/room/building. For example, the air supply system 135 generates a cool/cool air flow to circulate from the aisles 103 through the electronics racks 110A-110N to carry away the exchanged heat. The cold airflow enters the electronics rack through the front end of the electronics rack, while the warm/hot airflow exits the electronics rack from the rear end of the electronics rack. Warm/hot air with exchanged heat is exhausted from the room/building. Thus, the cooling system is a mixed liquid-gas cooling system, wherein a portion of the heat generated by the processor is removed by the cooling liquid through the respective cold plate, while the remaining portion of the heat generated by other electronic equipment or processing devices is removed by gas flow cooling.

In one embodiment, the electronics rack 102 may have its own internal airflow delivery system, such as a fan, to cause airflow to travel through the air space of the electronics rack 102 to exchange heat generated by the computing or storage PoD with the cooling coils of the modular cooling units of the cooling row. The cooling fluid of fluid supply/return lines 131 and 132 is supplied to the cooling coils to cool the air stream. After heat exchange with the cooling coils, the cooling airflow of the electronics rack 102 may flow out of the cooling rows and be used in a feedback loop to adjust and control the cooling capacity of the cooling fluid supplied to the cooling rows of the electronics rack 102.

Fig. 2 is a block diagram illustrating an electronics rack, according to one embodiment. The electronics rack 200 may represent any of the electronics racks shown in fig. 1, such as the electronics racks 110A-110N. Referring to FIG. 2, according to one embodiment, an electronics rack 200 includes, but is not limited to, a CDU 201, an optional Rack Management Unit (RMU)202, and one or more blade servers 203A-203E (collectively referred to as blade servers 203). Blade servers 203 may be inserted into the array of server slots from the front end 204 or the back end 205 of the electronics rack 200, respectively. Note that although five blade servers 203A-203E are shown here, more or fewer blade servers may be maintained within the electronics rack 200. It should also be noted that the particular locations of the CDU 201, RMU 702, and blade server 203 are shown for illustration purposes only; other arrangements or configurations of the CDU 201, RMU 202, and blade server 203 may also be implemented. In one embodiment, the electronics rack 200 may be open to the environment or partially housed by the rack receptacle, so long as the cooling fan can generate an airflow from the front end to the rear end. The electronics rack 200 may be part of a compute PoD, a storage PoD, or a heterogeneous compute and storage PoD.

In addition, for at least some of the blade servers 203, an optional fan module (not shown) is associated with the blade server. Each of the fan modules includes one or more cooling fans. The fan module may be mounted on the back end of the blade server 203 or electronics rack to create an airflow that flows out of the front end 204, travels through the air space of the blade server 203, and exits at the back end 205 of the electronics rack 200.

In one embodiment, the CDU 201 primarily includes a heat exchanger 211, a liquid pump 212, and a pump controller (not shown), as well as some other components, such as a reservoir, a power supply, monitoring sensors, and the like. The heat exchanger 211 may be a liquid-to-liquid heat exchanger or a cooling coil. The heat exchanger 211 includes a first loop 223 having inlet and outlet ports with a first pair of fluid connectors coupled to the fluid supply/return lines 131 and 132, the fluid supply/return lines 131 and 132 being integrated into the electronics rack 200 to form a primary loop. The integrated liquid supply/return lines 131 and 132 may be disposed at the bottom of the rear end 205 of the electronics rack 200. The liquid supply/return lines 131 and 132, also referred to as room liquid supply/return lines, are connected to the heat dissipation system 120 via the supply/return lines 131 and 132 of the cooling row's modular cooling units, as will be explained below. In addition, the heat exchanger 211 also includes a secondary loop having two ports with a second pair of liquid connectors that connect to the liquid manifold 225 to form the secondary loop, which may include a supply manifold (also referred to as a rack liquid supply line) that supplies cooling liquid to the blade servers 203 and a return manifold (also referred to as a rack liquid return line) that returns warmer liquid to the CDU 201. Note that the CDU 201 can be any variety of commercially available or customizable CDUs. In one embodiment, the heat exchanger 211 of the CDU 201 may be an air-to-liquid heat exchanger or a cooling coil. The details of the CDU 201 will not be described here.

Each blade server 203 may include one or more IT components (e.g., a central processing unit or CPU, a Graphics Processing Unit (GPU), memory, and/or storage devices). Each IT component may perform a data processing task, where the IT component may include software that is installed in a storage device, loaded into memory, and executed by one or more processors to perform the data processing task. In one embodiment, blade server 203 may be part of a compute PoD and may include a master server (referred to as a master node) coupled to one or more compute servers (also referred to as compute nodes, such as CPU servers and GPU servers, and servers with ASIC and FPGA units). A main server (having one or more CPUs) typically interfaces with a client over a network (e.g., the internet) to receive requests for specific services, such as storage services (e.g., cloud-based storage services such as backup and/or restore), execute applications to perform certain operations (e.g., image processing, video streaming, deep learning algorithms or modeling, etc., as part of service software or SaaS platform). In response to the request, the host server distributes the task to one or more of the compute nodes or compute servers (e.g., having one or more GPUs or ASICs) managed by the host server. The compute servers perform the actual tasks, which may generate heat during operation. In one embodiment, when the IT component is part of a storage PoD that includes an array of storage devices, the IT component may provide a storage service, such as a cloud-based storage service, for a client or computing PoD.

The electronics rack 200 also includes an optional RMU 202 and Power Supply Unit (PSU) (not shown) configured to provide and manage power to the supply server 203 and the CDU 201. The RMU 202 may be coupled to a battery backup unit (also not shown) to provide backup energy to the electronics rack 200 and the server 203. The PSU may include the necessary circuitry (e.g., Alternating Current (AC) to Direct Current (DC) or DC to DC power converters, batteries, transformers or regulators, etc.) to regulate the power supplied to the remaining components of the electronics rack 200.

In one embodiment, RMU 202 includes an optimization module 221 and a chassis management controller (RMC) 222. The RMC 222 may include monitors to monitor the operational status of various components within the electronics rack 200, such as the blade servers 203, CDUs 201, and fan modules. In particular, the monitor receives operational data from various sensors that are representative of the operating environment of the electronics rack 200. For example, the monitor may receive operational data indicative of the temperature of the processor, the cooling fluid, and the airflow, which may be captured and collected by various temperature sensors. The monitor may also receive data indicative of fan power and pump power generated by the fan module and liquid pump 212, which may be proportional to the respective speeds of the fan module and liquid pump. These operational data are referred to as real-time operational data. Note that the monitor may be implemented as a separate module within RMU 202. Based on the operational data, the optimization module 221 performs an optimization using a predetermined optimization function or optimization model to derive a set of optimal fan speeds for the fan module and an optimal pump speed for the liquid pump 212. In one embodiment, the RMU 202 may regulate and control the cooling capacity of the cooling fluid supplied to the cooling units connected to the racks 200 based on the temperature of the airflow after heat exchange through the heat exchanger 211 or the cooling coils of the cooling units.

FIG. 3 is a block diagram illustrating a processor cold plate configuration according to one embodiment. The processor/cold plate structure 301/303 may represent any processor/cold plate structure of the blade server 203 as shown in fig. 2. Referring to FIG. 3, a processor 301 is plugged into a processor socket, wherein the processor socket is mounted on a Printed Circuit Board (PCB) or motherboard 302, and the Printed Circuit Board (PCB) or motherboard 302 is coupled to other electronic components or circuitry of the data processing system or server. The processor 301 also includes a cold plate 303 attached thereto, the cold plate 303 being coupled to the liquid supply line and the liquid return line. A portion of the heat generated by the processor 301 is removed by the cooling fluid through the cold plate 303. The remainder of the heat enters the open air space 305, which can be removed by the airflow generated by the cooling fan 304.

Returning to fig. 2, according to one embodiment, the electronics rack 200 also includes one or more Liquid Distribution Units (LDUs), such as LDUs 250A-250E (collectively LDUs 250), located between the blade servers 203 and the rack manifold 225. Each LDU operates as a local liquid distribution manifold and a cooling device for the blade server 203. In fig. 2, the LDU 250 is shown external to the blade server 203, but it LDU 250 may be designed into the blade server 203. In this example, there is an LDU corresponding to one of the blade servers 203. However, in other embodiments, although not shown, an LDU may be associated with multiple blade servers 203.

Similar to the CDU, a primary loop is formed between the LDU and the rack manifold 225, while a secondary loop is formed between the LDU and the blade server. As a result, the liquid distribution circuit is significantly shortened compared to conventional systems. In conventional systems, the cooling fluid received from the supply line 132 or from the CDU 201 must pass through each cold plate in the blade server 203. As a result, the liquid distribution circuit is much longer and the power requirements for pumping the liquid are much higher. Each secondary loop coupled to the cold plate of the blade server is a local single loop through the LDU 250. In one embodiment, the cooling fluid distributed to the blade servers is a two-phase cooling fluid that transitions between a liquid form and a vapor form based on temperature. In such a configuration, a liquid pump for the secondary circuit may not be required.

In one embodiment, a cooling row is connected to each rack row, such as rack 200 of fig. 2, to accommodate the PoD to provide a cooling distribution subsystem to the PoD. The cooling distribution subsystem may include a recirculation loop that supplies cooling fluid to the PoD and returns warm fluid to the cooling source of the data center infrastructure. Different types of IT racks (e.g., computer racks, storage racks, networking device racks) may be disposed anywhere in the PoD. Any space of the data center may be configured to compute a PoD, store a PoD, or heterogeneous compute and store a PoD. The cooling rows may be flexibly configured to meet the cooling requirements of various layouts, arrangements, thermal loads, and configurations of the PoD.

FIG. 4 is a top layout view of a cooling row 401 and associated rows of racks 200, showing the cooling row 401 arranged in the same direction throughout the data center 100, according to one embodiment. The cooling fans in the racks 200 may draw air heated by the electronic components through the air spaces of the racks 200 to be cooled by the cooling coils (not shown) of the cooling rows 401. The cooling coils of the cooling rows 401 are supplied with cooling fluid from cooling supply/cooling return lines 131 and 132. For all of the racks 200 of the cooling row 401, as shown by the outlet airflow 403, and for all of the racks 200 of the various cooling rows 401, after the cooling coils of the cooling row 401 have cooled, the air from the racks 200 flows out of the cooling row 401 in the same direction. Thus, the outlet airflow 403 from one cooling row 401 may be in the same direction as the inlet airflow into an adjacent cooling row 401. For example, the air outlets from one row of racks 200 and associated cooling rows 401 may face the air inlets of an adjacent row of racks 200 and associated cooling rows 401.

In one embodiment, the temperature of the outlet airflow 403 from the cooling row 401 may be measured to control the cooling capacity, such as temperature or volume, of the cooling liquid supplied to the cooling row 401. The rack 200 may accommodate computing or storage pods of different arrangements regardless of the power load requirements of the pods. The number, ratio, and configuration of compute and store pods may be varied to serve different IT applications. One PoD 405 is shown enclosing two rows of racks 200. Because racks 200 housing the PoD may have different power loads and power densities, different amounts of heat may be dissipated, thus possibly requiring different cooling transport subsystems. In one embodiment, in addition to or in lieu of providing liquid cooling of the airflow, the cooling transport subsystem may include liquid-liquid cooling coils, cold plates 303 shown in fig. 3, or other liquid-liquid cooling devices (e.g., rack-level liquid cooling units, rack manifolds, submerged cooling tanks, etc.) to flexibly support the thermal management functions of the PoD.

In one embodiment, the cooling capacity of the different cooling rows 401 may be independently controlled to account for different power loads of the PoD based on the temperature of the outlet airflow 403 from the different cooling rows 401. Thus, the air temperature from the different cooling rows 401 may be controlled to be relatively uniform regardless of the power load of the racks 200 connected to the cooling rows 401. Each location in the data center 100 may be considered identical in thermal management due to the uniformly directed airflow from the racks 200 and the uniform temperature of the airflow. This means that any changes to the IT racks can be deployed in any IT space in the data center 100 without causing any thermal impact, or only a negligible thermal impact, on any other space. Thus, all of the data centers 100 may be considered cold aisles using the cooling rows 401, which also enables the configuration of the cooling distribution subsystem to be independent of the fixed cooling infrastructure of the data centers 100.

Fig. 5 is a top layout view of cooling rows 401 and associated racks 200 in the data center 100, the cooling rows 401 and associated racks 200 being arranged such that each pair of cooling rows 401 face each other, according to one embodiment. This embodiment may be considered as incorporating the design concepts set forth in the present disclosure with a conventional hot aisle-cold aisle data center. For example, when retrofitting a data center to support the presently disclosed design, instead of doing a complete rebuild, some existing facilities, such as electrical or networking facilities, may remain complete and current thermal design configurations may also be implemented to save costs. The outlet airflows 503 from each cooling row 401 of the pair flow towards each other. In one embodiment, the cooling capacity of the pair of cooling rows 401 may be controlled jointly if a pair of racks 200 cooled by the pair of cooling rows 401 has similar power loads based on the outlet airflow 503 from either cooling row 401.

In one embodiment, the cooling capacity of the pair of cooling rows 401 may be independently controlled to account for different power loads of the pair of racks 200 cooled by the pair of cooling rows 401 based on the temperature of the outlet airflow 503 from the respective cooling row 401, even though the outlet airflows 503 from the pair of cooling rows 401 may mix. This independent control of the cooling rows 401 allows the temperatures of the outlet airflow 503 from both cooling rows 401 to be relatively uniform. This arrangement of racks 200 is similar to the alternating hot aisle cold aisle arrangement of a conventional PoD, but because the cooling capacity of the cooling rows 401 connected to the racks 200 are independently controlled, the arrangement does not experience the temperature gradient between the hot and cold aisles of a conventional PoD.

FIG. 6 is a top layout view of a cooling row 401 and associated racks 200 in a data center 100 according to one embodiment, the cooling row 401 and associated racks 200 arranged such that upper and lower portions of the cooling row face in opposite directions. The upper portions of all cooling rows 401 may be in one direction and the lower portions of all cooling rows 401 may be in the opposite direction. Thus, the outlet airflows 603 from the two portions of the cooling row 401 may flow in opposite directions. This layout enables different datacenter chamber airflow dynamics designs, such as PoD zones, that can be used for different use cases. The cooling capacity of the cooling rows 401 may also be independently controlled to remove any temperature gradients between locations in the data center 100, or to maintain room temperature uniform. In one embodiment, optimization and parametric analysis may be used to optimally control the gas flow 603. For example, the RMU 202 of FIG. 2 may monitor the temperature and volume of airflow through the rack 200. Based on this data, the optimization module 221 of the RMU 202 may perform an optimization using a predetermined optimization model to derive a set of optimized fan speeds for the fan modules of the rack 200 to control the airflow 603.

FIG. 7 is a side or rear view of a cooling row 401 showing fluid connections 701 for the liquid supply and return loops 131 and 132 to each cooling coil 703 for individually cooling each rack 200, according to one embodiment. The cooling rows may include racks 200 of modular cooling units arranged in rows to supply the cooling rows to the liquid cooling circuit. The primary unit in each cooling unit may be a cooling coil 703 for cooling the airflow blown through one or more racks 200. The cooling coil is supplied with cooling fluid from a cooling supply line 131 through a cooling distribution subsystem, and functions as a liquid-to-air heat exchanger to cool the temperature of the heat-carrying airflow from the electronic components housed in the racks 200.

In one embodiment, the liquid-to-liquid cooling device (e.g., a rack-level liquid cooling unit, a rack manifold, a submerged cooling tank, etc.) or cold plate 303 in fig. 3 may be connected to the cooling distribution subsystem of the cooling row 401 in addition to or instead of the liquid-to-air cooling coil 703. In one embodiment, liquid-to-liquid cooling of the airflow using cooling coils 703 may not be required if the liquid-to-liquid cooling device has sufficient cooling capacity to cool rack 200. In this case, the cooling coil 703 can be easily removed.

Different types of IT racks (e.g., computer racks, storage racks, networking device racks) may be disposed anywhere in the rack 200. Any portion of the chassis 200 may be configured or partitioned into compute PoDs, store PoDs, or heterogeneous compute and store PoDs. The cooling distribution subsystem of the cooling row 401, which includes liquid-to-gas cooling coils 703 and liquid-to-liquid devices, has a cooling capacity that is flexibly adaptable to the cooling requirements to support the various layouts, arrangements, thermal loads, and configurations of the PoD housed in the rack 200.

In one embodiment, the modular cooling units of the cooling rows 401 have integrated fluid distribution piping that can be interconnected to form a cooling distribution subsystem for distributing cooling liquid from the cooling supply line 131 to the cooling coils 703 and liquid-liquid equipment of the racks 200 and returning heated liquid through the return line 132. For example, fluid connection 701 and fluid port 705 may supply cooling liquid to cooling coil 703 and/or a liquid-liquid cooling device such as a rack-level liquid cooling unit, rack manifold, immersion cooling tank, cold plate, etc., and return heated liquid from cooling coil 703 and/or the liquid-liquid cooling device. In one embodiment, the fluid port 705 may be used as a redundant port to the primary port (e.g., fluid connection 701) of the cooling coil 703. In one embodiment, the cooling coils 703 may be positioned higher than the racks 200 for more efficient heat exchange with the warm air from the racks 200. The modular cooling units of the cooling row 401 may be serviced from the rear side of the cooling row 401, which is easily accessible. In one embodiment, the cooling row 401 may include internal structure for preventing fluid from escaping in the event of any leakage of the cooling distribution subsystem.

The cooling capacity of the cooling rows 401 may be independently controlled by controlling the temperature and/or volume of the cooling liquid from the cooling supply line 131. In one embodiment, the cooling capacity of the modular cooling units of the cooling row 401 may be independently controlled by a pump (not shown) that controls the volume of cooling liquid supplied to the racks 200. Fig. 7 shows that the cooling coils 703 of each modular cooling unit of the cooling row 401 can meet the cooling requirements for each rack 200, respectively. The modular configuration of the integrated fluid distribution ducts and cooling rows 401 enables the configuration of the cooling distribution subsystem for the racks 200 to be independent of the fixed infrastructure of the facility, thereby improving scalability, efficiency, flexibility, availability, and reliability of the liquid cooling system.

FIG. 8 is a side or rear view of a cooling row 401 showing fluid connections for the liquid supply and return loops 131 and 132 to each cooling coil group 703 shared between two adjacent racks 200, according to one embodiment. The two cooling coils of the cooling coil group 703 may also serve as redundant backups for each other. The cooling coil groups 703 may be supplied with cooling fluid from the cooling distribution subsystem of a pair of adjacent modular cooling units via fluid connections 701. The cooling coil assembly 703 may function as a liquid-to-air heat exchanger to cool the temperature of the heat-carrying airflow from the electronic components housed in the pair of racks 200. In one embodiment, more than two cooling coils may be arranged in one common area.

FIG. 9 is a top layout view of a cooling row 401 and associated racks 200, the cooling row 401 and associated racks 200 configured such that the temperature T of the air flow from the cooling row 401₁For independently controlling the cooling capacity of the cooling liquid supplied to the cooling rows 401. The rack 200 may accommodate various layouts, arrangements, and configurations of compute PoDs, storage PoDs, or heterogeneous compute and storage PoDs to serve different IT applications. The pods may have different power loads, power densities, etc., thus requiring different cooling delivery subsystems, such as liquid-to-gas cooling, liquid-to-liquid cooling, or a combination thereof, for heat exchange.

For example, the first PoD 901 may include two conventional racks 200, such as racks for homogeneous computing. The second PoD 902 may include two high-density racks 260 with higher power densities for running machine learning or High Performance Computing (HPC) workloads, and thus have more stringent cooling requirements than the conventional rack 200. The first and second PoD 901, 902 may use liquid-to-gas cooling coils (not shown). In addition to liquid-to-liquid cooling coils, the third PoD 903 may also use liquid-to-liquid cooling devices, such as a submerged cooling tank 905. The fluid flowing in the cooling coil may be the same for the entire room. If certain IT rooms or racks require a different type of working fluid, such as immersion cooling fluid, additional units may be added to the cooling coils, racks 200, or immersion tank 905.

In one embodiment, the temperature T of the outlet airflow of the cooling row 401 of racks connected to the PoD may be based on the temperature T of the outlet airflow of the cooling row 401 after the airflow has been cooled by the cooling coils of the cooling row 401₁The cooling capacity of the liquid supplied to the cooling delivery subsystem of the PoD is independently regulated and controlled in a feedback loop. For example, RMU 202 or facility of a rackThe temperature T of the cooling line 401 may be monitored₁To adjust the temperature or volume of the cooling liquid supplied from the cooling supply line 131 to the cooling row 401. In one embodiment, the volume of cooling fluid supplied to the cooling row 401 may be adjusted by a valve or pump (not shown) of the cooling row 401. In one embodiment, the cooling capacity of each PoD or cooling row of a PoD may be monitored by the monitored temperature T from the PoD or cooling row of the PoD₁To be controlled independently. Thus, the temperature T can be considered₁The cold aisle temperature control, which is a PoD, enables the PoD to independently and dynamically adjust cooling capacity according to the PoD, regardless of the fixed infrastructure of the facility. The control of the cooling capacity may be performed at different levels, such as at the cooling coil level, the cooling row 401 level, or at the PoD level, such as PoD 405, 901, 902, or 903.

In one embodiment, the temperature T₂May be the temperature of the air within the rack. Rack-based temperature T₂The airflow in the housing is controlled and regulated in a feedback loop. For example, the RMU 202 in FIG. 2 may monitor the temperature T of the airflow through the rack₂And volume. Based on this data, the optimization module 221 of the RMU 202 may perform an optimization using a predetermined optimization model to derive a set of optimal fan speeds for the fan modules of the rack to control airflow. Also, the PoD may independently and dynamically adjust the airflow of the PoD regardless of the fixed infrastructure of the facility.

In one embodiment, if the cooling delivery subsystem uses liquid-liquid cooling, the temperature of the warm liquid returned by the liquid-liquid cooling device of the PoD after heat exchange may be measured. The racks or facilities of the RMU 202 may monitor the liquid temperature of the cooling rows 401 to adjust the temperature or volume of cooling liquid supplied to the cooling rows 401 from the cooling supply line 131. In one embodiment, different cooling supply and return lines 131 and 132 may supply cooling fluid to different cooling rows 401 or PoDs and return warmed fluid from different cooling rows 401 or PoDs.

Fig. 10 is a top view of a cooling row 401 according to an embodiment, the cooling row 401 being connected to the infrastructure of the facility to form the cooling liquid supply loop 131 and the warm liquid return loop 132 of the liquid cooling system before the rack is attached. As shown in fig. 4, 5, 6, the cooling rows 401 may be flexibly arranged in various layouts to accommodate different layouts, arrangements, and configurations of compute pous, storage pous, or heterogeneous compute and storage pous. FIG. 10 illustrates a data center room setup prior to deploying any IT devices. The individual devices within the cooling row 401 are not shown in this figure. Advantageously, the modular configuration of the cooling rows 401 and the cooling distribution subsystem may flexibly support rapidly changing IT requirements, including changes in power density, form factor, cooling methods, power delivery mechanisms, rack layout, networking configuration, PoD configuration, etc., without outdating the data center infrastructure.

In the foregoing specification, embodiments of the invention have been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.