Exploring the Practicalities of Installing Liquid Cooling in Data Centers

Rethinking infrastructure, procedures, and skills

Switching to liquid cooling requires a rethink of the whole data center, from the mechanical and electrical systems and the building’s structure to forging a new commissioning path and overhauling resiliency and maintenance plans.

Operations will need a total overhaul with updated procedures and support equipment. This may include lifting gantries to pull servers out of immersion tanks safely or modified small form-factor pluggable transceivers (SFP) to interface servers in an immersion environment. Leak detection systems will become critical and coolant distribution units (CDU), tertiary cooling loops, and distribution manifolds are required to manage fluid flow effectively.

All these elements add complexity to air cooling setup. Operations managers will need to upskill and develop strategies to deal with new technology and methods to deploy them. Rather than just patching in data and power, they may now find themselves connecting coolant piping, bleeding off cooling loops, and trying to manage contamination within the systems.

Transitioning to hybrid data centers will present an especially complicated challenge for infrastructure design. Two very different types of cooling technologies need to be accommodated, each with their own subsets and a huge amount of traditional gear that remains air cooled.

Getting liquid-cooled ready

How far do you want to go?

One of the first questions is, how far do you want to take liquid cooling deployment? Operators need to decide if they want:

• Tap-off points to existing chilled water systems
• Fully deployed suites ready to start accepting direct-to-chip (D2C) equipment
• To upsize the electrical infrastructure and floor loadings

Before going on to the design stage, it is important to know:

• Technology type, such as D2C or immersion
• Power densities
• Target client
• Specific equipment requirements, such as temperatures, pressures, and flow rates
• Resiliency requirements and if any formal accreditation is being sought

Liquid cooling design and installation options

• Partial design and installation: adding tap-off points from existing pipework and branching off the primary water loop to support liquid-cooled equipment. Spatial allowances must be made in the data hall for this hardware, allowing for gradual adoption.
• Design without installation: equipment is chosen, infrastructure reviewed, and a design is completed but installation is postponed. This allows the hall to remain temporarily compatible with traditional air-cooled equipment.
• Extended design and installation: installing CDUs and extending secondary pipework into the data hall. This method is more comprehensive, preparing the facility to support advanced liquid cooling immediately.

Equipment and servers

While industry standards will emerge over time, it’s crucial to understand the exact requirements of the equipment you plan to install to avoid investing in a technology that becomes obsolete.

The first step is to determine the equipment going in and its performance requirements. These might include water supply temperatures, temperature rise across equipment, and pressures and case temperatures on the graphics processing unit (GPU).

If you don’t know, don’t go too far. Instead, work on your strategy for procurement, design, and build. Hold off on action until you’ve secured your customer and confirmed their requirements and equipment compatibility.

Key equipment details you will need to know

• Required supply temperature: this varies from 25 to 40°C and has a significant effect on the rating of equipment facility water temperatures and the system topology of heat rejection plant.
• Expected temperature rise: the return water loop temperature impacts CDU selection and required performance.
• Compatible cooling fluids: the industry looks to be aligning on PG25 in single phase D2C solutions, but compatibility should always be reviewed. Incompatible materials present a significant risk to cooling performance caused by enhanced degradation that could lead to leaks, contamination, or blockages.
• Pressure requirements and equipment: equipment needs to be compatible with the proposed cooling loop pressure.
• Residual heat loss that still requires air cooling: in D2C applications, traditional air cooling is still required to dissipate some residual load from power supplies, memory, and more.
• Server warranty: is it warranted for installation in an immersion environment? If going for an immersion solution, who carries the warranty risk for equipment?
• Cold plate channel size: filtration is required to protect the cold plate from fouling, reducing flow rate, and risking overheating.
• Allowable change and rate of change to flow temperatures at server level: thermal stability is imperative due to the much larger heat-rejection quantities.

Internal equipment configuration also matters

• Are the cold plates configured in parallel or multiple in series? This affects expected supply temperature and rise expected across the equipment.
• What materials make up the cold plates and what other materials may be present in the equipment? Material compatibility influences coolant selection and infrastructure design to prevent corrosion or degradation.

The racks for this equipment are also trending up in size and weight to support the servers and additional cooling infrastructure. If you’re building a new data hall, your floor-to-floor heights, rack layouts, and footprints need to accommodate this size change. If retrofitting an existing space, you need to review existing limitations to ensure compatibility.

Installing D2C in an existing facility

While liquid cooling offers an opportunity to reduce power usage effectiveness (PUE) and limit the energy devoted to keeping servers cool, the technology driving this transition can significantly increase energy consumption.

Essential factors to consider when using existing equipment and infrastructure

• Structural loading capacity of the existing floors: liquid-cooled racks have substantially higher weights and floor loadings.
• Space: a data hall needs sufficient space to install LCT. Pipework needs to be reticulated in an often already-cramped space. Considering installation and management of pipework and leaks into the future may impact where pipework can be installed.
• Electrical supply and distribution: this needs to be sized to accept the higher rack loadings that come with higher density equipment.
• Chilled water infrastructure: do you have existing infrastructure to supply LCT and are the temperatures compatible?

Plant and equipment compatibility

Many data centers have adopted energy-saving technologies like indirect evaporative coolers. If you have centralized chillers, you need to assess their compatibility with the water temperatures required for your liquid cooling loops.

Some rack supply temperatures may not be achievable with dry coolers alone, meaning new central plant equipment will be essential.

Resiliency considerations

Thermal conditions

With significantly greater thermal output, D2C cold plates require a constant supply of coolant to maintain thermal conditions. This is more important than in a traditional air-cooled data center, as some manufacturers suggest that just seconds of cooling loss can cause the GPU to go into thermal overload.

Redundancy

Traditional redundancy models focus on room-level mechanical systems. But with liquid cooling, redundancy must be reconsidered down to the rack level. In an environment already becoming more congested, this presents spatial challenges.

Managing a single loop

Currently D2C solutions typically operate with a single cooling loop per equipment unit. If that loop fails, the rack is at risk. To mitigate this, smart valving strategies, automatic cut-offs, and rapid response systems become essential. These controls can isolate faults and prevent cascading failures across racks.

Commissioning: a critical, ongoing process

Commissioning is a critical element for data centers, demonstrating that the facility will perform as designed and operational risk will be minimized. However, liquid cooling introduces new challenges to commissioning.

The biggest limitation is that LCT systems can only be commissioned so far down the network. While you can fully commission the secondary pipework network and verify CDU performance, it’s impossible to simulate the impact of each rack’s installation accurately before going live.

In traditional mechanical cooling systems, commissioning was typically a one-time activity. With D2C solutions, the equipment needs to be recommissioned with every new piece of equipment. Systems need to be purged and coolant concentrations, flow rates, and fluid temperatures reviewed.

Most importantly, records need to be maintained. Effective management of this process will be essential and will require monitoring and recordkeeping.

Evolving roles: new skills for new challenges

The move to liquid cooling shifts the responsibilities of many roles within the data center. Mechanical designers will find themselves down at rack level delivering bespoke design solutions until the industry begins to standardize.

Designers will need greater information on proposed equipment to allow solutions to be tailored and compatibility verified. Currently there is no one-size-fits-all solution.

Client support teams will require new knowledge so they can ask future tenants the right questions about their systems requirements and proposed equipment.

Operators will face a steep learning curve to understand, operate, and maintain new equipment and will need expertise in chemical treatment and management, hydraulic purging, contamination control, and an understanding of commissioning and balancing flow rates.

Standards organizations, such as the Institute of Electrical and Electronics Engineers (IEEE), International Electrotechnical Commission (IEC), and ASHRAE, and manufacturers will need to work together to standardize equipment and performance parameters to allow standardized solutions.

Redundancy guidelines, such as the Telecommunications Industry Association (TIA) and Uptime Institute, must also continue to adapt and define expectations in the performance of these new technologies.

While many of the components of LCT are similar to existing applications, the boundary between customer and facility owner responsibilities will become more blurred. For traditional cooling systems, facility owners were responsible for providing air at a set temperature to the rack. This has changed to supplying liquid which must be provided to the CDU, the tertiary water loop, the rack, or the server itself.

Enterprise customers must tighten IT procurement policies and work closely with their operations teams to ensure compatibility with any LCT infrastructure.

It’s not a question of “if” but “when”

As the market continues to evolve, those who take the initiative to become liquid-cooled ready will be well-equipped to leverage the benefits of enhanced performance and increased energy efficiency.

The journey won’t be simple, but the potential rewards—better cooling for denser and faster equipment, lower energy use, and greater sustainability—make it essential for the future of data center operations.