John Herboth is a leading authority in commissioning mission-critical facilities. He has nearly 20 years of experience commissioning buildings across the United States.
John focuses on the commissioning process, quality control, and assurance activities that help project teams identify issues early in design and construction to improve the quality, functionality, and performance of building systems.
John provides technical and team leadership on projects, specializing in central utility plant hydronic systems, fire life safety special inspections, enclosure commissioning, and direct-to-chip liquid-cooled systems. He is currently serving as past president of the Building Commissioning Association (BCxA) Northwest Chapter and believes that sharing project successes, lessons learned, and best practices is a professional responsibility.
The industry is experiencing growing pains at every level, from utilities and building systems infrastructure to the materials and products that connect, cool, monitor, and control today’s new supercomputers.
Liquid-cooled IT systems have been around for decades. How have requirements for pre-commissioning and cleaning evolved?
While direct-to-chip liquid cooling has been around for decades—since the IBM System/360 Model 91 was developed in the 1960s—the computation output and thermal heat load was relatively small. Now artificial intelligence is driving the industry forward at rocket speed, with ever-increasing computational output resulting in high electrical consumption and thermal heat generation. The industry is experiencing growing pains at every level, from utilities to building systems infrastructure and all the way down to the materials and products that connect, cool, monitor, and control today’s new graphics processing units (GPU).
This has drastically changed the supporting infrastructure needed. Single racks are now approaching 1 megawatt of power input, requiring a high level of heat removal. New GPU chip designs use microchannel heat exchangers, which require five-micron filtration at the coolant distribution unit (CDU) level to avoid damaging blockages that could ruin multimillion-dollar servers. These changes at the component level have resulted in challenges within each system that owners must address to stay on schedule with deployment plans and operate successfully in the long term.
Within the commissioning process, there have been steps added to meet the new cleanliness requirements, such as pre-commissioning of materials and assemblies prior to installation and when final connections are made. Everything from the materials selected to pipe-joining and installation methods and the processes used to clean these hydronic systems, is focused on avoiding contamination of the technology cooling system (TCS) loop and soon-to-be-connected rack equipment.
What lessons have been learned around sequencing commissioning steps—for example when to flush, clean, and passivate relative to module installation?
At every stage of the process, care must be taken to reduce contaminants from being introduced or reintroduced into the hydronic piping systems. Pre-flushing is needed for all equipment coming from a CDU or liquid-cooled load bank vendor to keep the system clean. Each aspect of the system, from the design of the loops to the components used to fill, flush, and filter the system, should be aligned to help support effective cleaning and flushing during construction. The system design should also allow for continuous monitoring, filtering, and—in the worst case—each loop to be drained, flushed, and cleaned again if necessary.
As the particle size filtration and fluid flow requirements tighten, permanent sidecar bag filtration for the TCS loop has been implemented in some laboratory setups to capture particulates without impacting operations and fluid pumping capability at the CDU level.
What determines recommended flow rates and durations for effective flushing of TCS systems, and how do these vary between site-fabricated piping and modular pre-cleaned assemblies?
For standard primary and secondary cooling loops, a minimum of five to six feet per second for fluid velocity is standard for flushing to bring particulate matter back to the system filters and strainers during the flushing and cleaning process. Closed loop detergent circulation in the system typically ranges from 24 to 48 hours for newly constructed systems to remove grease, scale, and other products from the internal walls of pipes and devices.
While industry standards still apply to the primary and secondary facility cooling water loops, there is a need to increase flushing velocity in the TCS loop. This is due to the increasing operational fluid velocities needed to keep up with the large thermal heat rejection requirements of today’s GPUs. This means that cleaning and flushing of the TCS loop may now require flush velocities of at least 10 to 20 feet per second so that any debris is dislodged when the system is being cleaned, rather than later when the expensive new server racks are connected.
Once the CDUs are fully operational—but racks have not been installed—a week of closed loop circulation with the final system fluid using the CDU pumps should be conducted. The final fluid will need to go through either the planned operational filter size or one that is one size finer in the CDU. This is to confirm that it can operate continuously with minimal debris buildup on the filters, which can be confirmed using the CDU’s filter differential pressure sensors. Each CDU has a primary and secondary filter to allow it to continually operate if a filter needs to be cleaned. However, it would be impractical to have facilities teams continually clean filters at every CDU in data halls on even a weekly basis. This is why this final test is conducted.
When are flushing rigs required and how do they compare to CDUs?
The basis of the facility design should be developed to support flushing and filtering the system loops. Before construction, the project team should confirm whether a flushing rig is needed. A flushing rig will be required if the house systems are not up to the challenge of the cleaning and flushing requirements.
How should commissioning teams document and verify cleanliness results for owner or operator acceptance?
Success depends on careful attention to detail. Processes for system fabrication, handling, and installation should be in place to minimize debris. Success relies on planning out each step with checks to verify the level of cleanliness before proceeding to the next step. Chemical analysis will still be required throughout the process and after construction to verify that fluid used stays within its operational specifications to avoid biofouling, acidification, and other degradations of the different working fluid properties.