Birds of a Feather

First Steps Toward Adopting Direct Liquid Cooled HPC

Authors: Dave Martinez (Sandia National Laboratories, Energy Efficient HPC Working Group), David Sickinger (National Renewable Energy Laboratory (NREL), Energy Efficient HPC Working Group), Sammy Zimmerman (NVIDIA Corporation), Nevil Joseph Silverius (Queen's University, Canada), Chris Tanner (NASA), Chris DePrater (Lawrence Livermore National Laboratory, Energy Efficient HPC Working Group), David Grant (Oak Ridge National Laboratory, Energy Efficient HPC Working Group), Tim Shedd (Dell Technologies Inc), Karsten Kutzer (Lenovo), Steve Martin (Hewlett Packard Enterprise (HPE))

Abstract: Liquid cooling mitigates the effects of heat density, reduces energy consumption and increases performance. It is now a requirement to stay on the chip technology roadmap. After a decade's experience with liquid cooling in large-scale supercomputing centers, many data centers are still facing challenges with adoption. Building on deep expertise from major supercomputing centers, this BoF will present recommendations for initial adoption of direct liquid cooling (DLC). See https://sites.google.com/lbl.gov/ee-hpc-wg-liquid-cooling/home. There will be presentations on experiences from sites that have just adopted DLC. We are expecting a lot of audience discussion and networking that extends beyond the BoF.

Long Description: The main goal of this BoF is to bring together people who are knowledgable in liquid cooling as well as those who are new to liquid cooling. It will also provide a real-time forum for those experiencing roadblocks to seek feedback and advice from others with more experience or different perspectives.

The Energy Efficient HPC Working Group [https://eehpcwg.llnl.gov/] has an open community-based Liquid Cooling How-to Team that has written a liquid cooling how-to guide. This guide gives advice for taking an incremental, "small-steps how-to approach" for transitioning without a lot of re-do and cost. The guide is in final review and the will be available for public review in August at the following site: https://sites.google.com/lbl.gov/ee-hpc-wg-liquid-cooling/home.

The primary authors of this guide are Dave Martinez from Sandia NL and David Sickinger from NREL. Collectively, they have 30+ years of hands-on experience with liquid cooling, including multiple liquid cooling technologies. They are not only experts, but advocates for liquid cooling in high performance computing.

This guide was written with extensive review and input from many other people who are active members of the EE HPC WG. The EE HPC WG has 950+ members from around the globe with ~50% of the membership from major supercomputing centers, 30% from the vendor community and 20% from academe. The EE HPC WG encourages sharing of experiences and collaboration.

This guide is intended to bring a much needed "small-steps how-to approach" to transitioning form air to liquid cooling and to leverage (without replicating) a lot of excellent work already published by the Energy Efficient High Performance Computing Working Group (EE HPC WG), ASHRAE TC9.9, the Open Compute Project (OCP) and The Green Grid.

This topic is very relevant to the HPC community. The increase of compute densities in supercomputers pose a growing demand to more effectively cool power-dense equipment and improve energy efficiency.

The EE HPC Working Group has held 6 Birds of Feather sessions on various aspects of liquid cooling at SC since 2013. All of them have been well attended with strong audience participation. Four of the six have been 90 minute evening sessions. This is an ideal time, especially Tuesday night, because it is generally the only evening session on that night with a facilities focus.

We will do outreach through the EE HPC WG meetings and at the EE HPC WG SC23 booth. Besides operations managers from HPC sites, this BoF will attract many participants from the vendor community; system integrators, liquid cooling suppliers, architecture and engineering firms. We know that there will be strong participation from large-scale HPC facilities in the United States, Europe and Japan via the EE HPC WG.

The EE HPC WG has active collaboration with ASHRAE and the Open Compute Project. The work developed by the EE HPC WG and vetted with the HPC community at SC has been adopted by these broader enterprise and cloud focused organizations.

Website: https://sites.google.com/lbl.gov/ee-hpc-wg-liquid-cooling/home