Birds of a Feather

LUSTRE Community BoF: Lustre in HPC, AI, and the Cloud

Authors: Kevin Harms (Argonne National Laboratory (ANL), OpenSFS), Frank Baetke (European Open File System Association (EOFS)), Andreas Dilger (Whamcloud/DDN), Peter Jones (Whamcloud, inc.), Domini Manno (Los Alamos National Laboratory (LANL)), Ellis Wilson (Microsoft Corporation), Patrick Manning (Hewlett Packard Enterprise (HPE)), Dustin Leverman (Oak Ridge National Laboratory), Gael Delbary (Atomic Energy and Alternative Energies Commission (CEA))

Abstract: Lustre is the leading open-source and open-development file system for HPC. Around two thirds of the top 100 supercomputers use Lustre. It is a community developed technology with contributors from around the world. Lustre currently supports many HPC infrastructures beyond scientific research, such as financial services, energy, manufacturing and life sciences. Lustre clients are available for broadly deployed instruction set architectures such as x86, POWER, and ARM.

At this BoF, Lustre developers, administrators, and solution providers will gather to discuss recent Lustre developments and challenges, including the role of Lustre in AI and its use in Cloud environments.

Long Description: Lustre is the leading open-source and open-development file system for HPC. Around two thirds of the top 100 supercomputers use Lustre file systems. Lustre is a community developed file system with contributors from around the world. Lustre supports many HPC infrastructures beyond its traditional stronghold of scientific research including financial services, energy, manufacturing, life sciences and animation and Lustre clients are available for ISAs such as x86, POWER, and ARM.

At this BOF, Lustre developers, administrators, and solution providers will gather to discuss recent developments, such as the upcoming Lustre 2.16 and new challenges and corresponding opportunities. Today Lustre is in production-use at most of the world’s TOP10 supercomputers including Frontier (USA), Fugaku (Japan), LUMI (Finland), Tianhe-2A (China), Leonardo (Italy) and NVIDIA's Sélene (USA). Lustre is also widely used across mid- and small scale HPC-systems with continued adoption attributable to the stability of the Lustre file system as it has matured. More recently Lustre has also been adopted by major HPC-Cloud providers like AWS and Microsoft Azure.

Vital to the technology is the community that continues to drive Lustre forward. As Lustre has evolved into a true open-source and open-development model, the end users, developers, and solution providers have come together through the worldwide OpenSFS and EOFS communities.

This community development model has resulted in significant new features, improved stability and broader adoption.

The 2023 Lustre BOF will focus on feature developments and discuss how they will shape future Lustre deployments including a panel discussion from key developers, site administrators, and vendors providing the audience a chance to ask questions. Across all key-application segments, Lustre is at the heart of many HPC infrastructures and must continue to evolve in order to support emerging use cases including the scalability challenges associated with exascale systems. We will explore these cases and discuss the Lustre roadmap for meeting the requirements that they present.

The Lustre BOF has been held at previous SC events and has been well attended and enthusiastically received. SC presents a rare opportunity for the worldwide Lustre community to meet and discuss how to make Lustre more successful. Attendance is usually very good with well over 150 attendees. The Lustre BOF at SC17 had about 190 attendees, the BOF at SC18 had approximately 180 and the BOF at SC19 had about 150 attendees. The virtual and hybrid SC20 and SC21 BOFs had 200-300 live attendees. SC22 had 120-130 and the equivalent event at ISC23 had about 190 live attendees.

The targeted audience includes all who are involved with Lustre deployments such as administrators, system architects, developers, solution providers and end users. It includes all who have been contributing or reviewing new features or providing additional tools for management and control.



Back to Birds of a Feather Archive Listing