Birds of a Feather

Open Cloud Infrastructure Solutions to Run HPC Workloads

Authors: Stig Telfer (StackHPC Ltd, OpenStack Scientific SIG), Sadaf Alam (University of Bristol), John Lowe (Indiana University), Henk-Jan Zilverberg (University of Groningen), Happy Sithole (Center for High Performance Computing (CHPC), South Africa), Martial Michel (InfoTrend Inc, OpenStack Scientific SIG), Paul Calleja (University of Cambridge)

Abstract: Cloud-native methods are increasingly used for HPC infrastructure. The advantages claimed include agility in system management and flexible support of new and evolving workflows.

In the last ten years, open cloud infrastructure has become widespread in scientific computing and OpenStack is the dominant open source cloud solution. The OpenStack Scientific SIG represents this community.

This session brings together leading practitioners of OpenStack and related technologies for open solutions in production operations. The session will present current use cases of cloud-native open infrastructure. The advantages and challenges of this approach will be presented. Attendees will be invited to share experiences.

Long Description: Virtualization and containerization have grown to prominent use within the realm of HPC, collectively gathered under terms such as cloud-native or software-defined infrastructure. Adoption of these technologies has enabled IT organizations to improve the flexibility of their service, and adapt more quickly to evolving demands. DevOps automation has enabled teams to manage complex environments at scale.

Cloud infrastructure is sometimes assumed to be the preserve of public cloud, either the hyperscalers or specialist HPC service providers. Private cloud, implemented on-premise, exists as an alternative, retaining the cost models and independence associated with hardware ownership and operation. Hybrid cloud describes a combination of both, in varying forms.

OpenStack is the de facto open-source free software for private cloud infrastructure. OpenStack has become widely used within research computing for a range of workloads, including data analytics, AI/ML, Kubernetes platforms, virtual desktop infrastructure and hosting institutional services. More recently, OpenStack has also been breaking down performance barriers and challenging for space within the HPC domain. There is a growing trend for using OpenStack’s bare metal support to provide cloud-native APIs for flexible HPC infrastructure management.

Software-defined supercomputing requires the elimination of performance overhead without sacrificing the flexibility of cloud. Factors such as performance overheads, networking integrations, identity management and system complexity pose daunting architectural challenges. The expert panel leading this BoF are all from institutions that have taken this path and implemented OpenStack environments capable of running intensive workloads. Their use cases are diverse but represent operational deployments at scale.

The OpenStack project supports scientific computing through the OpenStack Scientific SIG. This thriving community of operators, users and developers is a voluntary organization that advocates for scientific use cases within the OpenStack community, and provides "social infrastructure" facilitating gatherings of birds of a feather and sharing of knowledge and experience. This includes workshops, BoFs, forums and fringe social events at conferences around the globe.

OpenStack is the de facto solution for private cloud but poses its own set of unique benefits and challenges. This BoF is aimed at architects, administrators, software engineers, and scientists interested in designing and deploying cloud infrastructure solutions to run HPC workloads.

Examples of operational sites and data centres:

https://www.hpc.cam.ac.uk/cambridge-research-cloud https://jetstream-cloud.org https://www.rug.nl/news/2023/06/habrok-and-codex-hpc-cluster-officially-inaugurated

Website: https://www.openstack.org/use-cases/science/