Workshops Archive

PoliMOR: A Policy Engine "Made-to-Order" for Automated and Scalable Data Management in Lustre

Workshop: PDSW23: 8th International Parallel Data Systems Workshop

Authors: Anjus George, Christopher Brumgard, Rick Mohr, Ketan Maheshwari, James Simmons, Sarp Oral, and Jesse Hanley (Oak Ridge National Laboratory (ORNL))

Abstract: Modern supercomputing systems are increasingly reliant on hierarchical, multi-tiered file and storage system architectures due to cost-performance-capacity trade-offs. Within such multi-tiered systems, data management services are required to maintain healthy utilization, performance, and capacity levels. We present PoliMOR, a pragmatic and reliable policy-driven data management framework. PoliMOR is composed of modular, single-purpose agents that gather file system metadata and enforce policies on storage systems. PoliMOR facilitates automated and scalable data management with customizable agents tailored to HPC facility-specific storage systems and policies. Our evaluations demonstrate the scalability and performance of PoliMOR both by its individual agents and as a collective entity. We believe PoliMOR is widely applicable across HPC facilities with large-scale data management challenges and will garner interest from the HPC community, given its flexible and open-source nature.

Back to PDSW23: 8th International Parallel Data Systems Workshop Archive Listing

Back to Full Workshop Archive Listing