Slicing and Dicing OpenHPC Infrastructure: Virtual Clusters in OpenStack
TimeTuesday, July 306:30pm - 8:30pm
LocationCrystal Foyer and Crystal B
DescriptionThe OpenHPC system provides a set of tools for setting up an HPC cluster. It has been gaining wide-spread adoption and support. It is best-suited for native bare-metal installations across homogeneous nodes; however, some groups have attempted to integrated OpenHPC with OpenStack Ironic to get the benefits of cloud elasticity and bursting, which essentially uses Ironic to create the same homogeneous bare-metal environments used in native OpenHPC installs.
Here, we explore the use of OpenStack VMs (not Ironic bare-metal nodes) as a platform to build a virtual cluster that runs OpenHPC tools. VMs allow administrators more flexibility in defining the characteristics of a cluster's nodes. This is important as systems obtain larger and larger nodes whose resources may not be used effectively unless they can be virtualized (sliced and diced) to the specific needs of applications.
We discuss how the head node/VM can be used to provide the base image for the compute VMs through PXE-booting. As a result, the head node supports the necessary tools needed to build the compute VM image and provision the compute VMs, as opposed to pre-building images outside the optimized OpenHPC workflow using OpenStack's image mechanisms.
Using our approach, we can build virtual OpenHPC clusters dynamically, allowing users to submit jobs using Slurm just like a bare-metal implementation. Our approach allows users to create virtual clusters on demand without adding complexity in setting up OS provisioning as it uses the conventional OpenHPC packages and allows non-HPC workloads to co-exist on the same hardware.