Improving HPC System Performance by Predicting Job Resources via Supervised Machine Learning
Event Type
Student Paper
Project Management
ML/AI Applications
TimeWednesday, July 314:30pm - 5pm
LocationGold Coast
DescriptionThe majority of our HPC users come from other disciplines than Computer Science. HPC users including computer scientists have difficulties and do not feel proficient enough to decide the required amount of resources for their submitted jobs on the cluster. Consequently, users are encouraged to over-estimate resources for their submitted jobs, so their jobs will not be killing due insufficient resources. This process will waste and devours HPC resources; hence, will lead to inefficient cluster utilization.
We created a supervised machine learning model and integrated it into the Slurm resource manager simulator to predict the amount of required memory resources (Memory) and the required amount of time to run the computation. Our model involved using different machine learning algorithms. Our goal is to integrate and test the proposed supervised machine learning model on the Slurm. We used over 1000 tasks selected from our HPC log files to evaluate the performance and the accuracy of our integrated model. The purpose of our work is to increase the performance of the Slurm by predicting the amount of the required jobs memory resources and the time required for each particular job in order to improve the utilization of the HPC system using our integrated supervised machine learning model.

Our results indicate that for large jobs our model helps dramatically reduce computational turnaround time (from five days to ten hours for large jobs), substantially increased utilization of the HPC system, and decreased the average waiting time for the submitted jobs