Towards Efficient and Robust HPC Software Infrastructure: A Data-Driven Intelligent Approach

Today’s high-performance computing (HPC) clusters, driven by the demanding computational and data access needs of diverse scientific applications, are becoming increasingly complex and heterogeneous. This complexity poses significant challenges for the HPC software infrastructure, particularly in areas such as resource provisioning and data access, necessitating innovative solutions. In this talk, the speaker will describe the data-driven, intelligent approach to building an efficient and robust software infrastructure for future HPC systems. By applying advanced machine learning methods to historical traces, performance logs, and real-time system statistics, this approach facilitates a deeper understanding of HPC system behaviors and superior solutions compared to existing heuristics. This presentation will review some of the speaker’s previous work and then focus on a more recent study of developing an intelligent agent to improve I/O performance for HPC applications.

Dong Dai is an Associate Professor in the Department of Computer and Information Sciences (CIS) at the University of Delaware. Before joining UD, he was an Assistant Professor at UNC Charlotte. Prior to that, he served as a Research Assistant Professor and postdoctoral researcher at Texas Tech University. His research interests include using machine learning and deep learning methods to optimize resource management and data access components in heterogeneous large-scale systems, such as HPC systems. Over the past five years, Dr. Dai has contributed over 20 peer-reviewed publications to top conferences and journals in distributed and parallel systems, including SC, IPDPS, HPDC, TOS, and TPDS. He was also a Best Paper Nominee at ACM HotStorage 2021.