A GPU-ready, Massively Parallel Ultra-scale E3SM Land Model Development on Summit

Abstract:

The Energy Exascale Earth System Model land model (ELM) is an advanced community model for understanding how natural and human changes in terrestrial land surfaces will affect the climate. ELM simulates several terrestrial system processes, such as land biogeophysics, hydrologic cycle, biogeochemistry, and ecosystem dynamics. This paper focuses on technical aspects of refactoring ELM software system to enable massively parallel, ultra-high resolution (1km x 1km) simulations on a leadership computer (Summit) at Oak Ridge National Laboratory. Considering the complexity of the ELM software system and technical readiness of several cutting-edge computing technologies, we start our software engineering effort by creating a single-site ELM simulation with a coupler-bypass technology. We then investigate new OpenACC features (i.e., deep copy and the routine directive from PGI Fortran) to expedite the data movement of large Fortran derived types and the generation of device code for over a hundred (nested) subroutines on a single Summit node. After that, we develop software utilities to generate OpenACC code segments for each individual ELM subroutine and go through rigorous ELM function validation and performance tuning within a functional unit testing framework. Our current experiments show that the new OpenACC features are robust enough to create dedicated data regions containing complex ELM data structures and to generate device copy of the ELM subroutines within several parallel regions. A single Summit node (with 6 NVIDIA V100 GPUs) can handle around 11300 site simulations in a massively parallel fashion. In collaboration with another data team who are generating ultra-scale (1km x 1km) resolution forcing datasets over North America (around 22 million gridcells), we are designing experiments that use up to 1800 Summit nodes (around the 1/3 capacity of Summit) for ultra-scale ELM simulations over North Ameria.

Keywords: E3SM Land model, Exascale Computing, OpenACC, SUMMIT, Functional Unit Testing, GPU Porting