A023-05
Challenges of high-resolution large-ensemble data assimilation on supercomputer Fugaku

Monday, 7 December 2020: 17:58
Virtual
Hisashi Yashiro1, Yuta Kawai2, Koji Terasaki2, Shuhei Kudo2, Takemasa Miyoshi2, Toshiyuki Imamura2, Kazuo Minami2, Masuo Nakano3, Chihiro Kodama4, Satoh Masaki5 and Hirofumi Tomita2, (1)National Institute for Environmental Studies, Tsukuba, Japan, (2)RIKEN Center for Computational Science, Kobe, Japan, (3)JAMSTEC, Yokohama-City, Japan, (4)JAMSTEC, Yokohama, Japan, (5)Atmosphere and Ocean Research Institute, The University of Tokyo, Kashiwa, Japan
Recorded Presentation
Abstract:
By achieving the world's most massive ensemble data assimilation (DA) benchmark experiment, we have demonstrated the availability of a state-of-the-art supercomputer in the weather/climate domain. The benchmarks have been conducted on the Japanese new flagship supercomputer "Fugaku" using Nonhydrostatic ICosahedral Atmospheric Model (NICAM) and Local Ensemble Transform Kalman Filter (LETKF).

As the successor of the K computer, which enabled the first global 870m-mesh atmospheric simulation, the development of Fugaku started in 2014. We have committed to systems-applications co-design to achieve 100 times higher performance of NICAM-LETKF than the K computer. Fugaku has a many-core general-purpose CPU (A64FX) based on the Armv8 instruction set and has a high memory transfer performance comparable to that of GPUs. The system also has a high bandwidth of the data transfer for inter-node communication and file I/O with high resiliency. These features are advantageous for weather/climate models, which has complex, huge codebase and data-intensive computational features. We have optimized the source code using a refactoring method, which we call the "household account" method, and also helped the compiler development about a function of partitioning loops with huge bodies. We enhanced the active usage of the single-precision floating-point calculation not only in the dynamical core and major physical processes in the simulation model but also in the eigenvalue decomposition used in the DA.

We performed a global 1024-member ensemble atmospheric simulation and data assimilation with horizontal resolutions of 14km and 3.5km. For the 3.5km-mesh case, we used the 131,705 nodes of Fugaku and the amount of data passed from the model simulation to the DA system was 1.3 PiB. In this study, we show that data volume and transfer performance are clear bottlenecks in near-future weather/climate simulations. It is more important to promote the "data-centric" design in future modeling studies.

<< Previous Abstract | Next Abstract >>