Hyperparameter Optimization and a Deep Learning Bridge to Fortran

Abstract:

Implementing artificial neural networks is commonly achieved via high-level programming languages like Python and easy-to-use deep learning (DL) libraries like Keras. These software libraries come pre-loaded with a variety of network architectures and support GPUs for fast and efficient computation. As a result, a DL practitioner will favor training a neural network model in Python, where these tools are readily available. However, many large-scale scientific computing codes across the AGU community are written in Fortran, which makes it challenging to integrate with modern DL methods for promising tasks such as subgrid process emulation via DL networks. Numerical modelers across the atmospheric, oceanographic, geological, seismological, as well as planetary sciences all face this technical barrier to entering the DL era.

To alleviate this problem, we introduce a software library, the Fortran-Keras Bridge (FKB). This two-way bridge connects environments where DL resources are plentiful, with those where they are scarce. FKB offers several unique features, such as customizable layers, loss functions, and network ensembles. We demonstrate how it can be especially helpful for machine learning process emulation in climate modeling when used in conjunction with another software for formal hyperparameter tuning. The context is that designing and training performant neural networks requires one to set many non-learnable values - hyperparameters (HP). Selecting HPs arbitrarily can lead to subpar performance; to improve performance one must search the space of HPs. SHERPA is a python library for HP optimization, which can be integrated to train neural networks in Keras.

FKB, used in conjunction with SHERPA, enables an HP search of candidate models, initially implemented in Keras, to then be transferred and used in Fortran to assess their emergent behavior. By leveraging these two libraries, practitioners can address open questions about model dynamics, stability, and robustness in offline and online (coupled) settings. As proof of concept, we demonstrate how these software packages are helping improve the reliability and robustness of prototype neural networks for subgrid cloud physics modeling in the atmospheric sciences, but the technique should be generally useful across the AGU community.