Earth System Science Machine Learning Challenge Problems

Abstract:

Major recent advances in machine learning have been driven and validated by community benchmark datasets, most notably ImageNet. Benchmarks provide a common standard of comparison for different approaches in terms of both the data and the problem definition. Earth System Science problems contain a variety of challenges not encountered in popular machine learning benchmarks, including space and time correlations across samples and the need to satisfy a mixture of physical constraints. Through collaborations spanning 6 of the 7 labs at NCAR, we have developed four Earth system science machine learning challenge problems that address issues with machine learning from both observational and modeling data. The four challenge problems are short-range lightning prediction with GOES-16, accelerating the processing of HOLODEC cloud particle imager data, emulating the GECKO-A organic chemistry model, and emulating warm rain processes in a bin microphysics scheme. Each of the challenge problems were publicly tested at the AI for Earth System Science Summer School and have since been refined based on feedback from the participants. In this presentation, we will describe the different challenge problems, the initial solutions we have developed, and the significant lessons learned from the process. Common challenges were shared across the problems. We had to build datasets that are small enough to be accessible on a single cloud virtual machine but large enough to for more complex machine learning approaches to converge. We had to develop evaluation metrics for each problem that could capture progress on the problem accurately but are also straightforward to interpret. We found some problems lended themselves easily to existing machine learning solutions while others required extensive investigation and reformulation of the problem. The data and code are now publicly available online at https://github.com/NCAR/ai4ess-hackathon-2020.