Benchmarking Earth System Models: Separating model deficiencies from uncertainties in model inputs and observations

Abstract:

Evaluating Earth System Models against observation-based reference provides invaluable guidance for future model development. Skill scores have been developed to summarize model performance across multiple statistical metrics. One challenge is to determine whether low scores are driven by deficiencies in (a) the model, (b) its inputs, or (c) the reference data. A second challenge is to judge whether a model performs sufficiently well given the uncertainties associated with model inputs and reference data. To address uncertainties in the forcing we compute model scores for an ensemble of model realizations that is based on different forcings. To account for observational uncertainty, we compute reference scores that quantify the level of agreement among independent reference data sets. Comparing model scores against reference scores shows how well our model performs relative to the performance of independent reference data sets. Our results demonstrate that model scores are very sensitive to the choice of forcing and reference data. For 10 out of 19 variables, the sign of the bias changes depending on what forcing and reference data are used. Reference scores are considerably low, implying large observational uncertainties. Model skill scores are very similar to reference scores, suggesting that model performance is good when considering observational uncertainties. We conclude that the interpretation of benchmarking results must consider the role of uncertainties in model inputs and reference data, and that failing to do so may potentially mislead model development. Our approach was implemented for the Canadian Land Surface Scheme Including Biogeochemical Cycles (CLASSIC), which we evaluated using the Automated Model Benchmarking R-package (AMBER).