A new methodology for analyzing under- and over-confidence using multiple metrics from large ensemble simulations and its application to a seasonal forecasting dataset.

Abstract:

Basic idea of ensemble forecasting and ensemble simulation is that the reality is equivalent to one of ensemble members, but this idea is not always true. In particular, “predictability paradox” recently revealed for seasonal forecasting and decadal prediction of NAO indicates that the ensemble can fail to correctly capture the reality in a way that the model can predict the reality better than expected for the model itself. This situation is called underconfident, and the opposite is overconfident. As a metric of predictability paradox, the ratio of predictable components (RCP) has been used. In this study, a new methodology that analyzes the underconfidence and overconfidence using multiple metrics of large ensemble data with statistical significance assessment is proposed. The present methodology can identify different types of overconfidence and underconfidence.

As an example of application, sea-level pressures (SLPs) in six seasonal forecasting systems available at the Copernicus Climate Change Service are analyzed. It is found that each ensemble systems tend to be overconfident, but this overconfidence is largely reduced in multi-model ensembles, probably because forecasting-system-specific time-varying biases could be canceled between systems. The seasonal forecast of SLPs of multi-model ensembles are underconfident in several areas and some of this underconfidence is related to the underestimation of teleconnection originating from the tropics. These results imply that there is a room of improvement of atmosphere-ocean interaction generating the teleconnections or signal propagation in the atmosphere associated with the teleconnections, and this improvement would result in better seasonal forecast. The method developed in the present study can be also used in time-varying forced experiments such as AMIP type simulation.