Improved estimators of model goodness-of-fit

Abstract:

Reliable metrics are needed to summarize the degree to which simulation model output reproduces observations. Two of the most widely used metrics are the Nash-Sutcliffe Efficiency (NSE) and the Kling-Gupta Efficiency (KGE). Most studies of the performance of these and similar metrics fail to distinguish between theoretical definitions of efficiency and their estimators, and so are limited in their ability to make general recommendations. We introduce two theoretical (probabilistic) definitions of efficiency, E and E’, based on the estimators NSE and KGE , respectively, which enable controlled Monte-Carlo experiments at 447 watersheds to evaluate their performance for daily hydrologic data. These experiments enable us to report the degree of bias and variability associated with these two indices. As expected, NSE is on average, equal to its theoretical value, thus it provides an unbiased estimate of its theoretical value. However NSE exhibits enormous variability from one sample to another due to the enormous skewness and periodicity of daily streamflow data. Improved estimators are introduced which account for skewness and periodicity of the streamflow observations. Our improved estimators yield considerable improvements over NSE and slight improvements over KGE and are shown to avoid most previous criticisms of NSE implied by the literature. Simulation models are increasingly needed to mimic high frequency observations which exhibit highly skewed and periodic behavior. In such instances, our improved estimators of efficiency are needed because NSE is no longer suited to such applications.