Diagnosing Model-structural Errors with a Sliding Time-window Bayesian Analysis

Abstract:

Model errors arise when a simplified model fails to represent all relevant processes of a natural system. Such errors can be time-dependent, especially when a missing process is active in certain periods or situations in nature. Examples in hydrological models include soil freezing, complex vegetation dynamics, or the effect of extreme floods on river morphology. We hypothesize that, for a given model approximation, such missing process can be made visible as time-dependent values of model parameters. This research aims to assist modelers in detecting and diagnosing this type of model error with what we call “time-windowed Bayesian inference”.

We suggest using a time-windowed form of Bayesian model evidence (tBME) as a model evaluation metric, indicating how much the data in specific time windows support the claim that the model is correct. We will explain how to make tBME values a meaningful and comparable indicator in the spirit of hypothesis testing. By using a sliding time window, the sequence of tBME values will reveal where such errors happen. Further, a time sequence of the posterior parameter distribution (or of best-fit calibration parameters) is obtained. This dynamic parameter posterior will provide information about the potential error source, and about influential parameters for model improvement.

We apply our proposed assessment and visualization tool to soil moisture modeling. We demonstrate tBME as model error indicator on several test scenarios, synthetically designed to cover different error sources and error time scales. We also show an application with real-world lysimeter data. Results prove the usefulness of the framework for analyzing and improving models that predict time series of state variables with time-varying importance of underlying processes.