Can Short and Partial Observations Reduce Model Error and Facilitate Machine Learning Prediction?

Abstract:

Predicting complex nonlinear turbulent dynamical systems is an important and practical topic. However, due to the lack of a complete understanding of nature, the ubiquitous model error may greatly affect the prediction skill. Machine learning algorithms can overcome the model error but they are often impeded by inadequate and partial observations in predicting nature. In this article, an efficient and optimal algorithm is developed that allows to sample multiple distinct nonlinear time series conditioned on the same short and partial observations using an imperfect model. The resulting time series succeed in reducing the model error and enrich the variability of the time evolution patterns, which effectively provide a massive training data set for machine learning forecasts. The sampling algorithm starts with a forward and a backward data assimilation procedure to characterize the correct uncertainty of the posterior estimates. Then the path-wise temporal dependence conditioned on the observations is combined with the point-wise state estimation from data assimilation to develop a recursive sampling scheme. For a rich class of nonlinear and non-Gaussian systems, the conditional sampling is carried out by solving a simple stochastic differential equation, where the short and partial observations serve as the implicit input. The sampling algorithm is applied to create a large training data of multiscale compressible shallow water flows from highly nonlinear and indirect observations. The resulting machine learning prediction significantly outweighs the imperfect model forecast. The sampling algorithm also facilitates the machine learning forecast of a highly non-Gaussian climate phenomenon using extremely short observations.