Lessons learned from a blind shoreline modelling competition.

Abstract:

In 2018 we launched a competition, “Shoreshop 2018”, to model and predict shoreline changes. Scientists from 15 institutions worldwide tested their models to predict shoreline evolution at Tairua (New Zealand). A total of 19 models between Machine Learning techniques and hybrid models (e.g. equilibrium models) were used and compared during the Shoreshop.

For the first time, prediction capability of models that are broadly used within the scientific community and new approaches were tested in a blind competition, “Shorecast”, where modellers did not have access to the actual shoreline data they were attempting to predict. Since no modeler could test the model performance over the Shorecast period, no parameter tuning was possible. Additionally, all the models were tested with the same input dataset allowing us to objectively evaluate and discuss model advantages and shortcomings.

Some of the main results from the Shoreshop indicate that during the shorecast period (unseen data), traditional models and machine learning techniques showed an evident decrease in models’ capability to predict the shoreline position. A model ensemble, which could somehow account for structural uncertainties in model architecture, performed better than any individual model.

A retrospective look at the Shorecast makes us question the current approach to prediction and we can show that one of the worst-performing models, could be turned into one of the best performing models by simply having the possibility to tune it for the “prediction”. Shoreshop also exposed the lack of a clear approach to long-term predictions and the attempts to predict shoreline changes until 2100 were hampered by problems related, for example, to uncertainty in the representation of the future wave climate.

What could not be “predicted” a priori was the development of a number of collaborations to address problems exposed during the meeting. Also, it is encouraging to know that other institutions are in the process of organizing new blind shoreline competitions, hopefully leading to a more objective framework for prediction.