Autogenerated metrics to measure the scientific impact of models

Abstract:

The Community Surface Dynamics Modeling System (CSDMS) national facility creates an integrated technological infrastructure to advance earth-surface dynamics modeling through knowledge scaffolding with support and buy-in from the global community of earth surface modeling scientists. Through software development, resource sharing, community coordination, knowledge exchange, and technical training, CSDMS has accelerated the pace of scientific discovery. And as computational modeling has become a critical component in better understanding natural processes, CSDMS provides a model repository that makes models more findable, accessible, interoperable and reusable — following the FAIR data principles.

The CSDMS Model Repository provides open access to codes, complete with metadata and bibliographic information. With a growing number of submissions, the repository now provides access to over 400 models. Before sharing open-source models became common practice only a few dozen models were available and so a simple categorized list by domains was sufficient to identify a model. Nowadays, selection criteria have become necessary to let users find the appropriate model. Various useful metrics have been considered for selection criteria, such as the number of downloads as an indication of interest, or submitted code changes or issues by the community, as an indication of active use. However, these metrics fall short of representing successful implementations of models in science related projects. Here, we present a metric, “the h-index for models”, that captures the adoption and scientific impact of models by automatically mining the Microsoft Academics database based on keywords to find all papers in which a model is described or applied. We then use the number of citations of each paper to estimate the h-index for that model. The h-index is now fully automated for 30% of all models and tools available in the CSDMS repository, with the goal to have this metric available for all models in the near future.