AMP: An Automated Metadata Pipeline

Abstract:

A key to facilitating multi-disciplinary research and analysis that uses the full spectrum of relevant data products is making data “FAIR” (findable, accessible, interoperable, and reusable) - not just for humans, but for automated systems such as numerical models. Effective data discovery services and fully automated, machine-driven transactions require metadata that can be understood by both humans and machines. But such metadata is uncommon. More commonly, metadata records are inadequately contextualized, incomplete, or simply do not exist. When they do exist, they often lack the semantic underpinnings to make them meaningful. Contributing to the problem of inadequate metadata is the fact that tools for generating metadata rely largely on manual curation and have little or no shared semantics. In some cases, metadata curators may be asked to pick from a controlled list of keywords, but this approach does not scale and consistency is difficult to enforce.

The Automated Metadata Pipeline (AMP) project seeks to develop a fully-automated metadata pipeline that integrates machine learning and ontologies to generate syntactically and semantically consistent metadata recordsthat advance FAIR objectives and support Earth science research for a diverse group of stakeholders ranging from scientists to policy makers. AMP uses machine learning techniques to auto-generate semantically consistent, variable-level metadata records for NASA data products and, in collaboration with the ARtificial Intelligence for Ecosystem Services (ARIES) developer and user communities, we demonstrate the value of robust, semantically consistent metadata in addressing usability and scalability issues for data providers and metadata curators; improving the semantic interoperability of NASA data products; and demonstrating the benefits of semantically interoperable, FAIR data across communities of practice.