BASIN-3D v2.0: A Researcher-focused Data Synthesizer

Abstract:

Earth scientists expend significant effort synthesizing data often from multiple data sources for both modeling and empirical analyses. As part of the U.S. Department of Energy’s (DOE) Watershed Function Science Focus Area and a DOE Early Career project, we developed BASIN-3D (Broker for Assimilation, Synthesis and Integration of eNvironmental Diverse, Distributed Datasets) as a data brokering approach to reduce researchers’ data processing burden. BASIN-3D synthesizes diverse data from a variety of remote sources in real-time without the need for additional storage.

At the core of BASIN-3D is a data synthesis model based on Open Geospatial Consortium standards and a plugin framework that maps data sources to the BASIN-3D synthesis model. We built version 1.0 as a Django web framework that supported an interactive web portal. However, the Django framework requires significant overhead to deploy and maintain for single or small teams of researchers. Additionally, it requires advanced software development skills that make the software difficult to use off-the-shelf.

In BASIN-3D v2.0, we extracted the core data model synthesis capabilities to be a stand alone Python package that synthesizes data and metadata into commonly used data objects like Pandas dataframes. Thus, in a few lines of code, researchers can connect to data sources, select data of interest, and generate synthesized data objects with common variable names, units, and aligned timesteps ready for analysis. Furthermore, the process is repeatable and can be expanded to additional data sources.

The current version supports sensor-based time series Earth Science observations across a hierarchy of spatial locations. In this tour, we give an overview of BASIN-3D v2.0’s features. Additionally, we highlight next development steps and challenges presented by this data synthesis approach, for example, finding a synthesis data model that supports diverse data types and maintaining plugins as data sources evolve.