IN015-07
Letting the community lead the way to data integration: Data standards and documentation developed by domain experts and the ESS-DIVE repository
Wednesday, 9 December 2020: 17:54
Virtual
Robert Crystal-Ornelas1, Charuleka Varadharajan1, Ben P Bond-Lamberty2, Kristin Boye3, Madison Burrus1, Shreyas Cholia1, Joan E Damerow1, Ranjeet Devarakonda4, Hesham Elbashandy1, Kim S Ely5, Amy E Goldman2, Susan L Heinz6, Valerie C Hendrix1, Christopher S. Jones7, Matthew B. Jones7, Zarine Kakalia1, Mario Melara8, Fianna O'Brien1, Stephanie Pennington9, William J Riley1, Emily Robles1, Alistair Rogers5, Makayla Shepherd8, Maegen Simmonds1, Peter Slaughter7, Terri Velliquette10, Pamela Weisenhorn11, Karen Whitenack1 and Deb Agarwal12, (1)Lawrence Berkeley National Laboratory, Berkeley, CA, United States, (2)Pacific Northwest National Laboratory, Richland, WA, United States, (3)SLAC National Accelerator Laboratory, Stanford Synchrotron Radiation Lightsource, Menlo Park, CA, United States, (4)Oak Ridge National Laboratory, Oak Ridge, TN, United States, (5)Brookhaven National Laboratory, Environmental and Climate Sciences Department, Upton, NY, United States, (6)Oak Ridge National Laboratory, Kingston, TN, United States, (7)National Center for Ecological Analysis and Synthesis, Santa Barbara, CA, United States, (8)Lawrence Berkeley National Laboratory, Berkeley, United States, (9)Pacific Northwest National Laboratory, Joint Global Change Research Institute, College Park, MD, United States, (10)Oak Ridge National Laboratory, Oak Ridge, United States, (11)Argonne National Laboratory, Argonne, United States, (12)LBNL, Berkeley, CA, United States
Abstract:
Earth and Environmental Science data repositories are tasked with storing data that comes in a wide range formats. Many repositories, including the US Department of Energy’s (DOE) Environmental Systems Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE) repository, see data integration and synthesis as a key step in harnessing the power of the large datasets contained within the repositories. However, the lack of standardization in data contributed by users can prohibit data reuse and integration.
To kickstart the generation of reporting standards, the ESS-DIVE repository funded six community partners from national labs around the US to develop 7 metadata/data related standards. In this talk, we begin by describing how our community partners achieved consensus on standards for some of the most common data types uploaded to ESS-DIVE. One challenge community partners faced was providing robust documentation so that any data producer could adopt the standards prior to uploading their data to ESS-DIVE. Documentation also needed to be dynamic so that when standards required modifications it was relatively easy to do so.
To overcome this challenge, ESS-DIVE has begun to implement a software versioning-style framework to allow for data standards to be transparently developed and updated. When standards are expanded or updated by community consensus, our versioning framework allows a clear view of any modifications. Data uploaded to the ESS-DIVE repository that adhere to these community standards will be more interoperable and reusable, facilitating synthesis across datasets. These standardized data contributions to ESS-DIVE would then enable a deeper integrated search of the individual data files within the repository through the ESS-DIVE “fusion database”. Ultimately, by developing standards, providing clear documentation, and a transparent way of updating standards, ESS-DIVE provides a sustainable path toward data integration through community-driven standard development.
