NOAA Environmental Data at NCEI: Cloud Ready - Tools and Tips

Abstract:

NCEI archives petabytes of NOAA’s environmental data and has been working to make data more available and in easier to use formats. Users have options in obtaining environmental data from on-premise servers or cloud providers. NOAA’s Big Data Partnership and cloud contracts position the archive to be a center for artificial intelligence (AI) and machine learning (ML). By developing a responsive suite of products and services, NCEI achieves goals of providing more data, more understanding, and better decisions for commerce and government. The presentation will cover two primary service areas (tools and communication):

1) A wide range of tools and formats supported by NCEI are key resources available for users to customize requests and output to meet their needs.

Enterprise Tools: The new enterprise tools known as Common Data Services provide search, discovery, and access to on-premise or cloud resources helping NCEI better steward environmental data.

GitHub Resources: Available client tools in Jupyter Notebooks, R, etc. which read data, metadata Application Programmer Interface (API’s) or service endpoints provide examples which can quickly help users meet their requirements.

Cloud Formats: Data formatted to be compatible with cloud use is increasingly important. Gridded products in Zarr and point data in Parquet

2) Keeping abreast of service interruptions is essential as the demand for data increases. Users can follow social media, web alerts, or RSS Feeds for all of the latest information.

Social Media: News and features, Facebook, Twitter, and Instagram provide daily updates of relevant NCEI news and events.

GIS Status RSS Feed: NCEI GIS map viewers are constantly being updated with new features and data follow all of the updates at

Data Status RSS Feed: NCEI archives and provides access to numerous operational products used by industry, local, national, state governments and research/education communities.

NCEI Web Site Alerts: The NCEI Content Management System provides text and JSON messaging capability to alert users of outages and system maintenance.