IN033-04
PARMAP: Cloud-Ready Pervasively Parallel Analytics and Climate Science

Monday, 14 December 2020: 11:45
Virtual
Brian D Wilson1, Joseph Charles Jacob2, Huikyo Lee3 and Thomas Huang2, (1)NASA Jet Propulsion Laboratory, Pasadena, CA, United States, (2)Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, United States, (3)Jet Propulsion Laboratory, Caltech, Altadena, CA, United States
Recorded Presentation
Abstract:
Many climate analyses now require high-resolution, high-cadence model or observation grids, including fast analytics, climate trending, model evaluation, instrument simulations, and mission design. Thus, there is a growing need for three capabilities: moving the compute to the big data wherever it is; fast, easy-to-develop parallel algorithms; and fast, parallel access to Cloud-ready datasets.

To meet this challenge, we have developed under a NASA CMAC grant an eScience Environment, PARMAP, in which collaborative science is developed using parallel Python codes & web service calls, assembled into custom workflows, and published in shareable Jupyter eNotebooks. The “par” in the name signifies that parallelism is used pervasively throughout the algorithms in the PARMAP library. The core module, parmap, provides a flexible abstraction of Map-Reduce programming patterns that can run, without any code changes, on multiple backends, including multicore, a PySpark cluster, an Xarray/Dask cluster, or a fleet of on-demand AWS Lambda functions.

NASA intends to move many datasets to the Cloud (e.g. AWS S3) and perform analysis there. PARMAP enables a Cloud-Ready “Serverless Science” paradigm in which datasets are stored in S3 as a series of netCDF/HDF/zarr files, science algorithms are easily invoked using parmap to process files/variables or spatial tiles in parallel, and variable subsets are accessed from the original files via Cloud-ready servers using RANGE GET’s to subset variables out of the files without copying them from S3 (e.g. OpenDAP or HSDS).

PARMAP is being applied to a variety of geoscience and climate science algorithms, including: parallel analytics (e.g. time-series of area-averaged statistics) as a new capability in the Apache Science Data Analytics Platform (SDAP), Multi-Model Evaluations (i.e. multi-model, weighted ensembles with uncertainty quantification) under a NASA CMAC grant, and Observation System Simulation Experiments (OSSE’s) for mission design under a NASA AIST grant.

The presentation will discuss the architectural tradeoffs and the Cloud-ready Serverless paradigm, present benchmarks and parallel speedups for the applications, demonstrate “live code” notebooks, and show code examples to illustrate the ease of use PARMAP makes possible.

<< Previous Abstract | Next Abstract >>