Exploring the advantages of cloud native, easily consumable, scalable formats for downstream scientific exploitation of point cloud data.

Abstract:

As the amount of geospatial data continues to grow, special considerations must be given for accessibility and exploitability. When working with large scale point cloud data, it quickly becomes apparent that the current storage/delivery mechanisms of delivering full files and subsetting locally are antiquated and insufficient. Currently utilized tools/platforms do allow for aggregation of data into a single access point, but many do not scale well to allow for parallel exploitation/streaming and most rely on a complex architecture for distribution. In this study we are investigating the Entwine Point Tile (EPT) stack as a potential future solution. EPT is an open-source library that allows for parallel data ingestion, serverless data distribution, and optimized data streaming, facilitating quick visualization and data sampling.

A sample set of ICESat-2 ATL06 source data over the Ross Ice Shelf was collected directly from the National Snow and Ice Data Center. Each file was sorted by orbital cycle as a proxy for temporal indexing, then each cycle was processed into a EPT dataset. The resultant datasets were then uploaded into individual S3 buckets, thus establishing distinct EPT endpoints for consumption into a variety of unmodified scientific applications.

An overall reduction in total wire time was noted in correlation with a reduction in file size of roughly 90%. As EPT is optimized based on spatial indexing, additional wire/processing time efficiencies are possible when requesting a spatial subset. Lastly, independent access to each orbital cycle, allows for end users to perform calculations across temporal frames.

The results of the small scale transformation of the ICESat-2 ATL06 product show promising efficiency and accessibility gains; future work for large scale implementations could yield additional progress in increasing consumability, and ultimately further facilitate scientific exploitation.