Using virtual sharding with AWS S3 to transparently supply missing variables to legacy datasets

Abstract:

The Web Object Store (WOS) found in Cloud Computing systems provides potentially vast storage potential. This, coupled with the ability to share data simultaneously with many more users than conventional client-server systems, is driving the movement to stage scientific data in these stores. However, a WOS does not support the random-access capabilities needed by virtually all data access API libraries (e.g., HDF5) and thus new techniques to 'subset in place' data when it is stored in a WOS (e.g., Amazon's S3) have been developed. These techniques include the Zarr data format and other techniques based on file sharding. Using these techniques requires data be reformatted, which entails a tradeoff between the data packaging originally designed by a Science Team and new needs presented by the new technology. We have developed a virtual sharding technique that can use 'legacy' HDF5 files and provide a subset in place functionality when those files are stored on a WOS such as AWS S3 or Google Cloud GCS. In the course of developing this system we discovered that many legacy datasets provide functional inputs which may be utilized to generate data variables not found in the source file and data servers (e.g., OPeNDAP servers) provide those missing variables by performing the computation on-the-fly at run-time. To do this the servers make extensive use of the random-access APIs. When those files are moved to a WOS, this capability is lost. We present a technique where these 'missing' variables are synthesized at the time when files are moved to S3 and can be accessed in a way that is completely transparent to the end user and to the server processing requests for data. Furthermore, software that bypasses the server and reads data 'shards' directly from the WOS could also read these missing variables. The synthesized data values are themselves stored on the WOS and are stored using the same virtual sharding technique as the original data.