Architecture, Principles and Practices for Enterprise Ready, Multi-Cloud Spatiotemporal Data Platforms

Abstract:

The need for a highly scalable spatiotemporal data platform (SDP) and toolset is generally recognized, and significant progress is now being made towards these from a range of initiatives such as Pangeo. The Met Office, in collaboration with Microsoft Azure, have been looking at the next challenges that these tools will need to address. Primary amongst these are multi-cloud (including on-prem) federated platforms, system admin overhead and integration with organisational user and billing management.

Here we present the conceptual framework which is needed in order to allow these challenges to be addressed. In addition we demonstrate applications of these principles to deliver effective Spatiotemporal Data Platforms.

We have approached this challenge by defining capabilities that are required and the application programming interfaces (APIs) between them. In this model capabilities are interchangeable or even “hot swappable” provided they meet the API contract.

The capabilities we define are:

• Interactive compute environment
• Environment management and definition
• Vertically (more powerful) and horizontally (more numerous) scalable compute clusters
• User home spaces
• User authentication and authorization

The APIs to which these services must adhere are defined not as a schema or OpenAPI specifications but rather as integration into one or more open source software packages. For example in order to be an interactive compute environment capability the system needs to have a spawner compatible with Jupyter Hub. To be considered a compute cluster capability it must be able to be orchestrated by the Dask Cloud Provider module.

We demonstrate examples of using this framework to create an SDP that works across multiple cloud providers, integrates with an organization's user management, and manages building and quotas in an operationally acceptable manner. Further work demonstrates how this framework enables working across organizational boundaries, such as a cloud hosted notebook running its computation in an on-premise data center.

We conclude that this model offers a compelling way to think about and drive progress in SDPs both within the open source community and closed source cloud vendors. For organisations requiring SDP this model maximises the advantages of individual cloud providers whilst minimising vendor lock in. Furthermore, it offers operational tools that are largely lacking the current offerings whilst off-loading most of the administration and system management to cloud providers.