Potential and challenges of using web services in big data and reproducible science workflows; a case study

Abstract:

The increasing availability of web services such as REpresentational State Transfer (RESTful), Web Feature Services (WFS), and Web Map Services (WMS) facilitates publishing reproducible scientific researches. These services allow researchers to develop and publish their workflows based on queries to these web services for data subsetting from sometimes massive databases without worrying about shipping the input data when sharing their workflow. Furthermore, their respective scientific community can reproduce and build upon their work more easily. One of the challenges, however, is the technical aspect of developing such workflows that requires a sufficient understanding of the protocols of the service for generating valid queries and Extensible Markup Language (XML) filters. The learning curve of these protocols and their varieties may impede the adoption of such workflows in the science community. To address these challenges, we developed an open-source Python software stack called Hydrodata that provides a unified and simple Application Programming Interface (API) to web services that are based on RESTful, WFS, and WMS. Although the package is more intended for the hydrology and climatology databases, the underlying libraries are generic and can be used to connect and query any database that follows the supported protocols. We demonstrate the capabilities of Hydrodata by developing a semi-distributed hydrology model that uses Hydrodata for gathering the required inputs. We showcase the convenience of exploiting Hydrodata for gathering the inputs from various databases and at large scales. These inputs are climate data from the Daymet dataset, streamflow observations from the National Water Information System (NWIS), vector river networks and all the watersheds characteristics from the National Hydrography Dataset Plus (NHDPlus), and land use/land cover from the National Land Cover Database (NLCD). Other practical applications of Hydrodata can be in a machine learning framework for feeding data into the input layer, a back-end for a web application that requires access to other web services.