Wrestling the four V's of small unoccupied aerial systems data in the cloud and on national cyberinfrastructure

Abstract:

The Airborne Environmental Observations Laboratory for Unoccupied Systems (AEOLUS) focuses on reducing complexity in the data management lifecycle of small unoccupied aerial systems (sUAS) data. Our project’s overall goal is to manage the volume, variety, velocity, and veracity (the so-called four V’s) of big data generated by sUAS which will allow researchers to do transformative research. AEOLUS uses cloud-native computing services, container orchestration, as well as continuous analysis and event driven actions for data processing workflows on existing national cyberinfrastructure. A ‘bring your own hardware’ approach allows researchers to select their own, commercial, or national research cyberinfrastructure, e.g., the Extreme Science and Engineering Discovery Environment (XSEDE), Open Science Grid (OSG), and CyVerse. To make sUAS data more discoverable, AEOLUS utilizes open source search engines, e.g., Apache Lucene, and data management software (DMS), e.g., the Integrated Rule Oriented Data System, for federated data storage. The DMS utilize community established ontologies and vocabularies for metadata schema based on FAIR linked data principles which are generated by the drones and their sensors and by additional annotation using scientific domain specific vocabularies consistent with scientific community of practice recommendations. A ‘bring your own software’ approach allows researchers to choose and develop in their preferred analytical software via containerized workflows. Using free and open source software on publicly available research cyberinfrastructure, researchers expand access to their workflows and accelerate data-intensive scientific discovery and reproducibility of analyses.