SeisFLINK: A Real-time Seismic Data Stream Processing System Based on Big Data Technologies

Abstract:

The real-time data processing capability is an indispensable part of any earthquake monitoring system. The seismic phase detection such as STA/LTA combined with the AIC algorithm scans every incoming data packet, which will turn into a bottleneck when the data ingestion rate increases significantly. To orchestrate all the modules in an earthquake monitoring system, a message exchanging center is often required. The breakdown of the mediator would result in the collapse of the whole system. The random-access of portions of waveforms are in demand for magnitude estimation. To improve the processing performance and ensure high availability, a cluster of computers or virtual machines in the Cloud should be used in the practice of earthquake monitoring in the future. Big data clusters involve a multitude of tools for batching or streaming data processing. Some of them have been already used in seismic processing. In this study, we develop a new loosely-coupled real-time seismic processing system (SeisFLINK) based on Apache Flink, Ignite, and Kafka clusters, which are three big data stream processing systems. The system has a three-layered architecture. The bottom layer contains a cluster of computers or virtual machines in the Cloud, providing hardware support. The middle layer is a big data supporting system that serves as a robust core, implementing common functionalities of message exchanging (Kafka), seismic data caching (Ignite), and computation framework (Flink) with features of horizontal scalability, high availability, replication, and fault-tolerance et al. The seismic specific processing modules are developed on the top layer. SeisFLINK decouples processing subsystems as much as possible. The highly loosely-coupled architecture allows newly developed modules to be integrated with the system very easily. Supported by the middle layer, seismologists don't need to consider low layer implementations when programming and can focus on seismological algorithms. More conveniently, after the improvement of information technology, we can easily upgrade the system by replacing certain subsystems. The system passed the stress test over 50,000 simulated real-time incoming streams. The SeisFLINK explores new technology for better acquisition and processing of massive seismic data streams in real-time.