Resiliency by Design at the MOPITT LANCE Node

Abstract:

The MOPITT SIPS performs three main tasks, namely 1) producing a validated time series of global CO retrievals, 2) assisting with product developments and algorithm enhancements, and 3) creating a Near-Real Time (NRT) product. Although each of these tasks is important in their own right, there is a time critical nature to the NRT data production that distinguishes it from the other two. Related to the expectation of rapid processing is the implicit requirement of high reliability. To address this requirement the MOPITT SIPS LANCE node includes several design features that are intended to mitigate the risk of downtime. Some of these include:

• Isolating the system from components of the system that are associated with occasional instability such as NFS connections to our primary RAID.
• Building the process on a Virtual Machine so that an image can be reinstated on alternate hardware.
• Practicing shifting from the primary LANCE server to our secondary by re-configuring a DNS entry.
• Creating a robust Contingency Plan and actively reviewing it.
• Implementing NIST 800-53 controls designed to reduce the risk of loss of availability.
• Maintaining a redundant system at a physically distant location.
• Preparedness for continuity of operations if our primary sources of ancillary data sources are interrupted.
• Focusing attention on vulnerabilities and removing or fortifying them as they are identified such as a dependence on receiving and properly processing mission-critical emails.
• Phasing out legacy software that is difficult to maintain, such as DPREP.

Through the implementation of these design elements, in particular by running a redundant server, the downtime of the system as a whole has been minimized to essentially 0%. Because of this, the MOPITT SIPS distributes approximately 2GB of NRT data daily.